Text and image based search

Abstract

To provide a computing system and a computer-implemented method that facilitate searching for information based on selection of an object of interest.SOLUTION: A system receives an image, determines an object of interest represented in the image, identifies the object with relation to the object and / or a segment of the image that contains the object, generates an object feature vector representing the object, and generates a label corresponding to the type of the object. The system then compares the generated object feature vector and / or label to stored feature vectors corresponding to the object represented by segments of stored images, generates a similarity score between the object feature vector and each stored feature vector, and generates a ranked list of stored images based on the similarity score determined for each image.SELECTED DRAWING: Figure 4

Description

[Background technology]

[0001] This application claims the benefit of U.S. Application No. 15 / 713,567, filed Sep. 22, 2017, entitled “Text and Image-Based Search,” which is incorporated herein by reference in its entirety.

[0002] As the amount of accessible digital content available to users and customers continues to grow, it becomes increasingly difficult for users to find the content they are searching for. Although several different search techniques exist, such as keyword searching, such systems suffer from many inefficiencies. [Brief description of the drawings]

[0003] [Figure 1A] 1 illustrates an input image captured by a user device in accordance with a described implementation. [Figure 1B] FIG. 1B shows visual search results for a selected object of the input image, and according to the described implementation, the results are images that contain objects that are visually similar to the selected object. [Diagram 2] 4 is an exemplary image processing process according to the described implementation. [Diagram 3] 1 is a representation of a segmented image, depending on the implementation. [Figure 4] 1 is an exemplary object matching process according to the described implementation. [Diagram 5] 13 is another example of the object matching process according to the described implementation. [Figure 6A] 4 illustrates an input image captured by a user device according to the described implementation. [Figure 6B] FIG. 6B shows visual search results for objects of interest in the input image, and according to the described implementation, the results include images related to the objects of interest. [Figure 7]1 is an exemplary object category matching process according to the described implementation. [Figure 8A] 1 illustrates a query with visual refinement options according to a described implementation. [Figure 8B] 1 shows input with visual refinement according to a described implementation. [Figure 8C] 8B shows search results for the query of FIG. 8A refined based on the visual refinement of FIG. 8B in accordance with a described implementation. [Figure 9] 1 is an exemplary text and image matching process according to the described implementation. [Figure 10A] 1 illustrates an exemplary visual refinement input of a query according to the described implementation. [Figure 10B] 10B shows search results and visual refinements for the query of FIG. 10A according to the described implementation. [Figure 11] 1 illustrates an exemplary computing device according to one implementation. [Figure 12] 12 illustrates an exemplary arrangement of components of a computing device such as that shown in FIG. [Figure 13] 1 is a pictorial diagram of an example implementation of a server system that can be used in various implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] Described herein are systems and methods that facilitate searching for information based on the selection of one or more objects of interest from a larger image and / or video. In some implementations, the image of the object may be supplemented with other forms of search input, such as text or keywords to narrow the results. In other implementations, the image of the object may be used to complement or improve existing searches, such as text or keyword searches.

[0005] In many image-based queries (e.g., fashion design, interior design, etc.), the user is not interested in the specific objects depicted in the image (e.g., a dress, a couch, a lamp, etc.), but rather the entire image including those objects and how those objects are arranged (e.g., a style choice between a shirt and a skirt, the placement of a couch in relation to a television). For example, a user may provide an image containing a pair of shoes, indicate the shoes as the object of interest, and wish to view other images containing shoes that are visually similar to the selected shoes, and style combinations of those other shoes with other objects such as pants, shirts, hats, purses, etc.

[0006] In one embodiment, a user may initiate a search by providing or selecting an image that includes an object of interest. The described implementation may then process the image to detect the object of interest and / or receive a selection from the user indicating the object of interest as represented in the image. The portion of the image that includes the object of interest may be segmented from the remainder of the image, the determined object of interest, and / or an object feature vector representing the object of interest generated. Based on the determined object of interest and / or the object feature vector, the stored feature vectors of the segments of other stored images may be compared to the object feature vector of the object of interest to determine other images that include objects that are visually similar to the object of interest. The stored images may be specific images of other visually similar objects, or images of multiple objects, often including one or more objects that are visually similar to the object of interest, thereby providing an image that shows how an object such as the object of interest combines with other objects. The user may select one of the presented images, select additional objects or other objects, or perform other actions.

[0007] In some implementations, a stored image may be segmented into various regions, objects represented in those segments may be determined, and feature vectors representing those objects may be generated and associated with the segments of the image. Once an object feature vector is generated for an object of interest, the object feature vector may be compared to the stored feature vectors of various segments of the stored image to detect images containing visually similar objects. By comparing the object feature vector to the stored feature vectors corresponding to segments of the image, images containing objects that are visually similar to the object of interest may be identified, even when the target image contains representations of many other objects.

[0008] In yet another embodiment, a user may select multiple objects of interest and / or may specify whether the selected objects of interest are objects of positive interest or objects of negative interest. Positive objects of interest are objects selected by the user who are interested in seeing images of other visually similar objects. Negative objects of interest are objects selected by the user that the user does not want included in other images. For example, if a user selects from an image a positive object of a chair and a lamp and a negative object of a rug, implementations described herein may identify other images that contain chairs and lamps visually similar to the selected chair and lamp, which may include representations of other objects, but do not include rugs visually similar to the selected rug.

[0009] In some implementations, an image of an object of interest may be processed to detect a type of the object of interest. Based on the determined type of the object of interest, it may be determined whether the type of object of interest corresponds to a defined category (e.g., food, fashion, home decor). If the type of object of interest corresponds to a defined category, multiple query types may be selected from which results of different queries are returned and mixed as results for the input image. For example, some query types may be configured to receive a query keyword and provide image results based on the keyword. Other query types may be configured to receive an image-based query, such as a feature vector, and compare the image query with stored image information to return results corresponding to the query.

[0010] Various query types can be used to provide results corresponding to the defined categories. For example, if an object of interest is determined to be a type of food, one query type may return content (e.g., text, images, video, audio, etc.) that is related to but does not include a visual representation of the object of interest. Another query type may return images or videos that include objects that are visually similar to the object of interest. In such an example, results from the various query types can be determined and blended to provide a single response to the query that includes results from each query type.

[0011] In yet another example, a user may initiate a text-based query and then use an image of the object of interest to complete the text-based query. Narrow your search For example, a user can enter a text-based query such as "summer clothes," and the described implementation can process the text-based query and determine that the query corresponds to a predefined category (e.g., fashion). The user can then provide an image that contains an object of interest, and use the object of interest to generate results for the text-based query. Narrow your searchFor example, if the object of interest is a red top, the search results that match the text query can be processed to find results that include representations of other tops that are visually similar to the object of interest (in this example, a red top). The results can then be ranked such that results that match the text-based search and that include objects that are visually similar to the object of interest are ranked highest and presented to the user first.

[0012] 1A illustrates an input image captured by a user device 100 according to a described implementation. In this example, a user wishes to search for images that include an object of interest 102, an object that is visually similar to a high-heeled shoe in this example. As will be appreciated, any object that can be represented in an image can be an object of interest. To provide an object of interest, a user can generate an image using one or more cameras of the user device 100, provide an image from a memory of the user device 100, provide an image stored in a memory external to the user device 100, select an image provided by the systems and methods described herein (e.g., a resulting provided image), and / or provide or select an image from another source or location.

[0013] In this example, a user has generated an image 101 using a camera on a user device 100. The image includes multiple objects, such as a high heeled shoe 102, a lamp 104-2, a bottle 104-1, and a table 104-3. Upon receiving the image, the image may be segmented and processed to detect objects within the image and determine objects of interest to perform a search. As described further below, the image may be processed using any one or more of a variety of image processing techniques, such as object recognition, edge detection, etc., to identify objects within the image.

[0014] An object of interest may be determined based on the relative size of the object, whether the object is in focus in the image, the location of the object, etc. In the illustrated example, a high heeled shoe 102 is determined to be an object because it is located toward the center of the image 101 and is physically in front of and in focus with other objects 104 depicted in the image. In other implementations, a user may select or designate an object of interest.

[0015] Once the object of interest is determined, the input image is segmented and a feature vector is generated that represents the object of interest. The generation of the feature vector is described in more detail below. In contrast to typical image processing, the object of interest is extracted or segmented from other portions of the image 101 and an object of interest feature vector is generated such that the object of interest feature vector represents only the object of interest. By generating an object feature vector that represents only the object of interest, rather than the entire image, the quality of the matching described herein is improved. Specifically, as described further below, the stored image is segmented to detect objects in various segments of the stored image and respective feature vectors are generated that represent the objects represented in those images. Thus, each stored image may include multiple segments and multiple different feature vectors, with each feature vector representing an object represented in the image.

[0016] Once an object feature vector representing the object of interest is generated, it may be compared to stored feature vectors representing individual objects contained in the segment of the stored image. As a result, even if the entire stored image is significantly different from the input image 100, it may be determined that the stored image contains a representation of an object that is visually similar to the object feature vector based on a comparison of the object of interest to a stored feature vector representing a smaller segment of the stored image than the entire image.

[0017] In some implementations, the type of object of interest may be determined and used to limit or reduce the number of stored feature vectors that are compared to the object feature vector. For example, if the object of interest is determined to be a shoe (such as a high-heeled shoe), the object feature vector may be compared only to stored feature vectors known to represent other shoes. In another example, the stored feature vectors may be selected for comparison based on locations within the image where objects of a certain type are typically located. For example, again, if the object of interest is determined to be a type of shoe, it may be further determined that shoes are typically represented in the bottom third of the image. In such an example, only stored feature vectors corresponding to segments of the image that are in the bottom third of the stored image may be compared to the object feature vector.

[0018] When the stored feature vector is compared to the object feature vector, a similarity score representing the similarity between the object feature vector and the stored feature vector is determined, and the stored image associated with the stored feature vector determined to have the highest similarity score is returned as a result of the search. For example, Figure 1B shows visual search results for an object of interest, a high heel shoe 102, in the input image 100 of Figure 1A, according to the described implementation.

[0019] In this example, the object feature vector representing the high heel shoe 102 is compared to stored feature vectors representing objects represented in different segments of the stored image which is returned as result image 110. As described below, the stored images may be segmented, objects detected, object feature vectors generated, and associations made between the stored images, the segments, the locations of those segments within the stored images, and the feature vectors held in a data store.

[0020] In this example, an object feature vector representing the object of interest 102 is compared to stored feature vectors representing objects 113-1, 113-2A, 113-2B, 113-2C, 113-3, 113-4, etc., to determine a similarity between the object feature vector and the stored feature vector. As shown, images 110 returned in response to a search include objects in addition to the object determined to be visually similar to the object of interest. For example, a first image 110-1 includes a segment 112-1 that includes an object 113-1 determined to be visually similar to the object of interest 102, as well as other objects such as a person 105, clothing, etc. As described further below, the returned stored images may include several segments and / or objects. Alternatively, the returned stored images may include only the visually similar objects. For example, the fourth image 110-4 includes a single segment 112-4 that includes an object 113-4 that is visually similar to the object of interest 102, while no other objects are represented in the image.

[0021] The second image 110-2 includes multiple segments 112-2A, 112-2B, 112-2C of the same type as the object of interest 102, and multiple objects 113-2A, 113-2B, 113-2C. In such an example, the object feature vector may be compared to one or more feature vectors associated with the second image 110-2 and representing different objects. In some implementations, the similarities between the object feature vector and the stored feature vectors associated with the second image 110-2 are averaged, and the average is used as the similarity of the second image 110-2. In other implementations, the highest similarity score, the lowest similarity score, the median similarity score, or other similarity score may be selected as representative of the visual similarity between the object of interest and the image.

[0022] Upon receiving the results from the comparison of the generated object feature vector with the stored feature vectors, the user may view and / or interact with the images 110 provided in response thereto. The images may be ranked and presented such that images associated with stored feature vectors having higher similarity scores are ranked higher and displayed before images associated with feature vectors having lower similarity scores.

[0023] FIG. 2 is an exemplary image processing process that may be performed to generate stored feature vectors and labels representing segments and objects of stored images maintained in a data store according to the described implementation. The exemplary process 200 begins with selecting an image to process, as at 202. Any image may be processed according to the implementation described with respect to FIG. 2. For example, an image stored in an image data store, an image generated by a camera of a user device, an image held in a memory of a user device, or other image may be selected for processing according to the exemplary process 200. In some cases, the image processing process 200 may be used to generate segments, labels, and / or corresponding feature vectors for all objects in the stored image such that the segments, labels, and / or feature vectors are associated with the stored image and may be used in determining whether an object of interest is visually similar to one or more objects represented in the stored image. In another example, the image processing process 200 may be performed on an input image to generate labels and / or object feature vectors for determined objects of interest.

[0024] Once the image is selected, the image is segmented, as at 204. Various segmentation techniques can be used, such as circle packing algorithms, superpixels, etc. The segments of the image can then be processed, as at 206, to remove background regions of the image from consideration. The background regions can be determined, for example, using a combination of attentive constraints (e.g., salient objects are more likely to be in the center of an image segment) and uniqueness constraints (e.g., salient objects are more likely to be distinct from the background). In one embodiment, each segment (S i ), a combination of color, texture, shape, and / or other feature detection can be used to compute unique constraints. The pairwise Euclidean distance of all pairs of segments: L2(S i , S j )teeth, The same calculation is performed for JPEG0007670764000001.jpg415. i Unique constraints on U or U i teeth, Each segment S i A careful constraint on It can be calculated as JPEG0007670764000003.jpg731, where X' and Y' are the center coordinates of the image.

[0025] Next, let us define one or more segments S', a subset of S as U(s)-A( s )>t, where t is a threshold that is either set manually or learned from data. The threshold t can be any defined number or amount that is utilized to distinguish segments as background information or potential objects, or JPEG0007670764000004.jpg727, and JPEG0007670764000005.jpg427, JPEG0007670764000006.jpg42 is an element of S' iis an element R-, where R- is the set of non-salient regions (background) of the image, which can be calculated and used as the similarity between each segment against a labeled database of labeled salient and non-salient segments. The final score is: JPEG0007670764000007.jpg787

[0026] In another embodiment, a selection of interesting parts for past interactions of the same user may be determined. The final segments S' are then clustered to form one or more segments, each of which is a distinctive part of the image.

[0027] Returning to FIG. 2, once the background segments have been removed, the objects remaining in the image are determined, as at 208. The objects remaining in the image can be determined, for example, by using a sliding window approach to calculate a score for each possible hypothesis of the object's location. Each segment can be processed to determine possible matching objects, using approaches such as boosted selection of wavelets like Harr, or multi-part based models. For example, a feature vector can be determined for the segment and compared to information stored for the object. Based on the feature vector and the stored information, a determination can be made as to how similar the feature vector is to a stored feature vector for a particular object and / or type of object.

[0028] The sliding window approach can be run N times, each using a different trained object classifier or label (e.g., person, bag, shoes, face, arm, hat, pants, top, etc.). After determining the hypotheses for each object classifier, the output is a set of optimal hypotheses for each object type. Since objects typically do not appear randomly in an image (e.g., eyes and noses usually appear together), position-dependent constraints can also be taken into account. For example, the position of the root object (e.g., person) is defined as W(root), and each geometric constraint for each object k is expressed as a 6-element vector The root object W is represented relative to the image as JPEG0007670764000008.jpg47. root For each object W oi The geometric "fit" of Defined by JPEG0007670764000009.jpg415. JPEG0007670764000010.jpg428 where dx, dy are the object box W oi is the average geometric distance between each pixel of and each pixel of the root object box. The problem of finding JPEG0007670764000011.jpg44 is arg min λ i JPEG0007670764000012.jpg719, where D train (Θ i ) is the function of Θ on training or other stored images. i is the observed value.

[0029] To optimize this function, the location of the object in the image can be determined. For example, the center of the root object (e.g. a person) in the image is marked as (0, 0) and the location of other objects in the processed image is shifted relative to the root object. Then, a linear Support Vector Machine (SVM) is applied to Θ i is applied as a parameter. The input to the SVM is Dtrain (Θ i ) Other optimization techniques such as linear programming, dynamic programming, convex optimization, etc. may also be used alone or in combination with the optimization techniques described herein. The training data D train (Θ k ) may be collected by having a user place a bounding box over both the entire object and the landmarks. Alternatively, semi-automated approaches such as face detection algorithms, edge detection algorithms, etc. may be used to identify the object. In some implementations, other shapes may be used to represent the object, such as ellipses, ellipses, and / or irregular shapes.

[0030] Returning to FIG. 2, a feature vector and label are generated and associated with each identified object, as at 210 and 212. Specifically, a bounding box containing the object is associated with the generated feature vector and association for the segment and the label maintained in data store 1303 (FIG. 13). In addition, the positions and / or sizes of the bounding boxes forming the segments of the image may be associated and stored with the image. The size and / or position of the segment may be stored, for example, as pixel coordinates (x, y) corresponding to the edges or corners of the bounding box. As another example, the size and / or position of the segment may be stored as column and / or row position and size.

[0031] A label may be a unique identifier (e.g., a keyword) that represents an object. Alternatively, the label may include classification information or object type. For example, a label associated with a representation of clothing may include an apparel classifier (e.g., a prefix classifier) ​​in addition to a unique identifier for the object. In yet other implementations, the label may indicate attributes of the object represented in the image. The attributes may include, but are not limited to, the object's size, shape, color, texture, pattern, etc. In other implementations, a set of object attributes (e.g., color, shape, texture) may be determined for each object in the image and the set may be concatenated to form a single feature vector that represents the object. The feature vector may then be converted to a visual label using a visual vocabulary. The visual vocabulary may be generated by running a clustering algorithm (e.g., K-means) on features generated from a large dataset of images, with the cluster centers becoming the vocabulary set. Each single feature vector may be saved and / or translated into one or more vocabulary terms that are most similar to the feature space (e.g., n).

[0032] After associating a label and feature vector with each object represented in the image, the objects and corresponding image segments are indexed, as at 214. Each object can be indexed using standard text-based search techniques, except that unlike standard text or visual search, multiple indexes can be maintained in the data store 1303 (FIG. 13) and each object can be associated with one or more of the multiple indexes.

[0033] FIG. 3 is a representation of a segmented image that may be held in a data store, according to one embodiment. An image, such as image 300, may be segmented using the segmentation techniques described above. Using the exemplary routine 200, background segments have been removed and six objects in the image have been segmented and identified: body object 302, head object 304, top object 306, pants object 308, bag object 310, and shoe object 312. As part of the segmentation, the root object (body object 302 in this example) is determined and the locations of the other objects 304-312 are considered when identifying those other objects. Once the object types are determined, a label or other identifier is generated and associated with the image segment and the image.

[0034] In addition to indexing segments, determining objects, generating labels, and associating segments and labels with image 300, feature vectors representing each object in image 300 are generated, stored in a data store, and associated with image 300, segments, and labels. For example, a feature vector representing the size, shape, color, etc. of a wallet object may be generated and associated with image 300 and segment 310. Feature vectors representing other objects detected in the image may be similarly generated and associated with those objects, segments, and image 300.

[0035] In other implementations, the image may be segmented using other segmentation and identification techniques. For example, crowdsourcing techniques can be used to segment the image. For example, when a user views an image, they can select regions of the image that contain objects and label those objects. As more users identify objects in an image, the reliability of the identification of those objects increases. Based on the user-provided segmentation and identification, the objects in the image can be indexed and associated with other visually similar objects in other images.

[0036] 4 is an example object matching process 400 according to a described implementation. The example process 400 begins with receiving an image that includes a representation of one or more objects, as at 402. As with other examples described herein, the image may be received from any of a variety of sources.

[0037] Once the image is received, the image is processed using all or a portion of the image processing process 200 described above to determine objects of interest represented in the image, as shown at 404. In some implementations, the entire image processing process 200 may be performed and then the object of interest may be determined from the objects detected as part of the exemplary process 200. In other implementations, one or more object detection algorithms may be performed to determine potential objects in the image, and then one of the potential objects may be selected as the object of interest and the exemplary process 200 may be performed on that potential object.

[0038] For example, edge detection or object detection algorithms may be performed to detect potential objects in the image, and the location of the potential objects, the clarity or focus of the potential objects, and / or other information may be utilized to detect the object of interest. For example, in some implementations, the object of interest may be determined to be toward the center of the image, in focus, and located in the foreground of the image. In other implementations, a user may provide an indication or selection of a segment of the image that contains the object of interest.

[0039] Once an object of interest has been determined, image processing process 200 is performed on the object and / or a segment of the image containing the object to identify the object, generate an object feature vector representing the object, and generate a label corresponding to the type of object, as at 406.

[0040] The generated object feature vector and / or label is then compared to stored feature vectors corresponding to objects represented in the segments of the stored images, as at 408, to generate a similarity score between the object feature vector and each stored feature vector. In some implementations, rather than comparing the object feature vector to all stored feature vectors, the label representing the type of object may be used to reduce the stored feature vectors to include only those with the same or similar label. For example, if the object of interest is determined to be a shoe, then the object feature vector is compared only to stored feature vectors that have a shoe label, thereby limiting the comparison to objects of the same type.

[0041] In other implementations, in addition to or as an alternative to comparing the object feature vector to stored feature vectors with the same or similar labels, the location of the segment in the stored image is such that the object of interest is expected to be located in a particular segment of the stored image. For example, if the object of interest is determined to be a shoe, then it may be determined that a segment in the bottom third of the stored image is most likely to contain the shoe object, and the comparison of feature vectors may be limited to the bottom third of the stored image. Alternatively, the location of the object of interest relative to a root object (e.g., a person) may be determined and utilized to select a feature vector corresponding to a segment of the stored image based on its relative location to the root object, as described above.

[0042] The comparison of the object feature vector with the stored feature vector generates a similarity score indicative of the similarity of the object feature vector with the compared stored feature vector. Images associated with stored feature vectors having higher similarity scores are determined to be more amenable to search and image matching than stored images associated with feature vectors having lower similarity scores. Because a stored image is associated with multiple stored feature vectors that can be compared with the object feature vector, in some implementations an average similarity score for the image is determined based on the similarity scores determined for each associated stored feature vector. In other implementations, the similarity score for an image having multiple stored feature vectors compared with the object feature vector may be a median similarity score, a minimum similarity score, or other variations of the similarity scores of the feature vectors associated with the stored images.

[0043] Based on the similarity score determined for each image, a ranked list of the stored images is generated, as at 410. In some implementations, the ranked list may be based solely on the similarity score. In other implementations, one or more of the stored images may be weighted higher or lower based on other factors, such as the popularity of the stored image, whether the user has previously viewed and / or interacted with the stored image, some stored feature vectors associated with the stored image, the feature vectors associated with the stored image compared to the object feature vector, the stored feature vectors associated with the stored image that have the same or similar labels as the object of interest, etc.

[0044] Finally, as at 412, the plurality of results of stored images are returned, for example, to the user device, based on the ranked results list. In some implementations, the exemplary process 400 may be performed in whole or in part by a remote computing resource remote from the user device, and the plurality of results of images corresponding to the ranked results list may be transmitted to the user device for presentation to the user device in response to the user device transmitting the image of the object of interest. In other implementations, parts of the exemplary process 400 may be performed on the user device and parts of the exemplary process 400 may be performed on the remote computing resource. For example, program instructions stored in a memory of the user device may be executed to cause one or more processors on the user device to receive the image of the object, determine the object of interest, and / or generate a label or object feature vector representing the object of interest. The object feature vectors and / or labels are transmitted from the user device to a remote computing resource, and code executing on the remote computing resource causes one or more processors of the remote computing resource to compare the received object feature vectors with one or more similarity scores, generate a ranked results list, and transmit images corresponding to the ranked results list to the user device for presentation to the user in response to an input image containing the object of interest. In other implementations, different aspects of the exemplary process 400 may be performed by different computing systems at the same or different locations.

[0045] 5 is another exemplary object matching process 500 according to the described implementation. The exemplary process 500 begins with receiving an image that includes a representation of one or more objects, as at 502. As with other examples described herein, the image may be received from any of a variety of sources.

[0046] Once the image is received, the image is processed using all or a portion of the image processing process 200 described above to determine one or more objects of interest represented in the image, as shown at 504. In some implementations, the entire image processing process 200 may be performed to determine candidate objects of interest from objects detected as part of the example process 200. In other implementations, one or more object detection algorithms may be performed to determine candidate objects in the image.

[0047] For example, edge detection or object detection algorithms may be performed to detect objects in the image, and the location of the potential objects, the clarity or focus of the potential objects, and / or other information may be used to detect candidate objects of interest. For example, in some implementations, candidate objects of interest may be determined to be toward the center of the image, in focus, located in the foreground of the image, and / or located near each other.

[0048] A determination is then made as to whether there are multiple candidate objects of interest represented within the image, as at 506. If it is determined that there are not multiple candidate objects of interest, then the single detected object is utilized as the object of interest, as at 507. If it is determined that there are multiple candidate objects of interest, then the image is presented to a user along with an identifier indicating each of the candidate objects of interest, as at 508, so that the user may select one or more of the candidate objects as the object. For example, the image may be presented on a touch-based display of a user device along with a visual identifier disposed adjacent to each candidate object. The user may then provide input by selecting one or more of the candidate objects as the object of interest. User input is then received by the example process, as at 510, and utilized to determine the object of interest.

[0049] In some implementations, a user may be able to specify both objects of interest and objects of no interest, or objects that are given a negative weighting in determining images that match a search. For example, if multiple objects are detected in an image and presented to the user for selection, the user may provide a positive selection that indicates the object as an object of interest, a negative selection that indicates the object as an object of no interest, or no selection that is not considered in determining saved images that match a search.

[0050] Upon determining objects of interest, or if there is only one object of interest, image processing process 200 is performed on those objects and / or segments of the image containing the objects to identify the objects, generate feature vectors identifying the objects, and create labels corresponding to each object type, as at 512. In examples including both objects of interest and objects of no interest, example process 200 (FIG. 2) may be performed on both types of objects and feature vectors / labels created for both objects of interest and objects of no interest.

[0051] Each of the generated object feature vectors and / or labels are compared to stored feature vectors corresponding to objects represented in the segments of the stored images, as at 514, to generate a similarity score between each object feature vector and each stored feature vector. In some implementations, rather than comparing the object feature vector to all stored feature vectors, a label representing the type of object may be used to reduce the stored feature vectors that are compared to different object feature vectors, such that only stored feature vectors are compared to object feature vectors of the same or similar type. For example, if one of the objects of interest is determined to be a shoe, then the object feature vector of that object may be compared only to stored feature vectors with a shoe label. Similarly, if a second object of interest is determined to be a top, then the object feature vector of that object may be compared only to stored feature vectors with a top label.

[0052] In other implementations, in addition to or as an alternative to comparing the object feature vector to stored feature vectors with the same or similar labels, the location of the segment in the stored image is such that the object of interest is expected to be located in a particular segment of the stored image. For example, if the object of interest is determined to be a shoe, then it may be determined that a segment in the bottom third of the stored image is most likely to contain the shoe object, and the comparison of feature vectors may be limited to the bottom third of the stored image. Alternatively, the location of the object of interest relative to a root object (e.g., a person) may be determined and utilized to select a feature vector corresponding to a segment of the stored image based on its relative location to the root object, as described above.

[0053] Comparison of the object feature vectors with the stored feature vectors produces a similarity score indicative of the similarity of each object feature vector with the stored feature vector being compared. Images associated with stored feature vectors having higher similarity scores are determined to be more amenable to search and image matching than stored images associated with feature vectors having lower similarity scores. Because a stored image may be associated with multiple stored feature vectors that can be compared with one or more object feature vectors, in some implementations an average similarity score for the image is determined based on the similarity scores determined for each associated stored feature vector. In other implementations, a similarity score for an image having multiple stored feature vectors compared with multiple object feature vectors may produce two similarity scores, one for each object feature vector. In examples involving similarity scores for objects of no interest, similarity scores may be determined similarly by comparing the object feature vectors of no interest with the stored feature vectors.

[0054] Based on the similarity scores determined for each image, a ranked list of the stored images is generated, as at 516. In some implementations, the ranked list may be based solely on the similarity scores. In implementations where multiple similarity scores are determined for different objects of interest, the ranked list may be determined such that an image associated with a high similarity score for both objects of interest is ranked higher than an image with a high similarity score for only one object of interest. Similarly, if a user specifies an object of no interest, an image that includes an object that is visually similar to the object of no interest may be down-ranked in one or more stored feature vectors associated with the image and the interest. In some implementations, other factors may be considered in ranking the stored images. For example, one or more of the stored images may be weighted higher or lower based on the popularity of the stored image, whether the user has previously viewed and / or interacted with the stored image, a large number of feature vectors associated with the stored image compared to the object feature vector, a large number of stored feature vectors associated with the stored image that have the same or similar label as one of the objects of interest, etc.

[0055] Finally, as at 518, the plurality of stored image results are returned, for example, to the user device, based on the ranked results list. In some implementations, the exemplary process 500 may be performed in whole or in part by a remote computing resource remote from the user device, and the plurality of image results corresponding to the ranked results list may be transmitted to the user device for presentation to the user device in response to the user device transmitting an image of an object of interest. In other implementations, portions of the exemplary process 500 may be performed on the user device, and portions of the exemplary process 500 may be performed on a remote computing resource. For example, program instructions stored in a memory of the user device may be executed to cause one or more processors on the user device to receive images of objects, determine objects of interest, and / or generate labels or object feature vectors representing the objects of interest. The object feature vectors and / or labels are transmitted from the user device to a remote computing resource, and code executing on the remote computing resource causes one or more processors of the remote computing resource to compare the received object feature vectors with one or more similarity scores, generate a ranked results list, and transmit images corresponding to the ranked results list to the user device for presentation to the user in response to an input image containing the object of interest. In other implementations, different aspects of the exemplary process 500 may be performed by different computing systems at the same or different locations.

[0056] FIG. 6A illustrates an input image 601 acquired by a user device 600 that is used to generate search results according to a described implementation. As with the above example, the input image can be received or acquired from any source. In this example, the input image is captured by a camera on the user device 600 and includes a representation of a pineapple 602, a bottle of water 604-1, and a sheet of paper 604-2. In other implementations, a user can select an image control 608 and select an image stored in a memory of the user device or otherwise accessible to the user device. Alternatively, a user may select a remote image control 606 and view / select an image from multiple images stored in a memory remote from the user device.

[0057] In this example, in addition to processing the image to detect one or more objects of interest in the image, it may be determined whether the objects of interest correspond to a defined category. The defined categories may include, but are not limited to, food, home decor, fashion, etc. A category may include multiple different types of objects. For example, food may include thousands of different food objects, such as pineapples.

[0058] If the object of interest is determined to correspond to a predefined category, multiple query types can be selected and utilized to generate results that are blended to respond to the query of the input image. Different query types may include different types or styles of queries. For example, one query type may be a visually based search that includes images that are visually similar to the object of interest, or image segments that are visually similar to the object of interest, as described above. Another query type is a text-based query that searches and determines content that indicates how the object of interest can be used or combined with other objects of interest. For example, if the defined category is food, a first query type may return results that include images of foods that are visually similar to the object of interest. A second query type may return results that include images of various food combinations, or recipes that include the foods determined to be the object of interest.

[0059] In the multiple query type examples, the inputs used for each query type may differ. For example, a first query type utilizing visual or image-based search may be configured to receive an object feature vector representing an object of interest, which may be compared to the stored feature vector to find stored images containing objects visually similar to the object of interest, as described above. In contrast, a query type may be configured to receive text / keyword input to determine stored images that are not visually similar to the object of interest, but contain labels that match the keywords or are related to the object of interest.

[0060] In an example where one of the query types is configured to receive text / keyword input to search the data store of stored images, keywords or labels corresponding to the objects and / or categories of interest are generated and used to query each stored image.

[0061] In some implementations, each query type may search content held in the same data source, but differences in the query types and how the stored content is queried may return different results, while in other implementations, one or more of the query types may search different content held in the same or different data stores.

[0062] Figure 6B shows visual search results for an object of interest selected from Figure 6A, where, according to a described implementation, the results include images obtained from multiple query types related to the object of interest 602.

[0063] In this example, it is determined that the object of interest, a pineapple, is a food and therefore corresponds to a defined category of food. It is further determined that there are two different query types associated with the food category: one is a visual or image-based search and the other is a text or keyword-based search.

[0064] In this example, a first query type generates an object feature vector representing a pineapple and compares the object feature vector to the stored feature vectors to determine images containing objects visually similar to the object of interest 602. A second query type generates a text query containing the keywords "pineapple + recipe" to search for images related to recipes that use pineapple. In some implementations, the keywords may be determined based on the object and / or category of interest. For example, based on image processing, it may be determined that the object of interest is a pineapple, and thus one of the labels may be the object type of interest (e.g., pineapple). Similarly, a food category may include or have a label associated with it, such as "recipe," that is used in creating a text-based query.

[0065] In other implementations, the keywords utilized by the text-based query may be based on labels associated with images determined from the image-based query. For example, if a first query type is an image-based search that returns images that resemble or contain image segments similar to the object of interest, the labels associated with those returned images are compared and the most frequently used labels are used as keywords for the second query type.

[0066] The results of each query type may be blended and presented as a ranked list of images on the user device 600. In this example, a first image 610-1 related to a recipe for making a pina colada is returned for the second query type, and a second image 610-2 is returned for the first query type 602 that contains an object visually similar to the object of interest (a pineapple), and the two are displayed as a blended result in response to image input by the user.

[0067] In some implementations, keywords or labels such as determined keywords 611-1 to 611-N may be presented on the user device and made selectable by the user to further refine the query. The user may also add their own keywords by selecting an add control 613 and entering additional keywords. Similarly, as described below, in this example, multiple objects are detected in the input image, and indicators 604-1, 604-2 are also displayed on other objects to allow the user to specify different or additional objects of interest. When the user selects different or additional objects of interest, the search results are updated accordingly.

[0068] The user can interact with the results returned and displayed on the user device to refine the search, provide additional or different keywords, select additional or different objects of interest, and / or perform other actions.

[0069] 7 is an example object category matching process 700 according to a described implementation. The example process 700 begins with receiving an image that includes a representation of one or more objects, as at 702. As with other examples described herein, the image may be received from any of a variety of sources.

[0070] Upon receiving the image, the image is processed using all or a portion of the image processing process 200 (FIG. 2) described above to determine one or more objects of interest represented in the image, as shown at 704. In some implementations, the entire image processing process 200 may be performed to determine candidate objects of interest from objects detected as part of the example process 200. In other implementations, one or more object detection algorithms may be performed to determine candidate objects in the image.

[0071] For example, edge detection or object detection algorithms may be performed to detect objects in the image, and the location of the potential objects, the clarity or focus of the potential objects, and / or other information may be used to detect candidate objects of interest. For example, in some implementations, candidate objects of interest may be determined to be toward the center of the image, in focus, located in the foreground of the image, and / or located near each other. In some implementations, object detection only scans the image for certain types of objects that correspond to one or more defined categories. Defined categories include, but are not limited to, food, home decor, fashion, etc. In such implementations, image processing only processes the image to determine whether an object type associated with one of the defined categories is potentially represented in the image. As discussed above, multiple types of objects may be associated with each category, and in some implementations, an object type may be associated with multiple categories.

[0072] A determination is then made, as at 706, as to whether an object of interest corresponds to the defined category or whether an object corresponding to the defined category has been identified within the image.

[0073] An object of interest may be determined to correspond to a defined category based on a type of object of interest determined when the object of interest is identified (e.g., identified as part of example process 200). In implementations where more than one object is determined to be an object of interest, some implementations may require that both objects of interest correspond to the same predefined category. Other implementations require that only one object of interest be associated with a predefined category.

[0074] If it is determined that the object of interest does not correspond to a defined category, the received image is compared to the stored image information, as at 707. For example, a feature vector representing the received image, but not the object of interest, may be generated and compared to the stored feature vector corresponding to the stored image. In other embodiments, a segment feature vector representing one or more objects identified in the received image may be generated and compared to the stored segment feature vector, as discussed above with respect to FIG. 4. Stored images determined to be visually similar to the received image and / or segments of the received image are then returned, as at 709.

[0075] If it is determined that the object of interest corresponds to a predefined category, then a query type associated with the predefined category is determined, as at 708. As discussed above, multiple query types can be associated with a predefined category and utilized to retrieve different types or styles of content in response to a search.

[0076] A determination is then made as to whether one or more of the query types are text-based queries for searching content, as at 710. If one of the query types is determined to be a text-based query, query keywords are determined based on the object, category, user, or other factors of interest, as at 712. For example, as described above, in some implementations, a visual or image-based query can be followed by a text-based query, and keywords can be determined from labels associated with content items / images that match the visual or image-based query. For example, the frequency of words in labels associated with images returned for the image-based query may be determined, and the keywords selected as those words in the most frequent labels.

[0077] The keywords are then used to query the labels and / or annotations associated with the stored content, and a results list is returned, as at 714, ranked based on keyword matches.

[0078] If none of the query types are determined to be text-based queries, or in addition to generating and transmitting a text query, the received image is also compared to the stored images, as at 715. As at block 709, the comparison may be a comparison of a feature vector representing the received image to a stored feature vector representing the stored image, and / or a comparison between one or more segment feature vectors corresponding to objects in the received image (e.g., the object of interest) to feature vectors of the stored segments. The comparison of segment feature vectors may be performed in a manner similar to that described above with respect to FIG. 4 to determine images that contain objects that are visually similar to the object of interest.

[0079] A result ratio is then determined indicating the proportion or percentage of content returned by each query type that is included in the ranked results returned to the user, as at 716. The result ratio or percentage can be determined based on a variety of factors, such as the category, the user's preferences, objects of interest, the quantity or quality of results returned from each query type, the user's location, etc.

[0080] Based on the ratio or proportion of the results, the ranked results for each query type are blended to generate blended results, as at 718. Finally, the blended results are returned to the user device, as at 720, for presentation to the user as responsive to the input image containing the object of interest.

[0081] In some implementations, the exemplary process 700 may be performed in whole or in part by a remote computing resource remote from the user device, and a plurality of results of images corresponding to the ranked results list may be transmitted to the user device for presentation to the user device in response to the user device transmitting an image of an object of interest. In other implementations, parts of the exemplary process 700 may be performed on the user device, and parts of the exemplary process 700 may be performed on the remote computing resource. For example, program instructions stored in a memory of the user device may be executed to cause one or more processors on the user device to receive images of objects, determine objects of interest, and / or generate labels or object feature vectors representing the objects of interest. The object feature vectors and / or labels are transmitted from the user device to the remote computing resource, and code executing on the remote computing resource causes one or more processors of the remote computing resource to compare the received object feature vectors with one or more similarity scores, generate a ranked results list, and transmit images corresponding to the ranked results list to the user device for presentation to the user in response to an input image including the object of interest. In other implementations, different aspects of the example process 700 may be performed by different computing systems at the same or different locations.

[0082] By providing mixed results, a user may be shown both images that contain objects visually similar to a provided object of interest and images that are related to the object of interest but do not necessarily contain representations of objects visually similar to the object of interest. Such mixing is beneficial because users often search for information about the object of interest, combinations of the object of interest with other objects, and recipes related to the object of interest, rather than other images of the object of interest, in a defined category.

[0083] FIG. 8A illustrates a visual Narrow down your search 8 shows a query on a user device with the option to provide a text-based query 807 that includes the keyword "summer clothes." In the illustrated example, the user has entered a text-based query 807 that includes the keyword "summer clothes." In this example, the search input begins with the text-based input and it is determined whether the text-based input corresponds to a defined category, such as food, fashion, home decor, etc. If the text input is related to a defined category, the user is presented with a visual refinement option, where the user can provide an image that includes an object of interest that is used to refine the results that match the text-based query.

[0084] For example, the text-based query 807 may be used to return images 810-1, 810-2, 810-3 through 810-N that are determined to contain annotations, keywords, or labels that correspond to the text-based query "summer clothes." In some implementations, other keywords or labels 811 may also be presented to the user to allow the user to further refine the query. In some implementations, if the input keywords are determined to correspond to a predefined category, a visual Narrow down your search Option 804 is presented.

[0085] In Figure 8B, visual Narrow down your search Upon selection of an option, the camera of the user device is activated and the image captured by the camera and / or camera's field of view is processed to detect the shape of the object represented in the captured image / field of view. For example, if the camera is pointed at a sweater 802, the shape of the sweater is detected and suggested object types 805 are presented to the user so that the user can confirm the object type of interest. Similarly, a shape overlay 803 may also be presented on the display 801 of the user device 800 to indicate the shape of the currently selected object type.

[0086] In this example, the determined object category is fashion and the currently detected object type of the object 802 in the field of view corresponds to the object type "Tops" 805-3. The user may select a different object type by selecting a different indicator, such as "Skirt" 805-1, "Dress" 805-2, "Jacket" 805-N, etc. As will be appreciated, fewer, additional, and / or different object types or indicators may be displayed. For example, the user may be presented with options to select based on color, fabric, style, size, texture, pattern, etc.

[0087] Similarly, in some implementations, rather than utilizing an image from the user device's camera, the user may select image control 808 and select an image from the user device's memory or images accessible to the user device, or the user may select remote image control 806 and select an image from a remote data store as input data.

[0088] As with the other examples, when an image is input, the image is processed to determine objects of interest, labels corresponding to the objects of interest are generated, and feature vectors representing the objects of interest are generated. The labels and / or feature vectors are then utilized to search for and display images determined to correspond to a keyword search. Narrow down For example, FIG. 8C shows search results for the query of FIG. 8A, which, according to a described implementation, can be re-ranked based on the visual input of FIG. Narrow downThe re-ranked "summer clothes" 807 is indicated by top icon 821. As with the other examples, the generated label and / or object feature vector for the object of interest is utilized to generate a similarity score by comparing it to stored feature vectors corresponding to objects contained in the stored images determined to match the original query. In this example, the object feature vector representing sweater 802 (FIG. 8B) is compared to stored feature vectors corresponding to segments of the images determined to correspond to the text query. The images are then re-ranked based on the similarity scores determined from the comparison of the feature vectors, as described above. The re-ranked images are then transmitted to the user device and displayed on the display of the user device in response to the input image. For example, stored images 820-1, 820-2, 820-3, and 820-4 may be determined to include objects visually similar to the object of interest and are ranked at the top of the re-ranked list and transmitted to the user, and displayed on the display of the user device.

[0089] FIG. 9 is an example text and image matching process 900 according to a described implementation. The example process 900 begins upon receipt of a text-based query, such as an input of one or more keywords into a search input box presented on a user device, as at 902. Stored content is then queried to determine images associated with labels or keywords that correspond to or match the query text input, as at 904. Additionally, it is determined whether the text query corresponds to a predefined category, as at 906. For example, categories can be defined and include one or more keywords or labels, and if the text-based input includes a keyword or label such as "outfit," it is determined that the query input corresponds to a predefined category. If it is determined that the query does not correspond to a defined category, the example process is complete, as at 908, and the user can interact with the results presented in response to the text-based query.

[0090] If the query is determined to correspond to a defined category, the user is presented with options to visually refine the search results, as at 910. Narrow down your search may be, for example, a graphical button or icon presented with the search results that is selected by the user to activate a camera to generate an image and / or to select an existing image. In some implementations, the determination of whether a query corresponds to a defined category may be omitted, and at each instance of process 900, a visual indicator of the search results may be displayed, as in 910. Narrow down your search Options can be presented to the user.

[0091] It is also determined whether an image has been received that will be used to refine the results of the query, as at 912. If an image has not been received, the exemplary process 900 is complete, as at 908. However, if an image has been received, the image is processed using all or a portion of the image processing process 200 (FIG. 2) described above to determine an object of interest represented in the image, as at 914. In some implementations, the entire image processing process 200 may be performed and then an object of interest may be determined from the objects detected as part of the exemplary process 200. In other implementations, one or more object detection algorithms may be performed to determine potential objects in the image, and then one of the potential objects may be selected as the object of interest, and the exemplary process 200 may be performed on that potential object.

[0092] For example, edge detection or object detection algorithms may be performed to detect potential objects in the image, and the location of the potential objects, the clarity or focus of the potential objects, and / or other information may be utilized to detect the object of interest. For example, in some implementations, the object of interest may be determined to be toward the center of the image, in focus, and located in the foreground of the image. In other implementations, a user may provide an indication or selection of a segment of the image that contains the object of interest.

[0093] Once an object of interest is determined, image processing process 200 is performed on the object and / or a segment of the image containing the object to identify the object, generate a feature vector representing the object, and generate a label corresponding to the type of object, as at 916.

[0094] The generated object feature vectors and / or labels are then compared, as at 918, to stored feature vectors corresponding to objects in the stored images that were determined to match the text-based query to generate a similarity score between the object feature vector and each stored feature vector.

[0095] As described above, the comparison of the object feature vector with the stored feature vector generates a similarity score indicative of the similarity between the object feature vector and the stored feature vector to which it is compared. Images associated with stored feature vectors having higher similarity scores are determined to be more responsive to visually sophisticated searches than stored images associated with feature vectors having lower similarity scores. Because a stored image is associated with multiple stored feature vectors that can be compared to the object feature vector, in some implementations, an average similarity score for the image is determined based on the similarity scores determined for each associated stored feature vector. In other implementations, the similarity score for an image having multiple stored feature vectors compared to the object feature vector may be a median similarity score, a minimum similarity score, or other variations of the similarity scores of the feature vectors associated with the stored images.

[0096] Based on the similarity score determined for each image, the results of the text-based query are re-ranked into an updated ranked list, as at 920. In some implementations, the ranked list may be based solely on the similarity score. In other implementations, one or more of the stored images may be weighted higher or lower based on other factors, such as the popularity of the stored image, whether the user has previously viewed and / or interacted with the stored image, some stored feature vectors associated with the stored image, the feature vectors associated with the stored image compared to the object feature vector, the stored feature vectors associated with the stored image that have the same or similar labels as the object of interest, etc.

[0097] Finally, the image with the highest rank in the ranked list is returned to the user device for presentation, such as at 922. In some implementations, the example process 900 may be performed in whole or in part by a remote computing resource remote from the user device, and a plurality of image results corresponding to the ranked results list may be transmitted to the user device for presentation to the user device in response to the user device transmitting an image of an object of interest. In other implementations, portions of the example process 900 may be performed on the user device and portions of the example process 900 may be performed on the remote computing resource. For example, program instructions stored in a memory of the user device may be executed to cause one or more processors on the user device to receive images of objects, determine objects of interest, and / or generate labels or object feature vectors representing the objects of interest. The object feature vectors and / or labels are transmitted from the user device to a remote computing resource, and code executing on the remote computing resource causes one or more processors of the remote computing resource to compare the received object feature vectors with one or more similarity scores, generate a ranked results list, and transmit images corresponding to the ranked results list to the user device for presentation to the user in response to an input image containing the object of interest. In other implementations, different aspects of the exemplary process 900 may be performed by different computing systems at the same or different locations.

[0098] FIG. 10A is yet another exemplary visual representation of a query according to the described implementation. Narrow down your search 1007 shows an input. In this example, the user enters a text-based query, "salmon recipes" 1007. If the query corresponds to a predefined category (e.g., recipes) and the user Narrow downIn this example, a streaming video of a field of view of a camera on the user device 1000 is processed in real time or near real time to detect objects in the field of view of the camera. In this example, the field of view in the streaming video is the inside of a refrigerator. In other examples, other areas may be included in the streaming video. Processing may be performed on the user device 1000, by computational resources remote from the user device, or a combination thereof.

[0099] Once an object within the streaming video is detected, e.g., using an edge detection algorithm and / or some or all of the example process 200 (FIG. 2), a keyword or label indicating the type of object detected is presented on the device's display 1001 concurrently with the presentation of the streaming video.

[0100] In this example, strawberries, avocados, and eggs have been detected as potential objects of interest within the field of view of the user device's camera. Once an object is detected, a label 1002 is visually displayed adjacent to the object to indicate that the object has been detected.

[0101] In some implementations, to speed the process of detecting candidate objects of interest and improve the user experience by identifying only candidate objects of interest that correspond to the keyword query, a corpus of potential objects is determined based on the text query, and objects that match the corpus are identified as candidate objects. For example, the text query can be processed to determine that a user is looking for recipes that contain salmon. Based on that information, a corpus of potential objects contained in or referenced in images related to recipes that also contain salmon is determined, and only objects that match the corpus are identified as candidate objects of interest.

[0102] In this example, candidate objects detected in the field of view of the user device's camera are identified by identifiers "strawberry" 1002-2, "egg" 1002-1, and "avocado" 1002-3. As the user moves the camera's field of view, the positions of the identifiers 1002 are updated to correspond to the relative positions of the detected objects, and as additional candidate objects come into view and are included in the streaming video, identifiers for those objects are presented as well.

[0103] A user may select one of the identifiers to indicate that the object is an object of interest. In FIG. 10B, the user has selected the object egg as the object of interest. In response, the keyword egg is added to the query "salmon recipe" 1001-1, as indicated by egg icon 1001-2, and images such as images 1010-1, 1010-2, 1010-3, and 1010-N are determined to contain or be associated with the labels / keywords "salmon," "recipe," and "egg" and are returned to the user for display in response to the query. In some implementations, further selection of the query results may be performed. Narrow down Therefore, other keywords 1011 may be presented to the user device 1000 as well.

[0104] Providing the user with the ability to utilize visual search and / or a combination of visual and text-based search and generate results based on predefined categories determined from the input and / or objects detected in the input improves the quality of results by better inferencing the type of content the user wishes to explore. The increased flexibility of the described implementations provides an improvement over visual search only to provide images that are visually similar to the input image by focusing the visual search (e.g., feature vectors) on segments or portions of the stored image rather than the entire image and / or automatically adding different forms of search (e.g., keywords) to the visual search. Additionally, providing a visual search using either or both visual matches via feature vectors or keyword matching to text-based queries provides a visual search that is similar to the input image by providing a visual search that is .... Narrow down your search By adding,users can express input parameters in different contexts (keywords, visuals) to,better determine and explore the desired information.

[0105] FIG. 11 illustrates an exemplary user device 1100 that may be used in accordance with various implementations described herein. In this example, the user device 1100 includes a display 1102 and at least one input component 1104, such as an optional camera, on the same side as the display 1102 and / or on the opposite side of the device. The user device 1100 may also include an audio transducer, such as a speaker 1106, and optionally a microphone 1108. In general, the user device 1100 may have any form of input / output component that allows a user to interact with the user device 1100. For example, various input components for enabling user interaction with the device include a touch-based display 1102 (resistive, capacitive, etc.), a camera, a microphone, a global positioning system (GPS), a compass, or any combination thereof. One or more of these input components may be included in or in communication with the device. Various other input components and combinations of input components may also be used within the scope of various implementations, as should be apparent in light of the teachings and suggestions contained herein.

[0106] To provide various functions described herein, FIG. 12 illustrates an exemplary set of basic components 1200 of a user device 1100, such as the user device 1100 described with respect to FIG. 11. In this example, the device includes at least one central processing unit 1202 for executing instructions that may be stored in at least one memory device or element 1204. As would be apparent to one skilled in the art, the device may include many types of memory, data storage, or computer-readable storage media, such as a first data storage for program instructions for execution by the processor 1202. Removable storage memory may be used to share information with other devices, etc. Typically, the device includes some type of display 1206, such as a touch-based display, electronic ink (e-ink), organic light-emitting diode (OLED), or liquid crystal display (LCD).

[0107] As described, the device in many implementations includes at least one imaging element 1208, such as one or more cameras capable of imaging objects proximate to the device. The imaging element can include or be based at least in part on any suitable technology, such as a CCD or CMOS imaging element having a determined resolution, focal range, visibility region, and capture rate. The device can include at least one search component 1210 for performing the process of generating search terms, labels, and / or identifying and presenting results that match selected search terms. For example, the user device can be in constant or intermittent communication with a remote computing resource and exchange information, such as selected search terms, images, labels, etc., with the remote computing system as part of the search process.

[0108] The device may also include at least one location component 1212, such as GPS, NFC location tracking, Wi-Fi location monitoring, etc. Location information obtained by the location component 1212 may be used with various implementations described herein as a factor in selecting images that match an object of interest. For example, if a user is in San Francisco and actively selects a bridge (object) displayed in an image, the user's location is considered a factor in identifying visually similar objects, such as the Golden Gate Bridge.

[0109] An exemplary user device may also include at least one additional input device capable of receiving conventional input from a user, including, for example, push buttons, a touch pad, a touch-based display, a wheel, a joystick, a keyboard, a mouse, a trackball, a keypad, or other device or element through which a user can enter commands into the device. In some implementations, these I / O devices may also be connected by wireless, infrared, Bluetooth, or other links.

[0110] 13 is a pictorial diagram of an example implementation of a server system 1300, such as a remote computing resource, that may be used in one or more of the implementations described herein. The server system 1300 may include a processor 1301, such as one or more redundant processors, a video display adapter 1302, a disk drive 1304, an input / output interface 1306, a network interface 1308, and a memory 1312. The processor 1301, the video display adapter 1302, the disk drive 1304, the input / output interface 1306, the network interface 1308, and the memory 1312 may be communicatively coupled to each other by a communication bus 1310.

[0111] The video display adapter 1302 provides display signals to a local display that allows an operator of the server system 1300 to monitor and configure the operation of the server system 1300. The input / output interface 1306 similarly communicates with external input / output devices, such as a mouse, keyboard, scanner, or other input / output devices that an operator of the server system 1300 can operate. The network interface 1308 includes hardware, software, or any combination thereof for communicating with other computing devices. For example, the network interface 1308 may be configured to provide communication between the server system 1300 and other computing devices, such as the user device 1100.

[0112] The memory 1312 typically includes random access memory (RAM), read only memory (ROM), flash memory, and / or other volatile or permanent memory. The memory 1312 is shown to store an operating system 1314 for controlling the operation of the server system 1300. A binary input / output system (BIOS) 1316 for controlling low-level operation of the server system 1300 is also stored in the memory 1312.

[0113] The memory 1312 further stores program code and data for providing a network service that enables the user device 1100 and external sources to exchange information and data files with the server system 1300. Thus, the memory 1312 may store a browser application 1318, which includes computer-executable instructions that, when executed by the processor 1301, generate or retrieve configurable markup documents, such as web pages. The browser application 1318 communicates with a data store manager application 1320 to facilitate data exchange and mapping between the data store 1303, user devices, such as the user device 1100, external sources, and the like.

[0114] The term "data store" as used herein refers to any device or combination of devices in which data can be stored, accessed, and retrieved, and may include any combination and number of data servers, databases, data storage devices, and data storage media in a standard, distributed, or clustered environment. The server system 1300 may include appropriate hardware and software to integrate with the user devices 1100, external sources, and / or data store 1303 as necessary to execute one or more application aspects of the search service 1305. The server system 1300 may cooperate with the data store 1303 to provide access control services and generate content such as matching search results, images containing visually similar objects, indexes of images containing visually similar objects, and the like.

[0115] The data store 1303 may include a number of separate data tables, databases, or other data storage mechanisms and media for storing data related to a particular aspect. For example, the illustrated data store 1303 includes digital items (e.g., images) and corresponding metadata about those items (e.g., labels, indexes). Search history, user preferences, profiles, and other information may be stored in the data store as well.

[0116] It should be understood that there may be many other aspects that may be stored in the data store 1303, which may be stored in any of the mechanisms listed above or in additional mechanisms in the data store, as appropriate. The data store 1303, through logic associated therewith, may be operable to receive instructions from the server system 1300 and, in response thereto, retrieve, update, or process data.

[0117] The memory 1312 may also include a search service 1305. The search service 1305 may be executable by the processor 1301 to implement one or more of the functions of the server system 1300. In one implementation, the search service 1305 may represent instructions embodied in one or more software programs stored in the memory 1312. In another implementation, the search service 1305 may represent hardware, software instructions, or a combination thereof. The search service 1305 may perform some or all of the implementations described herein, alone or in combination with other devices, such as the user device 1100.

[0118] The server system 1300, in one implementation, is a distributed environment utilizing multiple computer systems and components interconnected via communication links using one or more computer networks or direct connections. However, one skilled in the art will appreciate that such a system may operate equally well in a system having fewer or more components than depicted in Figure 13. Thus, the depiction in Figure 13 should be construed as being exemplary in nature and not limiting of the scope of the present disclosure.

[0119] Implementations disclosed herein may include a computing system having one or more processors and a memory storing program instructions. The program instructions, when executed by the one or more processors, can cause the one or more processors to receive at least a text query from a user device and determine and return a plurality of results corresponding to the text query. Following receipt of the text query, the program instructions, when executed by the one or more processors, cause the one or more processors to receive an image of an object from the user device, generate an object feature vector representing the object, compare the object feature vector with the plurality of stored feature vectors corresponding to segments of the image returned as the plurality of results, generate a ranked list of the plurality of results based at least in part on the comparison of the object feature vector to the plurality of stored images, and present the ranked list in response to receiving the images.

[0120] Each of the plurality of stored image segments may correspond to less than an entire image. The image may be at least one of an image generated by a camera of the user device, an image obtained from a memory of the user device, an image obtained from the plurality of results, or an image obtained from a storage medium remote from the user device. The program instructions further cause the one or more processors to at least determine that the text query corresponds to a defined category, and to store the image in response to determining that the text query corresponds to a defined category. Filter by The defined categories may be at least one of fashion, clothing, home decor, personal, or food.

[0121] Implementations disclosed herein may include computer-implemented methods that may include one or more of receiving a query from a user device, determining a first plurality of images based at least in part on the query, receiving an image of the object after receiving the query, comparing the image of the object to at least one image segment of each of the first plurality of images, determining a ranked list of at least a portion of the first plurality of images based at least in part on the comparison, and providing a representation of at least a portion of the first plurality of images in the ranked list.

[0122] Optionally, the computer-implemented method also includes processing the images to determine an object type of an object represented in the images, and comparing the images includes generating an object feature vector representing the object and comparing the object feature vector to stored feature vectors corresponding to objects represented in the first plurality of images having the same object type. The computer-implemented method further includes determining that the query corresponds to a predefined category and determining a visual Narrow down your searchand presenting an option to the user device to select an object type corresponding to the object. The computer-implemented method may also include detecting an object within a field of view of a camera of the user device, determining an object type corresponding to the object at the user device, and displaying an object type identifier on a display of the user device. The object type identifier may include at least one of a graphical representation corresponding to a shape of the object type, or a name of the object type. The computer-implemented method may also include enabling selection of a second object type identifier, thereby indicating a second object type corresponding to the object. The computer-implemented method may also include generating a keyword corresponding to the type of object and including the keyword as part of the query. The computer-implemented method may also include determining a plurality of keywords corresponding to at least one of a portion of the first plurality of images, the first plurality of images, or the query, and providing a user device, presentation and selection of each of the plurality of keywords. Comparing the images of the object may further include one or more of generating an object feature vector representing the object, determining an expected location of the object in the first plurality of images, determining an image segment based at least in part on the object type, and comparing the object feature vector to a stored feature vector associated with the image segment. The computer-implemented method may further include determining an object type of the object, wherein determining the expected location is based at least in part on the object type.

[0123] Implementations disclosed herein may include a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor of a computing system, cause the computing system to receive a query at least at a user device, determine that the query corresponds to a defined category, and display a visual of the user device. Narrow down your search Enable the option and receive visual Narrow down your searchAs part of the option, streaming video from the user device, processing at least a portion of the streaming video to identify an object type of one or more objects represented in the streaming video present on a display of the user device, receiving, concurrently with the presentation of the streaming video, an object type of the one or more objects, a selection of the object type, determining a plurality of stored images corresponding to both the query and the selected object type, and displaying the plurality of stored images on a display of the user device.

[0124] The defined category may be food, and the streaming video may include representations of food currently within a field of view of a camera of the user device. The query may be a text-based query. The instructions may further cause the computing system to at least determine candidate images to be considered in determining the plurality of stored images based at least in part on the defined category. The instructions for causing the at least one processor to determine the plurality of stored images may cause the computing system to at least determine at least one keyword corresponding to the query, at least one object type, or the defined category, and to determine the plurality of stored images based at least in part on the at least one keyword.

[0125] Implementations disclosed herein may include a computing system. The computing system may include an image data storage device capable of storing one or more of a first plurality of stored images having a plurality of image segments and / or image information corresponding to each image. The image information may indicate, for each image, one or more of a respective plurality of image segments, each image segment may represent a portion of a respective stored image that is less than the entire stored image and a plurality of stored feature vectors, each stored feature vector corresponding to an object represented in the image segment of the plurality of image segments. The computing system may also include one or more processors and a memory storing program instructions. The program instructions, when executed by the one or more processors, may cause the one or more processors to receive from a user device, an image as part of a visually based search, process the image, an interest represented in the image, generate an object feature vector representing an object of interest, compare the object feature vector to the plurality of stored feature vectors to determine a second plurality of stored feature vectors including representations of objects visually similar to the object of interest, and determine a ranked list representing a second plurality of images of the first plurality of stored images based at least in part on the comparison of the object feature vector to the plurality of stored feature vectors. Based at least in part on the image comparison, each image of the second plurality of images is sent to a display of the user device such that an entire image of each image of the second plurality of images is included for each image of the second plurality of images, the entire image of each image of the second plurality of images including at least one image segment that includes a representation of an object determined to be visually similar to the object of interest.

[0126] The program instructions for processing an image to determine an object of interest, when executed, may cause one or more processors to process the image to determine a first candidate object of interest and a second candidate object of interest, and to receive an input indicating the first candidate object of interest as the object of interest. The image may include representations of multiple objects, and the object of interest may be determined based at least in part on a location of the object of interest within the image, a portion of the image that is in focus, a size of the object of interest represented in the image, or a color of the object of interest compared to a background color. Optionally, the program instructions further cause the one or more processors to at least process the images to determine a second object represented in the images, generate a second object feature vector representing the second object, and determine a third plurality of stored feature vectors including representations of objects visually similar to the second object among the stored feature vectors corresponding to the plurality of respective image segments, the ranked list further comprising: an object feature vector having the plurality of stored feature vectors determined at least in part based on the comparison; comparing the second object feature vector to the plurality of stored feature vectors to identify a second plurality of images of the first plurality of stored images, the at least one image segment including a representation of an object visually similar to the object of interest, and the at least one second image segment including a representation of an object visually similar to the second object. The program instructions, when executed, may cause the one or more processors to at least receive a selection of the object of interest and the second object from at least the user device.

[0127] Implementations disclosed herein may include computer-implemented methods that may include one or more of receiving an indication of an image from a user device, processing the image to determine a first object represented in the image, generating an object feature vector representing the first object, comparing the object feature vector to a plurality of stored feature vectors, each of the plurality of stored feature vectors representing a respective image segment of the first plurality of images, each image segment being less than all of the respective images, generating a ranked list of a second plurality of images from the first plurality of images, each image of the second plurality of images including at least one respective image segment determined to include a representation of an object based at least in part on the comparison and visually similar to the first object, and presenting the plurality of images by the user device.

[0128] Optionally, the computer-implemented method may include, prior to receiving the indication of the image, segmenting the second image into a plurality of segments, and for each of the plurality of segments, associating each of the feature vectors with at least one of the image segments or the second image to which the feature vector corresponds, and storing the second image and each respective feature vector in a data store, the respective feature vector being included in the plurality of stored feature vectors. The computer-implemented method may further include one or more of: storing, for each of the plurality of segments, location information indicating a location of the respective segment within the second image. The computer-implemented method may include, for each of the plurality of image segments, one or more of: determining an object represented in the image segment, generating a label corresponding to the object, and the feature vector representing the object. Optionally, the label may indicate at least one of a type of object or a category of object. Optionally, the computer-implemented method may further include one or more of: determining a label of a first object represented in the image, and determining the plurality of stored feature vectors based at least in part on the label of the first object. Optionally, processing the image may include one or more of processing the image to determine a plurality of candidate objects, receiving a selection of a first candidate object, and the first candidate object being the first object. Optionally, the computer-implemented method may further include one or more of receiving a selection of a second candidate object, generating a second object feature vector representing the second candidate object, and comparing the second object feature vector to at least a portion of the plurality of stored feature vectors. Generating a ranked list of the second plurality of images from the first plurality of images is based at least in part on comparing the second object feature vector to at least a portion of the plurality of stored feature vectors. Optionally, the computer-implemented method may further include one or more of determining an object type of the first object, and determining the plurality of stored feature vectors based at least in part on the object type.Optionally, the multiple stored feature vectors may have an object type that is the same as the object type of the first object.

[0129] Implementations disclosed herein may include a non-transitory computer-readable storage medium storing instructions. The instructions, when executed by at least one processor of a computing system, may cause the computing system to maintain image information corresponding to a plurality of images in a data store. The image information may indicate, for each image, one or more of a respective plurality of image segments, each image segment representing a portion of a respective image, a respective plurality of feature vectors, each feature vector representing an object in a respective image segment, and a plurality of labels corresponding to the respective image segment. The instructions further cause the computing system to determine an object represented in the image, generate an object feature vector representing the object, determine a label for the object, determine a similarity score based at least in part on the label, the plurality of feature vectors, compare the object feature vector to each of the plurality of feature vectors, each similarity score representing a similarity between the object feature vector and a respective feature vector of the plurality of feature vectors, and generate a ranked list of the stored images based at least in part on the similarity scores.

[0130] Optionally, the label may indicate at least one of an object or an object type of the object. Optionally, each of the plurality of feature vectors may represent an image segment of a respective stored image that is smaller than the entire image, and the instructions may further cause the computing system to present at least the images shown in the ranked list of storage, each presented image including a respective image segment that is smaller than the entire image. Optionally, each stored feature vector may represent an object of the image segment that is smaller than the entire image. Optionally, the similarity score may represent a Euclidean distance between the feature vector and the stored feature vector.

[0131] Implementations disclosed herein may include a computing system. The computing system may include one or more processors and a memory storing program instructions. The program instructions, when executed by the one or more processors, cause the one or more processors to at least receive an image of an object from a user device, process the image to determine an object represented in the image, the object corresponding to a predefined category, determine a first query type and a second query type based at least in part on the object or the predefined category, generate keywords corresponding to the object for use in the first query type, generate a feature vector representing the object for use in the second query type, determine a second result for the second query type based at least in part on the keyword, the first query type based at least in part on the keyword, generate a feature vector representing the object for use in the second query type, mix the first results to generate a mixed result including a first percentage of the first results and a second percentage of the second results, and / or generate an image of the object.

[0132] Optionally, the program instructions may further cause the one or more processors to determine a result ratio of the mixed results indicative of the first ratio and the second ratio. Optionally, the result ratio may be based at least in part on the object, the defined category, the first query type, the second query type, a user submitting the image of the object, a user device, or a user setting. Optionally, keywords may be generated based at least in part on labels associated with the object, the defined category, or the image included in the second result. Optionally, the first result may include content corresponding to an item utilizing or including the object, and the second result may include content including a representation of the object.

[0133] Implementations disclosed herein may include computer-implemented methods that may include one or more of receiving an image of an object from a user device, determining that the object corresponds to a defined category, determining a first query type and a second query type based at least in part on the defined category or object, obtaining a first query result of the first query type corresponding to the object, obtaining a second query result of the second query type corresponding to the object, at least a first portion of the first query result and at least a second portion of the second query result generating a blended result, and transmitting the blended result in response to the image of the object for presentation by the user device.

[0134] Optionally, the defined category may be at least one of food, home decor, or fashion. Optionally, the computer-implemented method may further include one or more of processing the image to determine an object type of an object represented in the image, the object being determined to correspond to the defined category based at least in part on the object type. Optionally, the first query type is a text-based query and the second query type is an image-based query. Optionally, the computer-implemented method may further include determining a third query type associated with the defined category, obtaining results of the third query corresponding to the object, and at least one object identifier matching the third query type. Optionally, the first query type returns content related to the object, and the second query type returns content including a representation of an object of the same object type as the object. Optionally, the computer-implemented method may include one or more of generating keywords based at least in part on the object or object type of the object, and obtaining the first query result may include determining the first query result based at least in part on the keyword. Optionally, the computer-implemented method may further include one or more of determining a result ratio indicating a first proportion of the first query results and a second proportion of the second query results to include in the blended result, the blending being based at least in part on the result ratio. Optionally, the computer-implemented method may further include one or more of determining an object type for the object and determining a plurality of stored feature vectors based at least in part on the object type. Optionally, obtaining the first query result may be based at least in part on keywords determined for the object, the keywords textually describing the object, and obtaining the second query result may be based at least in part on feature vectors generated from a representation of the object, the feature vectors visually describing the object.

[0135] Implementations disclosed herein may include a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor of a computing system, cause the computing system to at least determine an object represented in the image, determine that the object corresponds to a defined category, determine a first query type and a second query type associated with the defined category, generate keywords corresponding to the object for use in the first query type, generate a feature vector representing the object, obtain first results for the first query type based at least in part on the keywords, obtain results for the second query type based at least in part on the second obtained feature vector, and transmit instructions to present at least one result from the first results and at least one result from the second results.

[0136] Optionally, the instructions further cause the computing system to segment the image into at least a plurality of image segments and process each of the plurality of image segments to determine an object represented in the image. Optionally, the first result may be obtained based at least in part on a comparison of the keywords to labels associated with the stored images. Optionally, the second result may be obtained based at least in part on a comparison of the feature vector to a stored feature vector representing the object represented in the stored images. Optionally, the first result includes content that describes the object and the second result includes content that is visually similar to the object.

[0137] The concepts disclosed herein may be applied in a number of different devices and computer systems including, for example, general purpose computing systems and distributed computing environments.

[0138] The above-described aspects of the present disclosure are intended to be illustrative. They have been selected to illustrate the principles and applications of the disclosure, and are not intended to be exhaustive or to limit the disclosure. Many modifications and variations of the disclosed aspects may be apparent to those skilled in the art. Those skilled in the art should recognize that the components and process steps described herein may be interchangeable with other components or steps, or combinations of components or steps, and still achieve the benefits and advantages of the present disclosure. Moreover, it should be apparent to those skilled in the art that the present disclosure may be practiced without some or all of the specific details and steps disclosed herein.

[0139] Aspects of the disclosed system may be implemented as a computer method or as an article of manufacture, such as a memory device or a non-transitory computer-readable storage medium. The computer-readable storage medium may be readable by a computer and may contain instructions for causing a computer or other device to perform the processes described in this disclosure. The computer-readable storage medium may be implemented by volatile computer memory, non-volatile computer memory, hard drives, solid-state memory, flash drives, removable disks, and / or other media. Additionally, one or more of the modules and components of the engine may be implemented in firmware or hardware.

[0140] Unless otherwise noted, letters such as "a" and "an" should generally be construed to include one or more of the described items. Thus, a phrase such as "an apparatus configured to" is intended to include one or more of the enumerated apparatus. Such one or more enumerated apparatuses may also be collectively configured to perform the stated enumeration. For example, "a processor configured to perform enumerations A, B, and C" may include a first processor configured to perform enumeration A working in conjunction with a second processor configured to perform enumerations B and C.

[0141] As used herein, language of degree, such as "about," "approximately," "generally," "approximately," "similar," or "substantially," refers to a value, amount, or characteristic that performs a desired function or achieves a desired result, representing a value, amount, or characteristic that is close to the stated value, amount, or characteristic. For example, the terms "about," "approximately," "generally," "approximately," "similar," or "substantially" may represent an amount that is less than 10%, less than 5%, less than 1%, less than 0.1%, or less than 0.01% of the stated amount.

[0142] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. 1. A computing system comprising: one or more processors; When executed by the one or more processors, the one or more processors are caused to Receiving a text query; Present visual refinement options; and determining and returning a plurality of results corresponding to the text query; receiving an image including a representation of an object as part of the visual refinement options; generating an object feature vector representing the object; comparing the object feature vector with a plurality of stored feature vectors corresponding to segments of visual items included in the plurality of results to determine a similarity score representing a degree of similarity between the object feature vector and each one of the plurality of stored feature vectors; generating a ranked list of the plurality of results based on the respective similarity scores; presenting some of the ranked lists in response to receiving the objects; a memory for storing program instructions for causing A computing system comprising:

2. The program instructions cause the one or more processors to at least: identifying a plurality of items representing the object; determining an item of interest from among the plurality of items; generating a feature vector for the item of interest as an object feature vector representing the object; The computing system of claim 1 , further comprising:

3. The program instructions cause the one or more processors to at least: determining a defined category corresponding to the text query; and determining a label for each of a plurality of items representing the object; identifying a first item of interest from among a plurality of items having said label associated with said defined category; The computing system of claim 2 , further comprising:

4. The program instructions cause the one or more processors to at least: For an item for which the label is associated with the defined category, presenting the label as a user-selectable control adjacent to a corresponding item represented on the object; receiving a selection of a first label; identifying an item corresponding to the first label as the first item; The computing system of claim 3 , further comprising:

5. The program instructions cause the one or more processors to at least: identifying a plurality of said segments of said object, each segment being smaller than the entire object; identifying a plurality of items represented in the object from the items represented in the plurality of segments; The computing system of claim 2 , further comprising:

6. 6. The computing system of claim 5, wherein the object is one of video content or image content, and the items represented in the object are identified from items represented in segments of the object according to an edge detection algorithm.

7. and for each segment of the plurality of segments of the object, identifying a context for the segment; removing the background of the segment; The computing system of claim 5 , further comprising processing a plurality of the segments from which the background has been removed to identify a plurality of items represented in the object.

8. 1. A computer-implemented method, comprising: Under control of one or more processors executing instructions on a computer system, in response to presentation of a first image on a display of said computer system: It provides visual refinement options, which are determining a first object from among a plurality of objects represented in the first image; generating a first feature vector representing the first object; comparing the first feature vector to a plurality of stored feature vectors, each stored feature vector of the plurality of stored feature vectors corresponding to an image held in a data store, each comparison resulting in a similarity score between the first feature vector and one stored feature vector of the plurality of stored feature vectors; selecting a feature vector result having the highest similarity score from among the plurality of stored feature vectors; identifying an image result corresponding to one of the feature vector results; presenting at least some of the image results on a display of the computer system; and A computer-implemented method comprising:

9. Segmenting the first image into a plurality of segments; identifying a plurality of objects from a plurality of segments of the first image; The computer-implemented method of claim 8 , further comprising:

10. For each segment of the plurality of segments, identifying a context for the segment; removing the background of the segment; The computer-implemented method of claim 9 , further comprising identifying a plurality of objects from the plurality of segments from which the background has been removed.

11. 9. The computer-implemented method of claim 8, wherein the first object is determined from a plurality of the objects represented in the first image according to a foreground and center location of the first object in the first image.

12. Determining the first object from among a plurality of the objects includes: presenting on a display user selectable segments corresponding to some of the objects; receiving an indication of a selection of a first user-selectable segment corresponding to a selected one of the objects; Identifying the selected object as the first object; The computer-implemented method of claim 8 , further comprising:

13. determining a label for each of the number of said objects; The computer-implemented method of claim 12 , further comprising presenting the label of the object to which the user-selectable segment corresponds.

14. 1. A computer-implemented method, comprising: Under the control of one or more processors executing instructions on a computer system: receiving a text query for an item of content maintained in a data store by the computer system; receiving an image as a visual refinement of the text query; processing the image to detect a plurality of objects depicted in the image; determining a text label for each of a plurality of said objects; presenting user-selectable segments corresponding to text labels of certain objects of the plurality of objects, each segment being presented proximate to a corresponding object of the plurality of objects and concurrently with the image on a display of the computer system; receiving a selection of a first segment corresponding to a first text label of a first object of the number of said objects; generating an object feature vector representing the first object; identifying a set of items of content from the items of image according to the object feature vector, where identifying the set of items of content includes comparing the object feature vector of the first object with object feature vectors associated with items of content to generate a similarity score representing a similarity between the object feature vector of the first object and each object of the items of content corresponding to the item of content; presenting the set of items of content on a display of a computer system; A computer-implemented method comprising:

15. determining a defined category corresponding to the text query; and identifying a subset of the items of content, wherein a text label of each object in the subset of objects is associated with a defined category; The computer-implemented method of claim 14 , wherein identifying the subset of items of content is performed from a subset of objects.

16. presenting a text query in an input field on the display of the computer system simultaneously with presenting the number of items of content, and further displaying an image representing a first object in the input field together with the text query; The computer-implemented method of claim 14 further comprising:

17. processing the image into a plurality of segments; processing the plurality of segments to identify a plurality of objects depicted in the image; The computer-implemented method of claim 14 further comprising:

18. For each segment of the plurality of segments, identifying a context for the segment; removing the background of the segment; processing the segments from which the background has been removed to identify a number of objects depicted in the image; 20. The computer-implemented method of claim 17, comprising:

19. 20. The computer-implemented method of claim 17, wherein the segments are processed according to an edge detection algorithm to identify the objects represented in the image.

Images