System and method for attribute-based visual search on computer communication networks

Abstract

A visual search system includes a computing device, which includes: an image processing engine for generating a feature vector representing a user-selected object in an image input; an object detection engine for locating one or more objects in the image input and for determining a category of the user-selected object from the objects in the image input, the object detection engine using the category to generate multiple attributes for the user-selected object; a product data store for storing multiple tables, the multiple tables storing one or more attributes associated with the category of the user-selected object; an attribute generation engine for generating multiple attribute options for each attribute of the user-selected object; and an attribute matching engine for comparing the attributes and attribute options of the user-selected object with the attributes and attribute options of visually similar products and images.

Description

Technical Field

[0001] The present disclosure relates generally to the fields of computing and communications, and particularly, but not exclusively, to systems and methods for performing visual searches over a computer communications network using user-specified attributes of physical objects. Summary of the invention

[0002] A visual search system and method are described that use image input received from a user of a mobile or other computing device as a search query to generate image search results for visually similar products and images. The image input may include one or more objects of interest to the user that are specifically specified or identified by the user in the image. When the user selects an object in the image, the visual search system and method determine a category for the object and then determine attributes associated with the object in the category of the selected object. The combination of the category and the determined attributes enables the system and method to determine and display user-selectable options for each of the attributes associated with the user-selected object. The search query, categories, attributes, and related attribute options are used in the visual search system and method to generate a list of products and images that are visually similar to the object selected by the user.

[0003] In determining attribute options for display to a user, the system and method identifies the options and displays them in an order of preference of the user, as determined from monitoring of user selections of objects in a category, or in an order of relative visual similarity to an object selected by the user. The attribute options are displayed within a user interface and, when selected, can enable searching, identification, and retrieval of products and images that are visually similar to the object selected by the user, including search queries.

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Background Art

[0005] The use of the Internet to search and retrieve vast amounts of information continues to grow, creating a need for enhanced capabilities to search, view, evaluate, and display information in a variety of ways and formats. Visual search capabilities are increasingly important, and new capabilities are emerging almost every day.

[0006] However, the array of current visual search services generally perform similar functions. The functions performed by these devices include processing images from online and offline resources using resources such as digital image capture software and embedded cameras in mobile devices, respectively. These images can then be processed by these visual search services and used to locate products that are similar in structure or function to the products shown in the images. Some contemporary visual search services can also be used to help users determine where to purchase the products shown in these images, or even find products that are similar in style or color to the products shown in the received images.

[0007] Despite the significance of these significant advances, contemporary visual search services offer little or no ability to dynamically adapt or adjust attributes associated with products of interest shown in received images prior to performing a search for those products. Visual search services also lack the ability to provide users with the ability to customize attributes associated with products prior to performing a search to find and display products having similar user-specified attributes.

[0008] Therefore, there is a large and rapidly growing need for intelligent attribute-based visual search systems and methods that will enable users to dynamically interact with visual search services and to interactively adjust attributes of products of interest in images used by the visual search services. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Non-limiting and non-exhaustive embodiments are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0010] Figure 1 is a diagram of an operating environment for a visual search system in one embodiment.

[0011] Figure 2 is an illustration of a visual search performed on an interface of a desktop computer, in one embodiment.

[0012] Figure 3 is an illustration of a visual search performed on an interface of a mobile device, in one embodiment.

[0013] Figure 4A is a flow chart illustrating one embodiment of a method for performing attribute-based visual search.

[0014] Figure 4B is a flow chart illustrating one embodiment of a method for attribute-based visual search result matching.

[0015] Figure 4C is a flow chart illustrating one embodiment of a method for ranking attribute-based visual search results.

[0016] Figure 5 An exemplary computing environment is shown. DETAILED DESCRIPTION

[0017] The following example embodiments relate to a search method that can increase the relevance of returned visual search results. The examples described below allow a user to specify attributes of an object of interest in a received image (such as an image captured by a camera of a mobile device) when interacting with a visual search service before executing a custom visual search query that is generated in part from the attributes specified by the user. In this way, the attributes specified by the user can be used to guide the visual search.

[0018] The technical effects of the embodiments described below relate to determining products that are visually similar to a user-selected object in a query image from an attribute-based visual search process. These embodiments result in visual-based searches that are more efficient, more accurate, and provide more relevant information to users.

[0019] As described in more detail below, an example computing system generates a visual search query based on objects in a received image and attributes specified by a user. The system analyzes the image using one or more object detection methods to classify and categorize objects included in the image. The system retrieves a default pre-stored attribute set associated with a category of a selected object, generates an attribute set for the selected object, and then generates a ranked ordered list of attribute options to enable a user to customize the options associated with the selected object for the purpose of performing a visual search for visually similar products. The attribute options shown to the user are initially determined from the pre-stored attributes associated with objects of the selected category, but can be modified based on the customized attribute options generated from the system. Upon the user selecting the attribute options, the system can perform a visual search using a representation of the input query image, the customized attribute options, and a multi-layer learning-to-rank computational framework, from which one or more ranked results of products and images related to the query image are generated.

[0020] As a preliminary matter, some drawings describe concepts in the context of one or more structural components, which are referred to as functionality, modules, features, elements, etc. in different ways. The various components shown in the figures can be implemented in any way, such as software, hardware, firmware or a combination thereof. In some cases, the various components shown in the figures can reflect the use of corresponding components in the actual implementation. In other cases, any single component shown in the figures can be implemented by multiple actual components. The depiction of any two or more separated components in the figure can reflect the different functions performed by a single actual component.

[0021] Other drawings have described concepts in the form of flow charts. In this form, some operations are described as constituting different frames executed in a certain order. Such implementation is exemplary and non-restrictive. Some frames described herein can be combined together and executed in a single operation, some frames can be decomposed into multiple composition frames, and some frames can be executed in an order different from the order shown in this article, including the parallel mode of executing frames. The frame shown in the flow chart can be implemented by software, hardware, firmware, manual processing, etc. As used in this article, hardware can include microprocessors, digital signal processors (DSPs), microcontrollers, computer systems, discrete logic components, and / or custom logic components, such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), programmable logic arrays (PLAs), etc.

[0022] Regarding the terminology, the phrase "configured to" encompasses any manner of functionality that can be constructed to perform the identified operation of any kind. The function can be configured to perform the operation using, for example, software, hardware, firmware, etc. For example, the phrase "configured to" can refer to the logical circuit structure of a hardware element arranged to implement the associated function. The phrase "configured to" can also refer to the logical circuit structure of a hardware element arranged to implement the coding design of the associated function of firmware or software. The term "module" refers to a structural element that can be implemented using any suitable hardware (e.g., processor, etc.), software (e.g., application, etc.), firmware, and / or any combination of hardware, software, and firmware. The term "logic" encompasses any function for performing a task. For example, each operation shown in the flowchart corresponds to the logic for performing the operation. The operation can be performed using software, hardware, firmware, etc. The terms "component", "system", etc. can refer to computer-related entities, hardware, and software, firmware, or a combination thereof in execution. A component can be a process running on a processor, an object, an executable file, a program, a function, a subroutine, a computer, or a combination of software and hardware. The term "processor" can refer to a hardware component, such as a processing unit of a computer system.

[0023] In addition, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard programming and engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computing device to implement the disclosed subject matter. The term "article of manufacture" as used herein is intended to encompass a computer program accessible from any non-transitory computer-readable storage device or medium. Computer-readable storage media may include, but are not limited to, magnetic storage devices such as hard disks, floppy disks, magnetic strips, optical disks, compact disks (CDs), digital versatile disks (DVDs), smart cards, flash memory devices, and the like. In contrast, computer-readable media (i.e., not storage media) may additionally include communication media, such as transmission media for wireless signals and the like.

[0024] Figure 1 is a diagram of an exemplary environment 100 for an attribute-based visual search system. The environment may include one or more clients 110, 112, 114. Each client 110, 112, 114 may be configured to communicate with a visual search system 130 via a computer communication network 120. The network 120 may be a variety of network types, including a public switched telephone network (PSTN), a cellular telephone network, and a packet-switched network (e.g., the Internet).

[0025] In some implementations, each client 110, 112, 114 may include a desktop personal computer, workstation, laptop, smart phone, PDA, cellular phone, or any WAP-enabled device or any other computing device capable of directly or indirectly interfacing with network 120, such as Figure 5 The computing device 500 shown. Each client 110, 112, 114 can run an HTTP client, such as a browsing program, such as MICROSOFT INTERNET EXPLORER or other browser, or a WAP-enabled browser in the case of a cell phone, PDA, or other wireless device, allowing the user of the client 110, 112, 114 to access information available to them at the visual search system 130 or to provide information to the visual search system 130. For example, other applications can be used by any of the clients 110, 112, 114 to access the visual search system 130 or to provide information to the visual search system 130. In some implementations, the visual search system 130 can use one or more general-purpose computing systems (such as Figure 5 In some other implementations, the visual search system 130 can be implemented using a dedicated computing system (such as a graphics processing unit or a tensor processing unit).

[0026] In some implementations, the visual search system 130 may receive a query image from a user of the client 110, 112, 114, and search a corpus of product database images 190 (or any corpus of searchable images) to find a matching image for the query image. The matching image may then be presented or identified to the user who submitted the query image. The matching image may include one or more images from one or more product databases 190. The product database 190 may store images received or collected from one or more users, or images displayed or presented on a website from a commercial supplier of the product. For example, the product database 190 may store images collected from the Internet, such as images from social media sites, e-commerce stores, or other commercial seller websites. Although the product database 190 is shown as being directly connected to the visual search system 130, it is contemplated that the product database 190 may be connected to the visual search system 130 via the network 120, or may be directly integrated into the visual search system 130.

[0027] The visual search system 130 processes images from search queries to generate image data. The visual search system 130 may include an image processing engine 140 to process the images into visual words generated from visual features detected in various regions of the images in the received search query, such as edges, corners, and spots in the images. Given their informative content, the visual words may be used to represent the images as, for example, a histogram analysis of the number of times the visual words (e.g., relevant visual features) occur in the images. The image processing engine 140 may include an object detection engine 142 for performing object localization and object category classification and for determining and generating attributes of objects represented in the images of the search query. An attribute generation engine 144 is also provided that continuously monitors user activity to learn patterns that reveal user preferences for various types of objects. The attribute generation engine 144 generates attribute options based on this continuous monitoring, a default set of pre-stored attributes from a product data store 192, and attributes generated and received from the object detection engine 142, where category and attribute associations are stored. A ranked ordered list of attribute options is then generated for each attribute by the attribute generation engine 144 based on a degree of visual similarity or likeness to the user-selected object in the query image, or based on user preferences, which depend on the frequency of occurrence of certain attribute options for attributes associated with frequently viewed, copied, or stored objects. After generation, the ranked ordered list of attribute options can be presented on a user interface to enable the user to perform a highly relevant, low-latency search for products and images related to the user-selected object in the query image.

[0028] In some implementations, the visual search system 130 may also include a similarity engine 150 that takes a query image and compares it to a database image retrieved from the product database 190 to assess the level of similarity between the two. The visual search system 130 may also include a result ranking engine 160 that generates a ranked ordered list of attribute matching search results based on the calculated matching criteria. The visual search system 130 may also include an attribute matching engine 170 that generates search results having attributes that match attributes of one or more user-selected objects in the image of the search query. For some implementations, the visual search system 130 may also include a learning engine 180 that characterizes the user-selected objects in the received search query and compares the objects to products and images in the product database 190 before the user-selected objects are categorized by the object detection engine 142.

[0029] Figure 2200 is an illustration of a user interface on a desktop computer for a visual search system with intelligent attributes. The user interface 200 shows several objects that have been detected by the object detection engine. Each detected object is marked by a hotspot (a box mark on each object in the image), and the user can click on the hotspot. Alternatively, as shown in this example, the user can specify a cropping frame 204 to select a product of interest, and to obtain visually similar products or images. After clicking on the hotspot or creating a cropped image frame around the object of interest, a first selection area 206 appears, such as a pop-up window shown in this example, showing several attributes and related attribute options, which can be selected by the user to perform the attribute-based visual search for the object. In an operational embodiment, the attribute options for each related attribute are presented in a ranked ordered list, so that the options determined to be consistent with the user's personal preferences are displayed before other attribute options. These user preferences for applicable options for the selected object are determined from the embedded attribute generation engine, which actively monitors user actions online to determine which options on the product or other object of interest have been selected with a higher frequency of occurrence. In an alternative embodiment, the attribute options are presented in the option selection area or window in a ranked order based on visual similarity to the object selected by the user, such that those attributes that more closely match the selected object appear first in the order, while those less similar attributes appear later in the ranked order. In the example shown, if the user's wallet selection is navy blue, several attribute options will appear in the selection area for the color attribute, such as Byzantine Blue, Light Blue, Pale Blue, Tar Heel Blue, Teal, Blue-Gray, etc. Once the appropriate attribute option is selected, the user can begin the visual search by clicking the search button 208, thereby displaying visually similar related images and related products in the second area of ​​the user interface as shown. Although Figure 2 The embodiment depicted in the drawings shows six attribute options for the user to select, but in practice, a pop-up window or other display area may display any number of attributes and associated attribute options for viewing and selection by the user after the object of interest is selected as determined by clicking on a hotspot or cropping of the object of interest in an image.

[0030] Figure 3300 is an illustration of a user interface on a mobile device. This embodiment of a user interface 300 on a mobile device includes a cropped image 302 of a selected object, which generates a first area of ​​customization options (e.g., a pop-up window) when selected, in which a number of customization options 303 (not shown) related to the properties of the selected object and a search button 305 (not shown) are depicted. The options shown in this area appear in a ranked ordered list and can be displayed based on learned user preferences or based on relative visual similarity to the selected object, which is determined from a visual analysis of the object and its unique properties (e.g., size, color, style, etc.). After clicking the search button 305, a series of related products 304 and related images 306 are displayed in a second area 308 below the location of the cropped frame of the selected object. In the illustrated embodiment, the second area 308 collectively includes the area in which the related products 304 and related images 306 appear. As known to those skilled in the art, other regions or zones can be used to depict such products and information, such as pop-ups or other movable areas. It should be noted that the image may be obtained from a mobile device, a repository of photos stored on a hard drive, copied from a website or social media site (e.g., Pinterest, Facebook, etc.) from any other location on the Internet or other network, or simply by taking a photo from the mobile device's embedded camera. Regardless of the source of the original received image, the user can view the image and select objects within the image to help identify related products and images available online.

[0031] Figure 4A4 is a flow chart illustrating an embodiment of a method performed for attribute-based visual search. When an image (e.g., a stored photo, an image obtained from an online site (e.g., Instagram, Pinterest, Twitter, etc.) is received on a device (e.g., a mobile phone, a personal computer, a laptop computer, etc.), preliminary image processing is performed to analyze and characterize the content of the image, as shown in step 402. After characterizing the content in the image, an object detection process is performed, as shown in step 404, to identify objects in the image and their locations in the image, including objects in images with limited clarity or visual acuity, such as truncated or occluded objects. The object detection process performs multiple related processes, including object localization and object category classification (i.e., object categorization), to fully characterize the image and the objects shown therein. In addition, the object detection process performed in step 404 detects unique characteristics and qualities of objects appearing in the received image, including detecting and determining physical and visual attributes, such as object color, object shape, object length, object type, etc. When determining the object type, the object detection process uses the object category classification to determine unique attributes of the detected object (e.g., an "apparel" object category enables the object type attribute to be one of a set including casual, cocktail, formal, wedding, etc.) Illustrative examples of other types of category-specific attributes applied in the fashion and sportswear domains are listed below.

[0032] For a piece of clothing, the following are representative examples of attributes:

[0033] Attribute_Type: Casual, Cocktail, Formal, Wedding

[0034] Attributes_Style: Fit, Dress, Maxi, Straight

[0035] Attributes_Hem: Short, Knee-Length, Medium, Long, High-Low

[0036] Attributes_Neckline: Boat neck, Collar, Halter, Turtleneck, Off-shoulder

[0037] Attributes_Sleeve Type: Strapless, Sleeveless, Short Sleeve, 3 / 4 Sleeve, Long Sleeve

[0038] Attributes_Pattern: Animal print, Plants, Geometric, Graphic, Cards, Plaid, Polka dots

[0039] Patterns

[0040] Attributes_Size: Regular 0-20, Small 0-16, Plus 0-4X, Maternity 0-18

[0041] Attribute_Color: black, gray, white, brown, purple, blue, yellow, orange, red, etc.

[0042] For shoes, the following are representative examples of attributes:

[0043] Attribute_Type: Ankle boots, boots, flat shoes, high heels, sandals, slippers, sneakers

[0044] &Sports Shoes

[0045] Attributes_Occasion: Bridal, Casual, Dress, Party & Evening, Holiday, Wedding, etc.

[0046] Attributes_Heel height: flat, low, medium, high, super high

[0047] Attributes_Toe Style: Closed Toe, Open Toe, Peep Toe, Pointed Toe, Round Toe

[0048] Attributes_Width: 4A Super Thin, 3A Thin, 2A Narrow, B Medium, C Wide, E Super Wide

[0049] Attribute_Size: 4, 4.5, 5, 5.5, 6, etc.

[0050] Attribute_Color: black, gray, white, brown, purple, blue, yellow, orange, red, etc.

[0051] Object localization is the process of determining the specific location of an object in a processed image. In an image with multiple objects, the object localization process identifies, locates, and sets a bounding box or other digital marker on the identified object, which is used to identify and establish the location of the object relative to other objects in the image. Object category classification is a process for determining the category of an object, which will be referred to as object categorization. For example, if an image depicts a glass of wine, a cat, and a gift box, the object localization process can be used to identify and locate these three objects relative to each other within the setting of the entire image. At the same time, each object can be roughly classified, in this case it will be a gift box, an animal, and a drink. In addition to its role in determining the approximate category of an object, the object categorization process can also be used to determine a set of attributes for each detected object within the image. In alternative embodiments of the visual search system and method, one or more object detection algorithms can be applied to perform object localization and object category classification. Among the different types of algorithms used for object detection, there is a single shot multi-box ("SSD") detection algorithm. SSD is a single deep neural network that can establish a set of default bounding boxes in image space and scale according to each feature map position. The SSD algorithm generates a score representing the presence of each object in the image, and adjusts the applicable bounding box to achieve a better match with the object shape. The method combines predictions with multiple feature maps to handle objects of different sizes, including truncated objects or objects with limited image resolution. Alternatively, the Faster R-CNN algorithm or faster regional convolutional neural network has been applied to object localization and object categorization. The algorithm includes two networks: a region proposal network and an object detection network, which uses the proposal generated by the region proposal network. The region proposal network ranks the calculated regional bounding boxes for the image and proposes the regional bounding box that is most likely to contain the relevant object as the foreground bounding box. The various bounding boxes / region proposals generated by the region proposal network are processed by the foreground background classifier and the bounding box regressor to check the appearance of the object and its corresponding position (e.g., foreground, background, etc.) within the query image. The role of the region proposal network is to predict whether the bounding box in the image is the background or the foreground, and to refine the prediction to achieve higher accuracy.

[0052] Referring now to step 406, after analyzing and characterizing the image content using the object detection process, an object selection request may be detected based on a user's action, such as a click on a hotspot, or an object within a cropped image, or a selection made by a finger touch or a stylus click on an object depicted in an image shown on a mobile device user interface. Upon detecting a user object selection, a feature extraction process is performed on the selected object, thereby generating a query image. In one embodiment, the query image is an image created from the juxtaposition of a crop box onto the selected object shown in the received image (i.e., storing, copying, or retrieving a photo received from the user). In an alternative embodiment, the query image is generated from a hotspot that marks an object of interest in the received image. The feature extraction process determines unique visual attributes of the selected object including the query image, and generates a feature vector based on the query image. The feature vector is a digitized representation of visually relevant features (e.g., edges, contours, spots, etc.) in the query image. In addition to generating the feature vector, the process also retrieves object attributes associated with the category determined for the selected object, as shown in step 408, and then combines the generated attributes generated during the object detection process (shown in step 404) with the retrieved object attributes 408 to generate attribute options 409 for the combined attribute set. In one embodiment, the generated attribute options 409 are generated from a rule-based attribute option generation process, while in an alternative embodiment, the generated attribute options 409 are generated from a model-based attribute option generation process. As shown in step 408, the rule-based attribute option generation process uses the object category to determine which attributes to retrieve from one or more tables stored in a database, where an associative array or similar data structure is used to store the association between the category and the predefined attribute set. In addition, each attribute has an associated option set (e.g., attribute: dress; attribute options: cocktail, formal, etc.), and each of the attribute options generated for each attribute associated with the object in step 409 can be presented in a ranked list or other ranked ordered manner based on user preferences (determined from an analysis of the frequency of occurrence). A ranked ordered list of options for each attribute based on user preferences is created using Figure 1 The attribute generation engine 144 shown is determined. The attribute generation engine 144 continuously monitors user actions to discern patterns of selections for different types of objects and develops a user preference profile based on the frequency of occurrence of such selections. The user preference profile enables the attribute generation engine 144 to generate and display user selection preferences as options for each attribute of an object within a specific category. Monitoring such usage patterns makes the visual search or shopping experience more convenient and targeted to each user's preferences and / or interests.

[0053] Alternatively, and as previously described, the generated attribute options 409 may result from executing a model-based attribute option generation process. As shown in step 406, when a user selection of an object is detected, a model-based attribute option generation process is executed, and in a first embodiment, the selected object image needs to be processed on a deep neural network that is trained on multiple observations to discern and learn the relationship or similarity between the content, attribute set, and a series of attribute options of an image (e.g., a query image). In this first embodiment, the deep neural network may implement a LambdaMART ranking model for generating a ranking score that enables a ranked ordered list of attribute options to be generated based on visual similarity to the object selected in the query image. For example, a red dress shown in the query image and selected by the user will result in the generation of attribute options for an attribute, such as "color" in a ranked ordered list of decreasing visual similarity (e.g., light red, rose, pink, fuchsia, purple, etc.). Likewise, for the attribute of "dress hat," a ranked ordered list may be presented that presents options of decreasing visual similarity (eg, fedora, beret, beanie, baseball cap, etc.).

[0054] In an alternative embodiment, after detecting a user object selection 406, a deep neural network is used to process the attributes generated during the execution of the object detection process as shown in step 404 based on training data and a training model, the training model enabling the network to learn similarities between attributes and attribute options. The trained network is used to generate similarity scores for each attribute pair (i.e., initial attribute, attribute option), and to rank these pairs based on their respective similarity scores. This ranking of these pair-wise associations between attributes and attribute options is used to generate and display a ranked ordered list of attribute options based on visual similarity, such that the attribute options are presented in order of decreasing similarity or likeness to the object depicted in the query image for each available attribute displayed for the object.

[0055] Once an object in the query image is selected, a cropped image of the user's object selection will appear on the user interface, or, alternatively, the object will be marked with a hotspot icon, as shown in step 410. Once a cropped image for the user's object selection is generated, in one embodiment, a pop-up window appears that displays a list of object attributes associated with the user's selected object, as shown in step 412. The displayed list of object attributes provides customization options for all or a subset of the object's attributes, which users with different or specific preferences for the detected object can click on. As shown in step 416, before generating a modified list of object attributes, the system detects or waits for user input of the object attribute options selected by the user at decision point 414. The modified list of object attributes 416 includes: a combination of the object attributes generated from the object detection process (shown in step 404) and the retrieved attributes (shown in step 408), and a ranked ordered list of attribute options for each attribute in the combined set of generated attributes and retrieved attributes. As previously described, in one embodiment, the options for each attribute can be ranked in order according to user preferences based on an analysis of the frequency of occurrence, or in another embodiment, based on the relative visual similarity to the object in the query image. If the user attribute option is not selected, as shown in step 418, a feature vector representing the query image (i.e., the user selected object), a combined set of displayed object attributes (i.e., the generated attributes and the retrieved attributes), and related options (as shown in step 412) are used as inputs to perform a visual search, then generate the best attribute matching results (as shown in step 420), and display a ranked ordered list of the best attribute matching search results, as shown in step 422, the ranked ordered list being ranked from the most visually similar results to the least visually similar results relative to the user selected object in the query image.

[0056] As shown in step 418, the performance of the visual search involves processing a feature vector generated from a feature extraction process performed on a query image in a multi-layered learning-to-rank computational framework that enables the operation of a visual search engine. The visual search engine framework performs a systematic quantitative reduction of the information contained in the image representation in the feature vector, and includes multiple computational components, including a visual word quantization component at an initial matching level (i.e., L0 matching), a product quantization component at a secondary intermediate result ranking level (i.e., L1 ranking), and a final complete result ranking component (i.e., L2 ranking) for ranking identified search results that have properties that are visually similar to matching attributes or related attributes of a user-selected object represented in the feature vector. The framework involves converting the feature vector into a visual word quantized representation of the query image (i.e., a "visual word vector"), followed by a product quantization of the visual word vector. The product quantification vectors generated from the product quantification process are then used to perform a comprehensive search of results across one or more product databases, including publicly accessible databases available from commercial suppliers, academic centers, non-profit organizations, and trade organizations, and the search results are then ranked to produce a list of products and images that are similar or related to the query image. The source of the product images can be an image index of pre-stored images compiled from data sources across the Internet or other public or private computer networks and from one or more product or supplier databases (on which visual representations of products are provided in the form of images). Products and images that are visually similar to the (multiple) user-selected objects in the received image that have the same or visually similar attributes (e.g., sleeveless, red dress, etc.) are displayed in a ranked order, where exact match results are presented first in a relevance ranked order, followed by visually similar results with matching attribute options (e.g., selected object: yellow dress with paisley pattern; relevance ranked results: yellow dress with small stars, light orange dress with small circles, etc.).

[0057] In step 404, the categories identified from the category recognition process performed during object detection can be used as an index to access attributes associated with the category stored in a database (such as an inverted index database or a key-value database) that can quickly identify and retrieve attribute sets associated with products in a given category. As shown in step 420, the generation of the best attribute matching search results uses an attribute matching engine to quickly compare, analyze and match the text and other metadata of products and images related to the user-selected object in the query image. Ranking weights are assigned to the results based on the similarity between the attributes of the user-selected object present in the query image and the attributes of related products and product images in the same category. The database implements an inverted index data structure using an associative array that stores the association between categories and attributes, which can quickly identify and compare products with the same (or similar) attributes, and assign greater ranking weights to visually similar products and / or images with attributes that match or are semantically similar to the attributes of the user-selected object. The result ranking engine performs sorting on these best attribute matching search results and displays a ranked ordered list of search results to the user, as shown in step 422.

[0058] Figure 4B A process 430 for attribute matching in a visual search system is shown and is intended to extend the Figure 4A The process 430 begins with the retrieval of attributes for the user-selected object, as shown in step 432, including the compilation of attribute sets generated from the user-selected customization options for the retrieved attributes, and comparing them with third-party product attributes for products similar to the user-selected object shown in the received image. As shown in step 434, the comparison step may include comparing text or other metadata associated with the third-party product description or attributes with the attribute set of the user-selected object retrieved from the indexed image database. The customization options are determined and set by the user before performing the visual search to ensure that all search results retrieved match the product attributes required by the user as much as possible. During the comparison between the attributes specified for the user-selected object and the attributes of the third-party products, a process is performed to exclude products within the same category that have no matching attributes relative to the attribute set for the user-selected object, as shown in step 436. This exclusion process reduces the level of processing performed and helps to increase the overall speed of the process for revealing a set of products that are most likely to match a given attribute set associated with the user-selected object. After compilation, a narrowed or refined product list is generated based on the best attribute matches achieved, as shown in step 438 .

[0059] Figure 4CA process 440 for ranking results using weighted search results is shown. A list of products with matching attributes is retrieved from an attribute matching engine, as shown in step 442. The list of products is displayed as related products and related images adjacent to the object selected by the user on the user interface. These search results are generated after the user clicks on a hotspot centered on an object in the processed image or applies a cropping box on an object of interest in the processed image. As shown in step 444, the ranker is used to sort the retrieved list of products using the maximum number of matching attributes. Matching criteria based on one or more statistical optimization processes are applied to a learning to rank computational framework to identify visually similar products and / or images with the same or similar attributes as the object selected by the user. In one embodiment, the optimization process applies a K-means clustering technique to a corpus of index images while applying a deep neural network to a query image to generate a feature vector that represents the image as a set of visual words (i.e., a visual word quantized representation). The n nearest neighbor centroid statistical technique is applied to the clustered data and feature vector to generate a visual word quantized representation of the query image. The n-nearest neighbor centroid statistical analysis technique enables visually similar products and / or images to be established based on the distances between the statistical means of the centroids in the clusters of the data. Centroids having statistical means that are far away from the statistical mean of the centroids representing the query image are determined to be more different (or, alternatively, less statistically relevant), while centroids having statistical means that are close to the statistical mean of the centroids representing the query image are determined to be more similar (or more relevant), and therefore more visually similar to the query image.

[0060] The attribute set of the object selected by the user can be based on a set of attributes pre-stored for a given object category or based on a set of one or more customization options specified by the user for each attribute in the set of attributes in the pre-stored list. Then, the attributes are combined and compared with the attributes of the set of products retrieved in the product list. After the product list is generated using the product with the maximum number of matching attributes, each supplier website will be checked to determine which inventory availability is applied to each product on the sorted product list. As shown in step 446, this confirmation of product availability ensures that the best match of the product can be generated based on the visual similarity with the object selected by the user and the product availability in the supplier's inventory. After identifying the attribute matching product and confirming the product availability in the supplier's inventory, starting from the exact matching product, the products are sorted and displayed in the attribute matching ranking order, as shown in step 448, for the user to view.

[0061] Figure 5An exemplary computing environment in which example implementations and aspects may be implemented is shown. The computing system environment is merely one example of a suitable computing environment and is not intended to impose any limitations on the scope of use or functionality. Many other general or special computing system environments or configurations may be used. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use include, but are not limited to, personal computers (PCs), server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, network personal computers, minicomputers, mainframe computers, embedded systems, distributed computing environments including any of the above systems or devices, and the like.

[0062] Computer executable instructions, such as program modules, that are executed by a computer may be used. Typically, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Distributed computing environments may be used in which tasks are performed by remote processing devices that are linked through a communication network or other data transmission medium. In a distributed computing environment, program modules and other data may be located in local and remote computer storage media including memory storage devices.

[0063] refer to Figure 5 , an exemplary system for implementing various aspects described herein includes a computing device, such as computing device 500. In its most basic configuration, computing device 500 typically includes at least one processing unit 502 and memory 504. Depending on the precise configuration and type of computing device, memory 504 may be volatile (such as random access memory (RAM)), non-volatile (such as read-only memory (ROM), flash memory, etc.), or some combination of the two. This most basic configuration is Figure 5 It is shown by dashed line 506 in FIG.

[0064] Computing device 500 may have additional features / functionality. For example, computing device 500 may include additional storage (removable and / or non-removable) including, but not limited to, magnetic disks, optical disks, or tapes. Figure 5 Such additional storage is illustrated in FIG. 5 by removable storage 508 and non-removable storage 510 .

[0065] Computing device 500 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by device 500 and includes both volatile and nonvolatile media, removable and non-removable media.

[0066] Computer storage media includes volatile and nonvolatile, and removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Memory 504, removable storage 508, and non-removable storage 510 are all examples of computer storage media. Computer storage media include, but are not limited to, RAM, ROM, electrically erasable program read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 500. Any such computer storage media can be part of computing device 500.

[0067] The computing device 500 may also have input device(s) 514, such as a keyboard, mouse, pen, voice input device, touch input device, etc. It may also include output device(s) 516, such as a display, speaker, printer, etc. All of these devices are well known in the art and need not be discussed in detail here.

[0068] It should be understood that the various techniques described herein may be implemented in conjunction with hardware or software or, where appropriate, with a combination of the two. Thus, the methods and apparatus of the presently disclosed subject matter, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) contained in a tangible medium such as a floppy disk, CDROM, hard drive, or any other machine-readable storage medium, where when the program code is loaded into and executed by a machine such as a computer, the machine becomes an apparatus for practicing the disclosed subject matter.

[0069] Although exemplary implementations may refer to utilizing aspects of the presently disclosed subject matter in the context of one or more stand-alone computer systems, the subject matter is not limited thereto, but may be implemented in conjunction with any computing environment, such as a network or distributed computing environment. Further, aspects of the disclosed subject matter may be implemented in or across multiple processing chips or devices, and storage may similarly be implemented across multiple devices. For example, such devices may include personal computers, network servers, and handheld devices.

[0070] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A computer-implemented method for visual search, the method comprising: Receiving image input by a computing device; determining, by the computing device, at least one object represented in the image input; detecting, by the computing device, a user selection of an object represented in the image input; In response to the user selection, generating, by the computing device, a feature vector representing a user-selected object in the image input and a first plurality of attributes for the user-selected object; retrieving, by the computing device, a category for the object selected by the user; retrieving, by the computing device, a second plurality of attributes predetermined for the category; combining, by the computing device, the first plurality of attributes and the second plurality of attributes into an attribute set; generating, by the computing device, one or more user-selectable options for each attribute in the attribute set; generating, by the computing device, a plurality of search results from processing, in an image processing framework, the feature vector, the set of attributes, and the options selected by the user, in response to a user selection of one or more of the user-selectable options; generating, by the computing device, a ranked ordered list of the plurality of search results, the plurality of search results being ranked ordered according to a matching criterion with the set of attributes for the object selected by the user; as well as The ranked ordered list is displayed on a user interface of a display device.

2. A computer-implemented method according to claim 1, wherein the determination of the object represented in the image input is performed using an object detection method, the method being at least one of the following: a Faster R-CNN (Faster Region-based Convolutional Neural Network) method and a Single Shot Multi-box Detection method.

3. The computer-implemented method of claim 2, wherein the determining of the object further comprises: applying an object localization method to said image input for establishing the position at which each object is represented in said image; as well as An object class classification method is performed for determining a class for each object represented in the image using the object localization method.

4. The computer-implemented method of claim 2, wherein the generating of the feature vector comprises applying a feature extraction method to the user-selected object, the method comprising: detecting one or more physical and visual features of the user-selected object; as well as A visual word quantified representation of the physical features and the visual features of the user-selected object is determined.

5. The computer-implemented method of claim 1 , wherein said generating of said first plurality of attributes comprises applying a feature extraction method, said method comprising: detecting a plurality of physical features and a plurality of visual features of the user-selected object represented in the image; as well as A plurality of detected features and a category are associated with the user-selected object, the category being identified from determining the object represented in the image input.

6. The computer-implemented method of claim 1 , wherein the retrieving of the second plurality of properties for the user-selected object comprises: searching a database using the category identified from determining the object represented in the image input; as well as The plurality of attributes predetermined in the database having a stored association with the category of the at least one object are retrieved.

7. The computer-implemented method of claim 1 , wherein the user-selectable options for each attribute are displayed on a user interface according to at least one of: User preferences monitored; and A ranking score for each user-selectable option, the ranking score representing a relative visual similarity ranking to the user-selected object.

8. The computer-implemented method of claim 6, wherein the database is at least one of an inverted index database and a key-value database, the database storing data in an index relationship between the category and the second plurality of attributes associated with the category.

9. The computer-implemented method of claim 1 , wherein said generating of said plurality of search results from processing said feature vector, said set of attributes, and said plurality of user-selectable options in said image processing framework comprises: generating a visual word quantized representation of the user-selected object from said processing of said feature vector in a visual word quantization component in said framework; generating, using a product quantization component, a product quantization representation of the generated quantized representation of the visual word; as well as A list of products and images that are visually similar to the user-selected object represented in the image input is generated from processing the product quantitative representations, the attribute sets, and the user-selected options in a result ranking component.

10. The computer-implemented method of claim 9, wherein said generating of said ranked ordered list of said plurality of search results comprises: generating a list of one or more products and images, each of the one or more products and images being visually similar to the user-selected object represented in the image input; comparing the set of attributes to at least one of the attributes for each of the one or more products and images in the generated list of products and images; as well as The ranked ordered list of the one or more products and images related to the user-selected object is sorted from most visually similar to least visually similar.

11. The computer-implemented method of claim 1 , wherein the matching criteria is determined from a statistical optimization process, the optimization process comprising: Apply k-means clustering to multiple images in the image index; as well as The centroid of each cluster for the image is compared to clusters including a plurality of image data from the feature vector, wherein the plurality of search results include one or more products and images, the plurality of search results being ranked for visual similarity to the user-selected object based on an n-nearest neighbor centroid statistical analysis.

12. A computer program product for visual search, the computer program product comprising a memory, the memory containing program instructions, the program instructions when executed by a processor, causing the processor to: displaying the image input on a graphical user interface of a user device; determining at least one object represented in the image input; detecting a user selection of at least one object represented in the image input; generating, in response to the user selection, a feature vector representing at least one user-selected object in the image input and a first plurality of attributes for the at least one user-selected object detected in the image input; retrieving a second plurality of attributes for the at least one user-selected object, the second plurality of attributes being predetermined for a category of the at least one user-selected object; displaying in a first area of ​​the graphical user interface the generated list of one or more user-selectable options for each attribute in a set of attributes, the set of attributes including the first plurality of attributes and the second plurality of attributes; responsive to a user selection of one or more of said user-selectable options, generating a plurality of search results from processing in an image processing framework said feature vector, said set of attributes, and said options selected by the user; as well as A ranked ordered list of the plurality of search results is displayed in a second area of ​​the graphical user interface, the plurality of search results being ranked ordered according to matching criteria with the set of attributes for the at least one user-selected object.

13. A computer program product according to claim 12, wherein the user-selectable options for each attribute are displayed on the graphical user interface based on at least one of the monitored user preferences and a ranking score for each option, the ranking score representing a relative visual similarity ranking to the user-selected object.

14. The computer program product of claim 12, wherein the generating of the plurality of search results from processing the feature vector, the set of attributes, and the plurality of user-selectable options in the image processing framework comprises: generating a visual word quantized representation of the user-selected object from said processing of said feature vector in a visual word quantization component in said framework; generating a product quantized representation of the generated quantized representation of the visual word from the processing performed in the product quantization component; as well as A list of products and images that are visually similar to the user-selected object represented in the image input is generated from processing the product quantitative representations, the attribute sets, and the user-selected options in a result ranking component.

15. The computer program product of claim 14, wherein the generating of the ranked ordered list of the plurality of search results comprises: generating a list of one or more products and images, each of the one or more products and images being visually similar to the user-selected object represented in the image input; comparing the set of attributes to at least one of the attributes for each of the one or more products and images in the generated list of products and images; as well as The ranked ordered list of the one or more products and images related to the user-selected object is sorted from most visually similar to least visually similar.

16. The computer program product of claim 12, wherein the matching criteria is determined from a statistical optimization process, the optimization process comprising: Apply k-means clustering to multiple images in the image index; as well as The centroid of each cluster for the image is compared to clusters including a plurality of image data from the feature vector, wherein the plurality of search results include one or more products and images, the plurality of search results being ranked for visual similarity to the user-selected object based on an n-nearest neighbor centroid statistical analysis.

Images