Hierarchical classifiers for image and text data

Abstract

Indexing data is disclosed. Image and text data associated with a dataset are received. Labels are generated using one or more hierarchical classifiers. The image and text data are input into at least one of the one or more hierarchical classifiers. A search index is generated based on at least the generated labels.

Description

[0001] This application is a divisional application of the application with the application date of June 7, 2016, application number 201680033242.8 and invention name “Image and text data hierarchical classifier”.

[0002] Cross-reference to other applications

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 173,828, filed June 10, 2015, entitled DEEP LEARNING SEARCH, which is incorporated herein by reference for all purposes. Background Art

[0004] When searching for information, search indexes are often used to speed up the search process. For example, the content being searched is preprocessed, and selected information about the content is stored in a search index. However, search results are often only as good as what is understood and indexed about the content being searched. For example, the results of a search query are highly dependent on what information can be indexed for the content being searched. Improving and increasing the scope of indexed information will improve the accuracy and performance of search results. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0006] Figure 1A and Figure 1B is a screenshot illustrating search results of a prior art search system.

[0007] Figure 2 is a block diagram illustrating an embodiment of a system for indexing a data set.

[0008] Figure 3 is a flow diagram illustrating an embodiment of a process for generating a search index.

[0009] Figure 4 is a flow diagram illustrating an embodiment of a process for automatically generating one or more labels for a data set.

[0010] Figure 5 is a flow chart illustrating an embodiment of a process for identifying one or more tags using an artificial neural network.

[0011] Figure 6 is a flow chart illustrating an embodiment of a process for resolving inconsistencies between a dataset and generated labels.

[0012] Figure 7 is a flow diagram illustrating an embodiment of a process for processing a search query. DETAILED DESCRIPTION

[0013] The present invention may be implemented in a variety of ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on and / or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of the disclosed processes may be changed within the scope of the present invention. Unless otherwise specified, a component (such as a processor or memory) described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform that task at a given time or as a specific component manufactured to perform that task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0014] A detailed description of one or more embodiments of the present invention is provided below together with the accompanying drawings that illustrate the principles of the present invention. The present invention is described in conjunction with such embodiments, but the present invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention encompasses numerous alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a comprehensive understanding of the present invention. These details are provided for illustrative purposes, and the present invention can be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material known in the art related to the present invention has not been described in detail so that the present invention will not be unnecessarily obfuscated.

[0015] Figure 1A and Figure 1B is a screenshot illustrating search results of a prior art search system. In response to a search query to find all available items described as "red tables" for sale on an online merchant's website, search results 100 are provided as the top search results by the prior art search system. Even though the merchant sells red-colored tables, none of these top results are actually red tables as expected by the search query. As evidenced by search result 100, these poor search results are likely due to the use of a search index that does not include a relevant set of indexed terms. In some embodiments, in order to increase the amount of descriptive information available about a product, image recognition is used to process images of the product and generate additional tags to be included in the search index. For example, image recognition is used to automatically identify the presence of a table and its color in an associated image of the product, and the identified features are associated with the product as tags.

[0016] Indexed datasets are disclosed. In some embodiments, image and text data associated with each of a plurality of datasets are retrieved. For example, each dataset corresponds to a different product for sale, and each dataset includes information about the corresponding product. Labels are generated using one or more hierarchical classifiers, and the image and text data are input into the one or more hierarchical classifiers. The generated labels are used to generate a search index for the datasets.

[0017] For example, rather than simply indexing the textual information specified within a dataset, additional information about each dataset is generated for search and indexing. In some embodiments, product images are analyzed using image recognition, computer vision, and / or neural networks to identify tags / attributes related to the image's content. By deriving additional information about the dataset to be searched, the scope of information that can be matched against each dataset is expanded. For example, while a dataset describing clothing or apparel items may identify a single, overall color of the garment within the dataset's textual data records, image recognition (e.g., guided by the textual data) may be performed on a photograph of the item to identify other colors or patterns on the item. To guide and improve image recognition performance, textual data (e.g., item names) included in the corresponding dataset is utilized. This textual data and the image data associated with the data can be used together in a deep learning neural network to improve the accuracy and detection of automatically generated descriptive tags for the dataset. Other examples of datasets include advertising data, video data, and any other content to be searched.

[0018] Figure 2 2 is a block diagram illustrating an embodiment of a system for indexing datasets. A search system 206 is connected to a dataset provider system 202 and a user system 204 via a network 208. The search system 206 receives a dataset to be indexed from the dataset provider system 202. For example, the dataset provider system 202 uploads information describing a product to the search system 206 to allow the search system 206 to build a search index, thereby enabling searches for the product using a search query. In addition to indexing the textual content included in the provided dataset, the search system 206 generates and indexes additional information associated with the dataset. For example, image recognition (e.g., using computer vision, artificial neural networks, deep learning, etc.) is used to analyze images associated with the dataset to extract additional information about the dataset. This new information can be output as tags, associated with the corresponding dataset, and indexed for search. In some embodiments, generating tags includes utilizing the textual content included in the dataset to guide / supplement image recognition. For example, the textual content and image content are used together as input to an artificial neural network to generate tags.

[0019] In some embodiments, a hierarchical classifier is used to generate labels. For example, there is a hierarchy of classifiers, and a lower level classifier is only utilized if a higher level classifier has detected enough information to match a lower level classifier. For example, a higher level classifier detects product categories associated with a data set, and for each possible detected product category, there are one or more lower level classifiers specific to the detected category. There may be any number of levels of the classifier hierarchy. Using the results of the hierarchical classifier, a search index is generated for implementing searches of the data set. Examples of search indexes include an inverted index that indexes the generated labels as well as text information included in the data set. The generated search index can be provided to another system that will process the search query and / or can be used by the search system 206 to process the search query.

[0020] User system 204 provides a search query to be processed to identify desired data for a dataset. The search query can be processed by any system with access to the generated search index. For example, the search query can be processed by search system 206, dataset provider system 202, and / or any other server / system with access to the generated search index. In some embodiments, processing the search query includes processing the search query to understand the query's search terms and their relationships (e.g., determining an ontology of the search terms) and using the search index to identify one or more datasets that match the processed search query.

[0021] Examples of dataset provider system 202 and search system 206 include one or more servers, computers, storage devices, network services, networks, computers, and / or other computer / network devices. For example, search system 206 includes a server and a network storage device. Examples of user system 204 include laptops, desktop computers, smartphones, tablet computers, IoT devices, wearable computers, wireless repeaters, wireless routers, or any other wireless computer or device.

[0022] Examples of network 208 include one or more of the following: direct or indirect physical communication connections, mobile communication networks, wireless networks, the Internet, intranets, local area networks, wide area networks, storage area networks, and any other form of connecting two or more systems, components, or storage devices together. Other communication paths may exist, and Figure 2 The examples have been simplified to clearly illustrate the examples. Figure 2 A single instance of the components shown in has been shown to simplify the drawings, but there may be Figure 2 For example, any number of client systems and dataset provider systems may access the search system 206 via the network 208. There may also be Figure 2Components not shown.

[0023] Figure 3 is a flow diagram illustrating an embodiment of a process for generating a search index. Figure 3 The process can be Figure 2 The search system 206 is implemented on the search system 206.

[0024] At 302, a dataset is received to be indexed for searching. In some embodiments, each dataset includes information about a particular product. For example, each dataset includes one or more of the following information about the subject of the dataset: color, price, text description, name, image, image location, reviews, comments, category identifier, related product identifiers, and any other information / attributes associated with the subject. In some embodiments, each dataset corresponds to a different product. Other examples of datasets include advertising data, video data, and any other content to be searched. In some embodiments, each dataset is a document and contains multiple documents. In some embodiments, the dataset is generated by a remote system (e.g., Figure 2 The data set is provided by the system 202 via a network. For example, the remote system uploads the data set to the search indexing system for analysis and indexing. In some embodiments, the data set is provided using an application programming interface (API) provided by the search system. In some embodiments, the data set is encoded in JavaScript Object Notation (JSON) format. In some embodiments, receiving the data set includes receiving a network location address at which at least a portion of the data set can be obtained. In some embodiments, the data set is placed in a processing queue. For example, in order to efficiently and fairly handle the processing of a large number of data sets, the data sets are placed in one or more processing queues. In some embodiments, data sets from different providers / merchants are placed in different processing queues, and the data sets from each processing queue are processed in a first-in, first-out order.

[0025] At 304, attributes are extracted from the datasets. In some embodiments, extracting attributes includes extracting data records included in the datasets to be indexed for search. For example, the datasets are read and parsed to convert the contents of the datasets into a format suitable for search indexing. In some embodiments, extracting attributes includes extracting text data included in each dataset. For example, metadata (e.g., data tags) and associated text content (e.g., text content describing the subject matter / product of the dataset) are obtained from the datasets.

[0026] At 306, one or more external resources identified by the dataset are obtained. In some embodiments, one or more network location addresses (e.g., URIs) of the one or more external resources are identified in the dataset. For example, each dataset includes data records associated with a subject (e.g., a product), and one or more of the data records indicates a network address at which an external resource (e.g., an image) related to the subject can be obtained. In some embodiments, obtaining the one or more external resources includes downloading the external resource via a network. For example, one or more images located at the location address(es) are downloaded and stored for analysis. In some embodiments, one or more images may be included in the dataset and the included images may be extracted from the dataset for analysis. Examples of external resources include images, videos, files, audio, and any other content accessible via a network. In some embodiments, the obtained external resources are stored in a local storage device. In some embodiments, the obtained external resources are placed in a processing queue for processing.

[0027] At 308, one or more hierarchical classifiers are used to automatically generate one or more labels for one or more datasets. The generated labels are associated with the corresponding datasets. For example, a label may include one or more textual items describing the subject matter of the corresponding dataset (e.g., a label identifying a product category, an inferred attribute, etc.), and the label is generated for the corresponding dataset by identifying the content of an image of the subject matter of the dataset. In some embodiments, the hierarchical classifiers are organized in a hierarchically related order. For example, each hierarchical classifier processes input data to determine one or more outputs. One or more of these outputs may be inputs to lower-level hierarchical classifiers and / or may be one or more generated labels associated with the dataset. Whether a classifier is utilized may depend on whether a higher-level classifier has already generated a result corresponding to a particular lower-level classifier (e.g., with at least a threshold confidence level). For example, once a classifier has identified a particular label (e.g., labeled "jewelry") as being associated with a dataset, one or more lower-level classifiers may be used to generate additional sub-labels related to that particular label (e.g., "ring," "gold," etc.). The lower-level classifiers are only utilized if the particular label has already been identified by a higher-level classifier. There may be any number of levels of classifiers, and each sub-level may include one or more classifiers that are selectively utilized based on the results of the classifiers at the immediately higher level.

[0028] In some embodiments, each hierarchical classifier may include one or more artificial neural networks and / or machine learning models trained using manually curated features. For example, an artificial neural network trained using training data may be used to perform image recognition on images in a dataset. In some embodiments, the input to the hierarchical classifier includes both the image data (e.g., obtained in 306) and textual content specified in the dataset (e.g., the attributes extracted in 304). For example, image recognition on images may be aided by the textual content included in the dataset.

[0029] At 310, a search index for the data set is generated. In some embodiments, index-ready documents for the data set are generated, and the index-ready documents include the tags identified in 308. These index-ready documents can be indexed to generate the search index. In some embodiments, generating the search index includes indexing the one or more attributes extracted in 304 and the tags generated in 308. For example, both the extracted attributes and the generated tags are used together to generate the search index. In some embodiments, the search index includes an inverted index. For example, keywords of the one or more extracted attributes and the generated tags are mapped to one or more topics (e.g., products) of the data set via the inverted index. The search index can be used to determine the results of a search query. For example, keywords of the search query are identified, and the search index is used to identify topics of the data set corresponding to the keywords.

[0030] Figure 4 is a flow diagram illustrating an embodiment of a process for automatically generating one or more labels for a data set. Figure 4 The process can be Figure 2 is implemented on the search system 206. In some embodiments, Figure 4 The process is included in Figure 3 In some embodiments, the process is repeated for each data set to be processed. Figure 4 For example, the received data set to be processed has been placed in the processing queue, and Figure 4 The process is used to process each data set in the order in which it is processed.

[0031] At 402, a top-level hierarchical classifier is used to identify one or more labels for the data set. For example, a hierarchical top-level classifier is used to identify a category classification for the data set. In some embodiments, there are multiple possible categories, and one or more categories among the possible categories are selected to be associated with the data set. For example, the subject matter of the data set is a product, and the product category (e.g., apparel, jewelry, electronics, beauty, toys, automotive, etc.) of the product is automatically determined. Although the data set may already include an identifier for the product classification, the determined category classification may correspond to a different classification scheme associated with the hierarchical classifier. In some embodiments, each hierarchical classifier may utilize one or more text data included in the data set and an image of the data set as input. For example, an image of the product and a product name included in the data set are used to determine the product category. The top-level hierarchical classifier may be one of multiple top-level hierarchical classifiers for each data set.

[0032] At 404, any of the one or more tags that have been identified with sufficient confidence are associated with the dataset. For example, when a tag is identified for a dataset, an indicator (e.g., a numerical value) is determined of the confidence that the tag correctly corresponds to the subject matter of the dataset. In some embodiments, the tag is associated with the dataset if the tag's confidence indicator satisfies a threshold (e.g., a threshold value). For example, by associating the tag, the tag is essentially generated for the dataset because the information identified by the tag was not previously specified in the dataset. Associating the tag with the dataset may include mapping the tag to the subject matter of the dataset to allow search queries associated with the tag to match the subject matter of the dataset. In some embodiments, associating the tag includes selecting a tag identified by a classifier from among multiple tag options (e.g., selecting the tag most likely associated with the dataset). In some embodiments, if no tag is identified with sufficient confidence, the unidentified tag (e.g., an "unknown" tag) is identified and associated with the dataset. The "unknown" tag can be used to de-prioritize products associated with the dataset in search rankings.

[0033] At 406, for each associated label, one or more sub-level classifiers (if any) corresponding to the associated label are used to identify one or more additional labels for the dataset. For example, there is a hierarchical organization of classifiers, and whether a particular classifier will be used to generate a label for a dataset depends on whether a higher-level classifier, hierarchically related to the particular classifier, has already identified an output corresponding to the particular classifier (e.g., a label with sufficient confidence). Therefore, a previously identified label identified with sufficient confidence can trigger the one or more sub-level classifiers to generate one or more additional labels associated with the previously identified label. For example, because a higher-level classifier has already identified the subject of the dataset as "ring" with high confidence and associated the label "ring" with the dataset, one or more sub-classifiers are triggered based on the label "ring." These sub-classifiers can generate one or more labels describing the type of gemstone (if any) in the ring, the metal material of the ring, the size of the ring, whether the ring is engraved, and so on. In some embodiments, a label is not associated with a sub-level classifier and does not trigger processing by the sub-classifiers. In some embodiments, a label is associated with multiple sub-classifiers, and identification of a label triggers multiple classifiers. A classifier can identify multiple labels. Examples of labels identified by a hierarchical classifier include the color of a particular object component included in the image, attributes of the color, quality of the color, darkness or lightness of the color, features of an item of clothing, neckline of an item of clothing, length of an item of clothing, sleeve attributes of an item of clothing, and the like.

[0034] At 408, any one of the one or more additional tags that have been identified as having sufficient confidence is associated with the data set. In some embodiments, the tag is associated with the data set if a confidence indicator (e.g., a numerical value determined by the classifier) ​​for the identified tag satisfies a threshold (e.g., a value of the threshold). Associating the additional tag with the data set may include mapping the additional tag to the data set to allow search queries associated with the tag to match the subject matter of the data set. In some embodiments, associating the tag includes selecting a tag from among a plurality of tag options that has been identified by the classifier as most likely associated with the data set.

[0035] At 410 , it is determined whether a stopping criterion has been reached. For example, the stopping criterion may not have been reached because there are one or more additional hierarchical classifiers triggered by one or more additional tags, and the next sub-level of classifiers is to be executed.

[0036] In some embodiments, there are multiple levels of classifiers, and when the lowest level is reached, the stopping criterion is met. In some embodiments, the stopping criterion is met when no additional labels are associated with the dataset at 408 or when the associated label(s) at 408 do not trigger other classifiers. If it is determined at 410 that the stopping criterion has not been met, the process returns to 406, where, for each of the associated additional label(s), one or more sub-level classifiers corresponding to the associated label (if any) are used to identify one or more additional labels for the dataset. If it is determined at 410 that the stopping criterion has been met, the process ends.

[0037] Figure 5 is a flow chart illustrating an embodiment of a process for identifying one or more tags using an artificial neural network. Figure 5 The process can be Figure 2 is implemented on the search system 206. In some embodiments, Figure 5 The process is included in Figure 3 In some embodiments, when Figure 4 When using each level classifier to identify the label in the process, Figure 5 The process is repeated.

[0038] At 502, one or more extracted attributes of a data set are selected for input. For example, a hierarchical classifier includes multiple neural networks, and one or more extracted attributes are selected that are suitable for labels to be identified by the hierarchical classifier (e.g., in Figure 3 304 ) as input. In some embodiments, text data included in the dataset is selected. For example, the text data is a text name specified in the dataset. Other examples of text data include at least a portion of a text description included in the dataset.

[0039] At 504, one or more attribute processing artificial neural networks are used to process the one or more extracted attributes of the dataset. In some embodiments, the type of neural network that will produce the best results is selected as the attribute processing neural network, and the attributes are processed using the one or more attribute processing artificial neural networks to generate an output. Examples of attribute processing artificial neural networks include neural networks best suited for identifying desired attributes of text data. Examples of outputs include data that can be provided to another neural network to identify one or more labels for the dataset. Examples of attribute processing artificial neural networks include one or more of the following: autoencoders, convolutional neural networks, recurrent neural networks, recurrent neural networks, fully connected neural networks, long short-term memory neural networks, memory networks, and neural Turing machines. In some embodiments, the attribute processing artificial neural network(s) have been trained using exemplary training data. For example, backpropagation has been used to train the attribute processing artificial neural network(s). Multiple attribute processing artificial neural networks can process data in stages by feeding the output of one attribute processing neural network as input to another attribute processing neural network.

[0040] At 506, one or more images of the dataset are selected for input. For example, one or more images referenced by the dataset and Figure 3 The image obtained in 306 is used as input. In some embodiments, the selected image is included in a dataset. In some embodiments, the selected image is part of a video. In some embodiments, the selected image may depict the subject of the dataset, and the image is selected as input for image recognition to generate additional labels for the dataset.

[0041] At 508, one or more images of the dataset are processed using one or more image processing artificial neural networks. In some embodiments, a neural network type that will produce the best results for image detection is selected as the image processing neural network, and the images are processed using the one or more image processing artificial neural networks to generate an output. Examples of image processing artificial neural networks include neural networks best suited for image recognition. Examples of outputs include data that can be provided to another neural network to identify one or more labels for the dataset. Examples of image processing artificial neural networks include one or more of the following: a convolutional neural network, a recurrent convolutional neural network, and a restricted Boltzmann machine. In some embodiments, the image processing artificial neural network(s) have been trained using exemplary training data. For example, backpropagation has been used to train the image processing artificial neural network(s). Multiple image processing artificial neural networks can process data in stages by feeding the output of one image processing neural network as input to another image processing neural network.

[0042] At 510, the results of the attribute processing artificial neural network(s) and the image processing artificial neural network(s) are both used as input to one or more joint processing artificial neural networks to identify one or more labels for the dataset. For example, the output of 504 and the output of 508 are both input to the joint processing artificial neural network to discover one or more labels for the dataset. By utilizing both image data and descriptive text data, the accuracy of the identified labels for the dataset can be improved. The joint processing artificial neural network can be organized into stages, with each stage utilizing the output of one joint processing artificial neural network as input to another joint processing artificial neural network. In some embodiments, identifying the labels includes determining whether one or more qualified generated labels detectable by the joint processing artificial neural network(s) correspond to the input. Qualified generated labels can be pre-configured and / or automatically detected / generated.

[0043] In some embodiments, the joint processing artificial neural network has been trained using exemplary training data. Back propagation can be used to continuously improve the accuracy of the joint processing artificial neural network. The attribute processing artificial neural network, the image processing artificial neural network, and the joint processing artificial neural network can be trained together. In some embodiments, identifying the label includes determining a confidence metric for each identified label, the confidence metric identifying the prediction accuracy of the label, and the identified label is associated with the data set only if the confidence metric meets a threshold. In some embodiments, the identified label is Figure 3 308, Figure 4 402 and / or Figure 4 The 406 is provided.

[0044] Figure 6 is a flow chart illustrating an embodiment of a process for resolving inconsistencies between a dataset and generated labels. Figure 6 The process can be Figure 2 In some embodiments, Figure 6 The process is included in Figure 3 308. In some embodiments, Figure 6 The process is included in Figure 4 of 404 and / or 408.

[0045] At 602, one or more tags to be associated with a data set are received. In some embodiments, the received tags are included in Figure 3 308, Figure 4 402 and / or Figure 4 The 406 is identified in the tag.

[0046] At 604, a determination is made as to whether the contents of the dataset are inconsistent with the received label(s). For example, a text description or other data record included in the dataset may be incorrect due to human data entry error (e.g., an apparel product is described in the dataset as being "red" in color when the actual apparel product, as evidenced by the product photo referenced by the dataset, is "blue" in color). In some embodiments, determining the inconsistency includes verifying that each label is consistent with the text attributes included in the dataset. The text attributes of the dataset may be in Figure 3 The 304 extracts the attributes.

[0047] If an inconsistency is determined to exist at 604, then at 606, the inconsistency is resolved. In some embodiments, resolving the inconsistency includes indicating the inconsistency to allow a human reviewer to resolve the inconsistency. In some embodiments, resolving the inconsistency includes modifying the content of the dataset to be consistent with the inconsistent label. In some embodiments, resolving the inconsistency includes removing the inconsistent portion of the dataset. In some embodiments, resolving the inconsistency includes determining not to associate the inconsistent label with the dataset. For example, the label is associated with the dataset only if the identified label is determined not to be inconsistent with existing attributes of the dataset and / or a previously associated label with the dataset.

[0048] If it is determined at 604 that no inconsistency exists, then at 608 the received label(s) are associated with the data set.

[0049] Figure 7 is a flow diagram illustrating an embodiment of a process for processing a search query. Figure 7 The process can be Figure 2 The search system 206 is implemented on the search system 206.

[0050] At 702, a search query is received. In some embodiments, the search query is a query for identifying one or more products that match specified constraints of the search query. In some embodiments, the search query is a query for searching for products in Figure 3 In some embodiments, the query of the data set received in 302 is Figure 2 The user system 204 receives the search query. Other examples of search queries include queries for finding locations within a video that include content matching the search query, queries for searching for relevant images and / or video advertisements, or queries for searching for matching images among a group of images.

[0051] At 704, the search query is analyzed to determine an ontology of the search terms of the search query. For example, the search query is parsed, and the meanings and relationships between the search terms are analyzed and determined to perform a search for the desired content specified by the search query. In some embodiments, the search query is vectorized to determine vectorized representations of the terms of the search query.

[0052] At 706, the search index is used to determine search results responsive to the search query. In some embodiments, the search index is Figure 3 The search index generated in 310. In some embodiments, the search index is generated by indexing one or more attributes and automatically generated tags of the data set, and the semantic terms of the search query are matched with the content referenced in the search index.

[0053] Although the foregoing embodiments have been described in considerable detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative rather than restrictive.

Claims

1. A system comprising: processor; and a memory coupled to the processor, wherein the memory is configured to provide instructions to the processor, the instructions, when executed, causing the processor to perform operations comprising: Get image and text data that are both presented in a specific list of a specific object; generating a first set of labels using one or more hierarchical classifiers based on the images presented in the particular list of the particular objects; generating a second set of labels different from the first set of labels based on the text data presented in the specific list of the specific object; determining, based at least in part on a determination that one or more labels are inconsistent with visual features of the image presented in the particular list, that an inconsistency exists between: (i) a particular label in a first set of labels generated based on the image presented in the list for the particular object and (ii) a different label in a second set of labels generated based on the textual data presented in the particular list for the particular object; resolving the inconsistency based on a comparison of the specific label and the different label with textual data and visual characteristics extracted from the image of the specific object; generating a search index based on at least tags generated from the first set of tags and the second set of tags remaining after resolving the inconsistency; receiving a search query comprising one or more search terms; utilizing the search index, determining that a search result identifying the particular object is responsive to the search query based on the one or more search terms matching the generated tags remaining after resolving the inconsistencies; and Returning a search result presenting the image and the text data, The tags represent attributes of the objects.

2. The system according to claim 1, wherein: The instructions further cause the processor to perform operations comprising obtaining the image from a remote network location address identified by the data set.

3. The system according to claim 1, wherein: The image is a graphical depiction of the particular object.

4. The system according to claim 3, wherein: The text data includes at least a portion of a description of the specific object depicted by the image graphics.

5. The system according to claim 1, wherein: One of the one or more hierarchical classifiers has identified previously identified labels that trigger another of the one or more hierarchical classifiers to generate the second set of labels. 6 . The system of claim 1 , wherein at least one hierarchical classifier included in the one or more hierarchical classifiers is conditionally executed based on a result of a higher-level hierarchical classifier also included in the one or more hierarchical classifiers.

7. The system according to claim 1, wherein: The instructions cause the processor to perform operations including: processing the image using a first artificial neural network; Processing the text data using a second artificial neural network; and The result of the first artificial neural network and the result of the second artificial neural network are used as inputs of the third artificial neural network.

8. A method comprising: Get image and text data that are both presented in a specific list of a specific object; generating, by one or more processors, a first set of labels based on the image presented in the particular list of the particular object using one or more hierarchical classifiers; generating a second set of labels different from the first set of labels based on the text data presented in the specific list of the specific object; determining, by one or more processors, based at least in part on a determination that one or more labels are inconsistent with visual features of the image presented in the particular list, that an inconsistency exists between: (i) a particular label in a first set of labels generated based on the image presented in the list for the particular object and (ii) a different label in a second set of labels generated based on the textual data presented in the particular list for the particular object; resolving, by one or more processors, the inconsistency based on a comparison of the particular label and the different label with textual data and visual characteristics extracted from the image of the particular object; generating, by one or more processors, a search index based on at least tags generated from the first set of tags and the second set of tags remaining after resolving the inconsistency; receiving a search query comprising one or more search terms; determining, by one or more processors, utilizing the search index, that a search result identifying the particular object is responsive to the search query based on the one or more search terms matching the generated tags remaining after resolving the inconsistencies; as well as Returning, by one or more processors, a search result presenting the image and the text data, The tags represent attributes of the objects.

9. The method according to claim 8, further comprising: The image is obtained from a remote network location address identified by the dataset.

10. The method according to claim 8, wherein The image is a graphical depiction of the particular object.

11. The method according to claim 10, wherein: The text data includes at least a portion of a description of the specific object depicted by the image graphics.

12. The method according to claim 8, wherein One of the one or more hierarchical classifiers has identified previously identified labels that trigger another of the one or more hierarchical classifiers to generate the second set of labels.

13. The method of claim 8, wherein at least one hierarchical classifier included in the one or more hierarchical classifiers is conditionally executed based on a result of a higher-level hierarchical classifier also included in the one or more hierarchical classifiers.

14. The method according to claim 8, further comprising: processing the image using a first artificial neural network; Processing the text data using a second artificial neural network; and The result of the first artificial neural network and the result of the second artificial neural network are used as inputs of the third artificial neural network.

15. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: Get image and text data that are both presented in a specific list of a specific object; generating, by one or more processors, a first set of labels based on the image presented in the particular list of the particular object using one or more hierarchical classifiers; generating a second set of labels different from the first set of labels based on the text data presented in the specific list of the specific object; determining, based at least in part on a determination that one or more labels are inconsistent with visual features of the image presented in the particular list, that an inconsistency exists between: (i) a particular label in a first set of labels generated based on the image presented in the list for the particular object and (ii) a different label in a second set of labels generated based on the textual data presented in the particular list for the particular object; resolving, by one or more processors, the inconsistency based on a comparison of the particular label and the different label with textual data and visual characteristics extracted from the image of the particular object; generating, by one or more processors, a search index based on at least tags generated from the first set of tags and the second set of tags remaining after resolving the inconsistency; receiving a search query comprising one or more search terms; determining, by one or more processors, utilizing the search index, that a search result identifying the particular object is responsive to the search query based on the one or more search terms matching the generated tags remaining after resolving the inconsistencies; as well as Returning, by one or more processors, a search result presenting the image and the text data, The tags represent attributes of the objects.

16. The non-transitory computer-readable storage medium of claim 15, wherein the instructions cause the one or more processors to perform operations comprising obtaining the image from a remote network location address identified by a data set.

17. The non-transitory computer-readable storage medium of claim 15, wherein: The image is a graphical depiction of the particular object. 18 . The non-transitory computer-readable storage medium of claim 17 , wherein the textual data includes at least a portion of a description of the particular object depicted by the image graphic.

19. The non-transitory computer-readable storage medium of claim 15, wherein one of the one or more hierarchical classifiers has identified a previously identified label, and the previously identified label triggers another classifier of the one or more hierarchical classifiers to generate the second set of labels.

20. The non-transitory computer-readable storage medium of claim 15, wherein at least one hierarchical classifier included in the one or more hierarchical classifiers is conditionally executed based on a result of a higher-level hierarchical classifier also included in the one or more hierarchical classifiers.

Images