Limit classification processing using graphs and neural networks

Abstract

Systems and methods are provided for learning a classifier for annotating documents with predicted labels under an extreme classification where there are more than one million labels. The learning includes receiving a joint graph comprising documents and labels as nodes. A multi-dimensional vector representation of the documents (i.e., document representations) is generated based on graph convolution of the joint graph. Each document representation varies in dependence on adjacent nodes to accommodate context. The document representations are feature transformed using a residual layer. A per-label document representation is generated from the transformed document representations based on adjacent label attention. The classifier is trained for each of the more than one million labels based on joint learning using training data and the per-label document representations. The trained classifier performs highly efficiently compared to other classifiers trained using disjoint graphs of documents and labels.

Description

BACKGROUND

[0001] Traditional systems provide recommendations as a result of receiving a search query by determining the recommendations from a set of candidates. These systems analyze the terms or context of a received search query and classify it into one or more of thousands of candidate classes. These classification systems can be in various applications such as web search, product recommendations, and query rewriting, among others. In practice, the accuracy of the recommendations becomes an issue when the size of the candidate classes is not large enough. For example, a set of candidate classes (e.g., thousands of classes) can not necessarily include candidates that accurately reflect the desired recommendations from the millions of possible recommendations. Improving the accuracy of providing recommendations requires more and more candidate classes. In fact, not thousands but millions of candidate classes are needed without sacrificing the performance of training or classification. Thus, it is desirable to develop techniques that better meet these requirements with sufficient performance in both the training and prediction phases of the classification system.

[0002] These and other general considerations have led to the disclosure of aspects herein. Further, although the examples can discuss relatively specific problems, it should be understood that the examples are not limited to solving the particular problems identified in the background or elsewhere. SUMMARY

[0003] According to the present disclosure, the above and other problems are solved by training an extreme classification system by generating a set of classifiers based on a joint graph of documents and labels. In aspects, a "document" can refer to an input to the system and a "label" can refer to an output of the system.

[0004] The present disclosure relates to generating a set of classifiers during a training process of a classification system based on a joint graph of documents and labels. Thereafter, the trained classification system can be used in a prediction phase to leverage a most relevant subset of labels from a vast set of labels (e.g., millions of labels) to annotate a given document. In particular, the joint graph includes combinations of documents and labels as nodes, and relationships between at least one pair of documents and labels as edges. Graph convolutions are performed on nodes adjacent to a node to generate document representations for learning. In aspects, a set of multiple vector representations of a document node can be generated based on successively increasing a node’s “neighborhood” relative to the document node within the joint graph. For example, a first node can have a direct relationship (or edge) with the document node, a second node can have a direct relationship with the first node, and a third node can have a direct relationship with the second node, and so on, relative to the document node. A “first hop” from the document node can refer to a first neighborhood level that includes the first node, a “second hop” from the document node can refer to a second neighborhood level that includes the first node and the second node, a “third hop” from the document node can refer to a third neighborhood level that includes the first node, the second node, and the third node, and so on. In aspects, information from a node (or label) can become more generalized and include more adjacent nodes (or labels) as the number of hops from the document node increases. Vector representations of a document node can be weighted differently based on the number of “hops” (or neighborhood levels) from the document node. In aspects, a vector representation of a document node within a particular number of hops in the joint graph can be weighted relative to another vector representation of a document node within another number of hops.

[0005] The feature label generator generates a vector representation of a document representation for each label in a vast number of labels (e.g., millions of labels) based on label attention. Each of the per-label document representations includes label attention to document representations in the document representations with a specified number of hops based on the position of the label in the graph (e.g., head label or tail label). The label “attention” weights one or more features of adjacent nodes in the joint graph to generate a document representation for the label. The output of the feature label generator based on label attention is millions of document representations, one for each label in the millions of labels. In some aspects, the present disclosure then learns a separate classifier for each label based on the per-label document representations as input. In some other aspects, the use of a shortlist during training and prediction can reduce the number of classifiers to process for improved performance. The accuracy of predictions or recommended candidates for a given query is improved based on classification using a vast number of classifiers for respective labels.

[0006] The disclosed technology includes a training process and a prediction process of a classification system. The training process is the main focus of the present disclosure. The process can include one or more of graph convolution, feature transformation, feature label attention, and classifier generation. The graph convolution includes generating a vector representation of a document based on varying hop counts from respective document nodes within a joint graph of documents and labels. The feature transformation includes transforming the vector representation of the document based on residual layers with skip connections. The feature label attention generates a separate document representation for each label. Each label differentially attends to different hops of a neighborhood depending on whether the node corresponds to a head label or a tail label in the joint graph. Adjusting the range of neighbors in the joint graph for label attention provides sufficient information to provide accuracy in labeling documents.

[0007] Some examples include a method for training a classifier. The method can include receiving a joint graph, where the joint graph includes nodes and directed edges, and where each node is either a document type or a label type, generating a plurality of multi-dimensional vector representations of a document based on graph convolution of the joint graph, where the document is represented by a document node in the joint graph, and where each multi-dimensional vector representation of the document corresponds to a different level of neighboring nodes from the document node in the joint graph, generating a transformed multi-dimensional vector representation of the document based on the generated plurality of multi-dimensional vector representations of the document using residual layers of a neural network, where the residual layers include a residual network and skip connections, generating a plurality of per-label multi-dimensional vector representations of the document using label attention, where each per-label multi-dimensional vector representation of the document corresponds to a label of a plurality of labels, and where the label attention applies variable weighting to one or more of the transformed multi-dimensional vector representations of the document to generate the per-label multi-dimensional vector representations, training a classifier for each label of the plurality of labels, and sending the plurality of trained classifiers for deployment in a label prediction process. The levels of neighboring nodes from the document node correspond to hop counts from the document node of the joint graph, and where the label attention for tail labels applies higher weighting to multi-dimensional vector representations of the document generated based on higher levels of neighboring nodes. The plurality of multi-dimensional vector representations of the document includes a first multi-dimensional vector representation based on nodes within one hop from the document node, a second multi-dimensional vector representation based on nodes within two hops from the document node, a third multi-dimensional vector representation based on nodes within three hops from the document node. The graph convolution includes vectorizing features of the document and labels associated with nodes within a predetermined number of hops from the document node to generate the multi-dimensional vector representation of the document. The number of the plurality of labels is at least one million, and where the number of the plurality of per-label multi-dimensional vector representations of the document is at least one million. The method can further include deploying the plurality of trained classifiers for the label prediction process, receiving a document, generating a set of recommended candidate labels for the received document using a subset of the plurality of trained classifiers, and sending the generated recommended candidate labels as a response to the document.

[0008] Some examples include a system for learning classifiers for labels in extreme classification processing. The system includes a processor and a memory storing computer-executable instructions that when executed by the processor cause the system to: receive a joint graph, wherein the joint graph includes nodes and directed edges, and wherein each node is either a document type or a label type, generate a plurality of multi-dimensional vector representations of a document based on graph convolution of the joint graph, wherein the document is represented by a document node in the joint graph, and wherein each multi-dimensional vector representation of the document corresponds to a different level of neighboring nodes from the document node in the joint graph, generate a transformed multi-dimensional vector representation of the document based on the generated plurality of multi-dimensional vector representations of the document using a residual layer of a neural network, wherein the residual layer includes a residual network and a skip connection, generate a plurality of per-label multi-dimensional vector representations of the document using label attention, wherein each per-label multi-dimensional vector representation of the document corresponds to a label of a plurality of labels, and wherein the label attention applies variable weighting to one or more of the transformed multi-dimensional vector representations of the document to generate the per-label multi-dimensional vector representations, train a classifier for each label of the plurality of labels, and send the plurality of trained classifiers for deployment in a label prediction process. The levels of neighboring nodes from the document node correspond to a number of hops from the document node of the joint graph, and wherein the label attention for a tail label applies higher weighting to multi-dimensional vector representations of the document generated based on higher levels of neighboring nodes. The plurality of multi-dimensional vector representations of the document includes a first multi-dimensional vector representation based on nodes within one hop from the document node, a second multi-dimensional vector representation based on nodes within two hops from the document node, a third multi-dimensional vector representation based on nodes within three hops from the document node. The graph convolution includes vectorizing the document and features of labels associated with nodes within a predetermined number of hops from the document node to generate the multi-dimensional vector representation of the document. The computer-executable instructions when executed further cause the system to: deploy the plurality of trained classifiers for the label prediction process, receive a document, generate a set of recommended candidate labels for the received document using a subset of the plurality of trained classifiers, and send the generated recommended candidate labels as a response to the document. The document includes at least one of a web page as a search result in a web search application, a targeted content snippet in a sponsored search application, another product in a product recommendation application, or another content title in a content recommendation application, and wherein the label includes at least one of a query in the web search application, a query in the sponsored search application, a product in the product recommendation application, or a content title in the content recommendation application. The computer-executable instructions when executed further cause the system to: generate a set of classifiers for learning; and learn each classifier in the set of classifiers in a batch.

[0009] Some examples include a computer-readable non-transitory recording medium storing computer-executable instructions. The computer-readable non-transitory recording medium can store computer-executable instructions that, when executed by a processor, cause a computer system to: receive a joint graph, wherein the joint graph includes nodes and directed edges, and wherein each node is either a document type or a tag type, generate, based on a graph convolution of the joint graph, a plurality of multi-dimensional vector representations of a document, wherein the document is represented by a document node in the joint graph, and wherein each multi-dimensional vector representation of the document corresponds to a different level of neighboring nodes from the document node of the joint graph, generate, using a residual layer of a neural network, a transformed multi-dimensional vector representation of the document based on the generated plurality of multi-dimensional vector representations of the document, wherein the residual layer includes a residual network and a skip connection, generate, using tag attention, a plurality of per-tag multi-dimensional vector representations of the document, wherein each per-tag multi-dimensional vector representation of the document corresponds to a tag of the plurality of tags, and wherein the tag attention applies variable weighting to one or more of the transformed multi-dimensional vector representations of the document to generate the per-tag multi-dimensional vector representations, train a classifier for each tag of the plurality of tags, and send the plurality of trained classifiers for deployment in a tag prediction process. The levels of neighboring nodes from the document node correspond to a number of hops from the document node of the joint graph, and wherein the tag attention for a tail tag applies higher weighting to the multi-dimensional vector representations of the document generated based on higher levels of neighboring nodes. The plurality of multi-dimensional vector representations of the document includes a first multi-dimensional vector representation based on nodes within one hop from the document node, a second multi-dimensional vector representation based on nodes within two hops from the document node, a third multi-dimensional vector representation based on nodes within three hops from the document node. The computer-executable instructions, when executed, further cause the system to: deploy the plurality of trained classifiers for the tag prediction process, receive a document, generate, using a subset of the plurality of trained classifiers, a set of recommended candidate tags for the received document, and send the generated recommended candidate tags as a response to the document. The document includes at least one of: a web page as a search result in a web search application, a targeted content snippet in a sponsored search application, another product in a product recommendation application, or another content title in a content recommendation application, and wherein the tags include at least one of: a query in the web search application, a query in the sponsored search application, a product in the product recommendation application, or a content title in the content recommendation application.

[0010] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of examples will be set forth in part in the description that follows, and in part will be apparent from the description, or can be learned by practice of the disclosure. Attached Figure Description

[0011] Examples of non-limiting and non-exhaustive features are described with reference to the accompanying drawings.

[0012] Figure 1 The illustration shows an overview of an example system for training a classifier and predicting labels according to various aspects of this disclosure.

[0013] Figure 2 An example of a system architecture based on various aspects of this disclosure is illustrated.

[0014] Figure 3 Examples of data structures for generating joint graphs according to various aspects of this disclosure are illustrated.

[0015] Figure 4 Examples of data structures for generating joint graphs according to various aspects of this disclosure are illustrated.

[0016] Figure 5 Examples of data structures for generating joint graphs according to various aspects of this disclosure are illustrated.

[0017] Figure 6 Examples of methods for training a classifier and predicting labels according to various aspects of this disclosure are illustrated.

[0018] Figure 7 This is a block diagram illustrating an example physical component of a computing device that can be used to implement various aspects of the present disclosure.

[0019] Figure 8A This is a simplified illustration of a mobile computing device that can be used to implement various aspects of the present disclosure.

[0020] Figure 8B This is another simplified illustration of a mobile computing device that can be used to implement various aspects of the present disclosure. Detailed Implementation

[0021] Various aspects of this disclosure will now be described more fully with reference to the accompanying drawings, which are taken from a part of this disclosure and illustrate specific example aspects. However, different aspects of this disclosure may be implemented in many different ways and should not be construed as limited to the aspects set forth herein; rather, these aspects are provided so that this disclosure will be thorough and complete and will fully convey the scope of these aspects to those skilled in the art. The aspects may be implemented as methods, systems, or apparatuses. Thus, aspects may take the form of hardware implementations, entirely software implementations, or implementations combining software and hardware aspects. Therefore, the detailed description below is not limiting.

[0022] Classification of information relies on understanding a given piece of information (e.g., a document) and determining a subset of classes or labels from a set of classes or labels in the given piece of information. Various applications use classification systems to generate and provide recommendations based on a given query. For example, web search applications provide a set of web pages (e.g., addresses of web pages) that match a given query. Many web search sites provide sponsored search that directs content (e.g., web advertisements) along with search results based on a given query. Product recommendation applications provide a set of recommended products based on a given product or a query for a product.

[0023] In aspects, a document can be a given query in web search or web advertisement recommendation, a given product in product recommendation, or a given movie in movie recommendation. A label can be synonymous with a class. In some aspects, a label can be each recommended website in a set of websites responsive to a query in web search, a piece of web advertisement (or direct content) in sponsored search, each product in a set of products in product recommendation, or each recommended movie in a set of movies in movie recommendation. In some other aspects, a label can be a set of websites responsive to a query in web search, a web advertisement (or directed content) in sponsored search, a set of products as recommendations for a given product in product recommendation, or a set of movies as recommendations for a given movie title in movie recommendation.

[0024] As the number of potential results grows, classification systems select candidate recommendations from the growing number of web pages, products, and directed content, respectively. Thus, as the number of potential results grows, providing accurate recommendations requires analyzing individual content and products. When there is a large set of candidate labels (e.g., over a million candidate labels), the latency associated with generating recommendations can become prohibitively high

[0025] Extreme classification (XC) involves performing a task of annotating a document with a most relevant subset of labels in a large set of candidate labels as recommended labels. Some XC systems learn documents and labels as two disjoint sets. Some other XC processes are unable to include metadata of labels in generating and training a classifier due to a large number of labels to train. This application enables efficiently training an XC process system by generating a document representation based on a joint graph that includes documents and labels. The training also transforms a feature-based document representation using a residual layer. The training then generates a per-label document representation with label attention to neighboring nodes of a range of changes to a node associated with the document in the joint graph.

[0026] As discussed in more detail below, the present disclosure relates to an XC processing system for predicting recommended labels for a given document based on a vast set of labels (e.g., millions). The XC system in accordance with the present technology receives a document and provides one or more recommended labels based on millions of candidate labels. In particular, the XC processing system uses a joint graph including the document and labels with feature transformations on residual layers and label attention to generate per-label document representations and further generate trained per-label classifiers. The present technology addresses the problem of providing high performance and high accuracy XC by including features of the document and candidate labels in the neighborhood of the document node within the joint graph to generate document representations.

[0027] Figure 1 An overview of an example system 100 for generating and using labels and classifiers based on documents in accordance with aspects of the present disclosure is illustrated. The system 100 represents a system for using graph convolution and feature transformations with label attention to generate trained classifiers for candidate labels in extreme classification (XC) processing. The system 100 includes a client device 102, an application server 110, an extreme classification server 120, a classifier generator 140, a network 130, and a data server 160. The client device 102 is in communication with the application server 110, which includes one or more sets of instructions that are executed as an application on the client device 102. The application server 110 includes a data retriever 112 and a data viewer 114. The one or more sets of instructions in the application server 110 can provide an interactive user interface through an interactive browser 104. The extreme classification server 120 includes a query receiver 122, a document classifier 124, a class storage 126, an answer provider 128, and an activity log 130. The network 130 provides network connectivity between the client device 102, the application server 110, the extreme classification server 120, and the data server 160. The classifier generator 140 includes a training database 142, a joint graph database 144, a graph convolution generator 146, a feature transformer 148, a label attention generator 150, and a classifier learner 152. The data server 160 stores data and provides the data to the extreme classification server 120 and the application server 110 through the network 130.

[0028] Client device 102 connects with application server 110 via network 130 to execute applications including user interaction through interactive browser 104. Application server 110 interacts with client device 102 and limit classifier server 120 via network 130 to execute various application services (e.g., web search, sponsored search, product recommendation, movie recommendation, etc.). Classifier generator 140 generates a set of classifiers during a training process. A classifier classifies a given document into one or more recommended labels. In aspects, there is at least one classifier for each label. Classifier generator 140 generates the set of classifiers based on a joint graph. The joint graph includes documents and labels for training. Classifier generator 140 deploys the generated classifiers to limit classifier server 120 for production use.

[0029] Client device 102 is a general purpose computer device that provides user input capabilities for searching information over network 130, e.g., via interactive browser 104. In some aspects, client device 102 optionally receives user input from a system developer for fine-tuning various parameters for training the XC process. The various parameters for adjustment can include, for example, parameters used in a network for feature transformation of a document representation, as detailed below, as well as adjustment weights in label attention for document representation per label.

[0030] Application server 110 is a server that enables users (who can search information based on deployed trained models) and developers (who can generate classifiers for deployment using classifier generator 140 during a training process) to interactively use system 100 on client device 102. Application server 110 can include applications including data retriever 112 and data viewer 114. Data viewer 112 can provide presentation of data for user viewing. Data retriever 114 can be used to query and receive data, e.g., by communicating with limit classifier server 120.

[0031] In aspects, data retriever 112 can connect with document receiver 122 of limit classifier server 120 to query information. Limit classifier server 120 operates as a prediction process of the XC processing system. For example, an application for retrieving data can include, but is not limited to, a web search that provides one or more recommended websites as recommended tags based on a query input (also referred to as a “document”). In aspects, there can be millions of websites from which a recommendation can be selected. Other examples can include a sponsored website that provides one or more web advertisements (i.e., targeted content) as recommended tags along with search results in response to a given query as a document. There can be millions of web advertisements. A product recommendation provides one or more products as recommended tags in response to a given product query as a document. A movie recommendation provides one or more movie titles as recommended tags in response to a given movie title query as a document.

[0032] Document classifier 124 can use a set of pre-trained classifiers to classify a given document into one or more recommended tags. In aspects, there can be a classifier for each class or tag. In aspects, document classifier 124 annotates a document with one or more recommended tags. Class store 124 stores a set of candidate classes or tags. In the XC processing, there can be more than a million candidate tags from which document classifier 124 can select for a given document. Answer provider 128 provides a set of recommended tags as an answer to a given document. Activity log 130 stores a log of activities of receiving a document and providing recommended tags associated with the document. In aspects, classifier generator 140 can use the content of activity log 130 as a basis for training joint graph database 144 of classifiers.

[0033] Classifier generator 140 represents an application / system for generating a set of classifiers during a training process of the XC processing. The generated set of classifiers can be used for deployment in limit classifier server 120. In embodiments, classifier generator 140 includes training database 142, joint graph database 144, graph convolution generator 146, feature transformer 148, label attention generator 150, and classifier learner 152. In aspects, classifier generator 140 receives input features for a document and outputs a probability of each tag belonging to the document.

[0034] The training database 142 stores training data used to generate a joint graph for training a classifier. In aspects, the training data can be based on the activity log 130, which associates a given query with a set of recommended labels responsive to the respective given query. Thus, the training data includes documents and associated labels. In a web search application, the training data can include a web search activity history, which refers to a web search query and a set of recommended websites (web addresses) responsive to the respective web search query. In a sponsored search application, the training data can include web queries and a set of web-recommended advertisements or targeted content responsive to the respective web queries. In a product recommendation application, the training data can include given product information and a set of recommended products responsive to the given product information. In a movie recommendation application, the training data can include given movie information and a set of movie recommendations responsive to the given movie information.

[0035] The joint graph database 144 includes one or more joint graphs used to train a classifier. A joint graph includes nodes and edges. A node represents a document as a document type or a label as a label type. An edge connects two nodes as a directed link (or relationship) from one node to another node. For example, one node can represent a document; an edge can originate from the document node to another node as a recommended label associated with the document. For example, a document node can represent a given query in a web search. An edge connects the document node to a label node that represents one website in a set of recommended websites in a web search responsive to the given query. In some aspects, the nodes for documents and labels can alternate within a joint graph. In some other aspects, there is no restriction on the order of node types, e.g., from a document node to a label node, from a document node to another document node (e.g., traversing a website), from a label node to another label node, or from a label node to another document node. In aspects, a joint graph includes a tree structure. In some other aspects, a joint graph includes a mesh or a net structure.

[0036] The graph convolution generator 146 uses the joint graph database 144 to perform graph convolution to generate a multi-dimensional vector representation of a document (hereinafter referred to as a document representation). The graph convolution generator 146 generates a document representation by vectorizing neighborhood features and contextual information that can be obtained from neighboring nodes associated with the document. The neighboring nodes are associated with a neighborhood hierarchy that depends on the degree of separation from the document node. For example, a first hierarchy neighborhood (e.g., “first hop”) includes first nodes that are in direct relationship or directly connected to the document node, a second hierarchy neighborhood (e.g., “second hop”) includes the first nodes and second nodes that are in direct relationship with the first nodes, and a third hierarchy neighborhood (e.g., “third hop”) includes the first nodes, the second nodes, and third nodes that are in direct relationship with the second nodes.

[0037] The graph convolutional generator 146 retrieves features and / or contextual information associated with neighboring nodes in different neighborhood levels from the document nodes for generating document representations. In aspects, the range of neighboring nodes used for a particular document representation is based on the number of hops (or neighborhood levels) from the document node. Each neighborhood level can be weighted differently, and to accommodate these various weights for training the classifier, the graph convolutional generator 146 can generate a set of document representations for the document. Each document representation can be based on a unique range of neighboring nodes (or neighborhood levels) based on the number of “hop(s).” For example, the graph convolutional generator 146 can generate a set of three document representations for the document, the first based on one hop, the second based on two hops, and the third based on three hops. In aspects, the graph convolution can also learn existing transfer relationships, which can result in better generalization to unseen documents. Other benefits of implementing the graph convolution in this manner include its flexibility to accommodate additional sources of information, such as items surfaced in a warm start setting. These additional sources of information can be used as important links and allow for more details to be added to the final node embedding.

[0038] The feature transformer 148 transforms the set of document representations from the graph convolution into a set of document representations based on residual layers of a residual neural network for deep learning. In aspects, the residual neural network includes a skip connection for combining the input document representation with an output of the residual network to generate a final output. Thus, the document representation from the graph convolution is multiplied by the weights of the residual layer and then added to a bias term. The residual layer includes a feedforward network, normalization processing, a rectified linear unit (ReLU), and a skip connection. The skip connection enables the document representation to be preserved during the feature transformation. In aspects, the feature transformer 148 generates a set of document representations, each corresponding to a feature transformation of the respective document representation from the graph convolution associated with a number of hops. For example, the feature transformer 148 generates a set of three transformed document representations for each of the first, second, and third hops described above.

[0039] The label attention generator 150 generates a set of per-label (or label-specific) document representations. A per-label document representation is a multi-dimensional vector representation of a document with respect to a particular candidate label. Thus, the number of per-label document representations in the set is the same as the number of candidate labels. In this manner, the extreme classification process enables documents to be annotated with candidate labels from over one million labels.

[0040] In aspects, the head and tail tags correspond to different views of the joint graph based on the number of neighboring tags connected to the respective tag. The head tags in the graph are connected to more neighboring tags than the tail tags. Thus, including neighboring tags in one hop from the head tags results in more tags than in one hop from the tail tags. A candidate tag that is a tail tag can need tags from a large neighborhood of neighbors (i.e., more hops) to include enough information to accurately classify the document as the candidate tag. Including more neighboring nodes enriches the per-tag document representation with contextual information. In contrast, a candidate tag that is a head tag can need fewer hops (e.g., one hop) to obtain enough information to accurately classify. Based on the tag attention, the tag attention generator 150 can learn different attention weights for each tag and hop.

[0041] In aspects, the weights can be normalized using a softmax operation before applying the weights to the respective per-tag document representation. The tag attention generator 150 generates the per-tag document representation by aggregating the weighted, feature-transformed vector representation of the document for each hop. Thus, the per-tag document representation includes the representational capacity to disregard whether the tag is a head tag or a tail tag in the graph.

[0042] The classifier learner 152 learns a set of classifiers, each corresponding to a single tag. The classifier learner 152 uses the training database 142 in conjunction with the per-tag document representation to learn a classifier for the respective tag. In aspects, the classifier learner 152 performs the learning (or training) process in a series of batches to improve the efficiency of learning millions of classifiers, each corresponding to a candidate tag. The batching reduces the training cost by randomly sampling negatives, which are negative labels for all documents in the batch. For example, one batch can include 256 classifiers, but is not limited to 256 classifiers. The batching improves the training performance by computing values of a multi-dimensional vector with the tag of interest and the classifier for each of the millions of tags.

[0043] In the prediction process, there can be two scenarios: cold start and hot start. In cold start, the given documents are not part of the joint graph used in the training process of the XC processing system. In aspects, approximate nearest neighbor search (ANNS) on top of existing nodes in the joint graph can be used to introduce new documents as recommended labels in response to the given documents. ANNS includes retrieving a selected number of nearest nodes for each document in the given documents and inserting them into the joint graph. Additionally or alternatively, one or more prediction points can be introduced in the joint graph by using auxiliary signals. For example, the joint graph can include information about the relevance between user queries based on query history as an auxiliary signal in the joint graph. In aspects, the shortlist of most likely labels for the given documents can be performed by using ANNS on top of graph label embeddings or on top of graph convolution based document representations.

[0044] It should be understood that the various methods, devices, applications, features, etc. described Figure 1 The various methods, devices, applications, features, etc. described are not intended to limit system 100 to the particular applications and features described, as additional controller configurations can be implemented using the methods and systems herein and / or the described features and applications can be excluded without departing from the methods and systems disclosed herein.

[0045] Figure 2 An example of a system architecture is illustrated in accordance with aspects of the present disclosure. System 200 illustrates the evolution of XC processing for document Dl, including processing by graph convolution generator 202, feature transformer 220, feature label generator (label attention) 240, and classifier learner 250. In particular, system 220 includes processing three different neighborhood levels (i.e., one-hop, two-hop, and three-hop) for generating a multi-dimensional vector representation of document Dl.

[0046] The graph convolutional generator 202 processes the joint graph based on three different levels (e.g., hops) of the highlighted nodes (e.g., nodes highlighted by bolded outlines). Each of the three levels is processed for the document node representing document Dl. For example, in the first hop of the joint graph, Dl connects to recommended label Ll, recommended label L2, and document D2. In the second hop of the joint graph, Ll connects to D3, L2 connects to D4 and D5, and D2 connects to L3. In the third hop of the joint graph, D3 connects to L4 and L5, D4 connects to L6, D5 connects to L7, and L3 connects to L8. In aspects, the joint graph can be generated, for example, from an activity log of a web search application. The activity log can include log entries that traverse from web page Dl to another web page D2. Another log entry can indicate that web page Dl was received as an input query to a web search, resulting in providing a recommended website as recommended label Ll and another recommended website as recommended label L2 as a response, and so on.

[0047] The joint graph 202A illustrates generating a multi-dimensional vector representation of document Dl using only the immediate (e.g., directly related) documents and labels (i.e., up to one hop from Dl) as Doc.Dl 210A (1 hop). The joint graph 202B illustrates generating a multi-dimensional vector representation of document Dl using the immediate documents and labels plus one additional hop of documents and labels (i.e., up to two hops from Dl) as Doc.Dl 210B (2 hops). The joint graph 202C illustrates generating a multi-dimensional vector representation of document Dl using the immediate documents and labels plus two additional hops of documents and labels (i.e., up to three hops from Dl) as Doc.Dl 210C (3 hops). The document representations 210 include three multi-dimensional vector representations of the document (i.e., document representations based on the first hop, the second hop, and the third hop).

[0048] The feature transformer 220 transforms the document representations 210 from the graph convolution into feature-transformed document representations 230. In particular, the feature transformer 220 uses the residual layers of the residual neural networks 220A-C. In aspects, the residual neural networks 220A-C include skip connections for the document representations that are combined with the output of the residual layer(s) to generate the final output. Thus, each document representation from the graph convolution is multiplied by the weight output of the residual layer and then added to the document representation. In this way, the residual network 220A transforms Doc.Dl 210A (1 hop) into TDoc.Dl 230A (1 hop); the residual network 220B transforms Doc.Dl 210B (2 hops) into TDoc.Dl 230B (2 hops); and the residual network 220C transforms Doc.Dl 210C (3 hops) into TDoc.Dl 230C (3 hops).

[0049] The feature tag generator (tag attention) 240 generates a set of per-tag document representations based on the tag attention. In aspects, the number of per-tag document representations corresponds to the number of candidate tags. In Figure 2 In the middle, the tag LI -Doc.Dl 245A represents a document representation for the tag LI. For the tag LI -Doc.Dl 245A, the line 242A connecting TDoc.Dl 230A (1 hop) to the tag LI -Doc.Dl 245A is thicker than the line 242B connecting TDoc.Dl 230B (2 hops) to the tag LI -Doc.Dl 245A, and also thicker than the line 242C connecting TDoc.Dl 230C (3 hops) to the tag LI -Doc.Dl 245A. The joint graph 202A indicates that the tag LI is in a first tier neighborhood based on one hop from Dl.

[0050] Thus in the example, the tag attention (represented by the line 242A) used to generate the per-tag document representation from TDoc.Dl 230A (1 hop) has the greatest weight in the per-tag document representation for the tag LI (i.e., the tag LI -Doc.Dl 245A). Similarly, as illustrated by the relative thickness of the line 244B, TDoc.Dl 230B (2 hops) has the highest weight in generating the tag L3- Doc.Dl 245B. Further, as illustrated by the relative thickness of the line 246C, TDoc.Dl 230C (3 hops) has the greatest weight in generating the tag L7- Doc.Dl 245C. In this way, based on the tag attention, the per-tag document representation for a particular tag can be generated by varying the weighting of different feature-transformed document representations (each based on a different number of hops from the document). In aspects, the varying weights for the tag attention can be determined based at least on the position of the candidate tag in the joint graph (e.g., head or tail tag). In some other aspects, the varying weights for the tag attention can be determined based on an exclusive graph for the candidate tag.

[0051] The classifier learner 250 trains classifiers for labels. The classifiers for labels are trained based on joint learning from the training data and the per-label document representation for labels. For example, the label classifier 1 for label L1 is trained using the per-label document representation label L1 Doc.D1 245A. The joint learning enables the classifier learner 250 to learn the label classifiers with discriminative features for respective labels based on a number of hops from a document in the joint graph. In aspects, the classifier learner 250 trains a subset of the label classifiers. The classifier learner 250 can select the subset of label classifiers to evaluate. In some other aspects, the weights of those label classifiers in the subset can be updated. The prediction process can use ANNS to form a shortlist of selected label classifiers for evaluation.

[0052] Additionally or alternatively, the present technology can use various techniques other than the above-described techniques (e.g., a combination of graph convolution and feature generation using label attention) to select a subset of label classifiers for evaluation. For example, generating the multi-dimensional vector representation of a document can use inductive representation learning of the joint graph (e.g., by using some inductive framework that uses node feature information of a graph to generate node embeddings for data). For example, the inductive representation learning can include sampling and aggregating features of a document and candidate labels for generating the multi-vector representation. In some aspects, the inductive representation learning can be combined with a nearest neighbor algorithm other than ANNS (e.g., an in-memory implementation of ANNS and using a hierarchical navigable small world graph) for retrieving nearest neighbor nodes of a given node.

[0053] Additionally or alternatively, a graph attention network can be used instead of graph convolution and neighborhood label attention based on hops in the joint graph. In aspects, the use of the graph attention network includes assigning different weights to nodes in a neighborhood of a node in the joint graph for generating the multi-dimensional vector representation of a document.

[0054] Figure 3 An example of a data structure is illustrated in accordance with aspects of the present disclosure. The example 300 illustrates an example activity log from a web search application. The web search activity log 302 can be a table including activities and results, where each activity and each result can correspond to a node. The graph 308 is a joint graph of activity nodes and result nodes based on the web search activity log 302.

[0055] As shown, the web search activity log 302 includes two columns. The activities 304 can correspond to input on a starting web query or web page for a query or traversal, and the results 306 can correspond to results or destination web pages. In aspects, the activities 304 can also include web traversals from a web page to another web page to capture relationships between web pages as documents. In aspects, the activities 304 in the left column of the table illustrate web search queries received from a starting web page or user from which the user traverses. Website traversals can be tracked using various tracking tools, e.g., including cookies. The results 306 on the right column of the table include web pages that are responsive to a given search query or web traversal operation. The right column also includes destination web pages that the user traverses to when following a link from a starting web page. Within each parenthesis, an identifier is provided for each query or web page. For example, Dl (310) is an identifier for a document, and LI is an identifier for a recommended label. An edge 312 connects from Dl to LI. In aspects, an operation of traversing from a web page to another web page or following a link from a web page to another web page results in a document in the left column and another document in the right column. In some other aspects, an operation of traversing from a results page of a web query to another results page of a web query can result in the left column having a recommended label and the right column having another recommended label.

[0056] In some aspects, the activities 304 can include only documents on the left column and only labels on the right column. For example, some systems can record activities of receiving a query (i.e., a document) and resulting web pages (e.g., recommended labels). When a user selects a web page as a result of a previous web search (i.e., a recommended label) and searches for related web pages to reach another website, both the web page and the other web pages are indicated as labels.

[0057] The present technology includes generating a joint graph based on the activity log. For example, the activities 304 and the results 306 can correspond to nodes of a joint graph. For example, a search query “What is a banana?” on a query (Dl) results in a recommended web page “Fruit Directory” (LI) as a label. Thus, there is a relationship between Dl and LI. The web search task based on Dl annotates Dl with LI as an associated label by connecting the document node Dl with the label node LI. Similarly, the web search based on Dl as a query results in another recommended web page “Banana World” (L2) as a search result. The classification task also annotates Dl with L2. The activity log also includes an activity of the user selecting “Jump to image search page via this page” on the query page, causing a web page traversal to an image search page (D2). Thus, Dl is associated with D2.

[0058] The training process of the XC processing can include generating a joint graph (e.g., joint graph 308) based on the activity log. The joint graph 308 indicates that node Dl is connected to recommended label LI, recommended label L2, and document D2. LI is connected to D3. L2 is connected to documents D4 and D5, respectively. D2 is connected to D2. D3 is connected to recommended labels L4 and L5. D4 is connected to recommended label L6. D5 is connected to link L7. And, link L3 is connected to L8. The joint graph 308 is an example based on a tree graph structure. In some other aspects, the joint graph can be in a mesh or web form, depending on the relationships between the documents and the recommended labels in the activity log.

[0059] In aspects, the technology uses the number of hops in the joint graph in determining the range of neighboring nodes to consider for a given document or recommended label. For example, in joint graph 308, recommended label LI is one hop away from Dl. Recommended label L2 and document D3 are also one hop away from Dl. Documents D3, D4, D5, and recommended label L3 are two hops away from Dl. Recommended labels L4, L5, L6, L7, and L8 are three hops away from Dl. The graph convolution generator (e.g., graph convolution generator 146 in Figure 1 The graph convolution generator 146 in uses the joint graph 408 to generate a set of multi-dimensional vector representations of document Dl (e.g., document representations of Dl). Each multi-dimensional vector representation of a document can use different number of hops (e.g., one hop, two hops, and three hops) to accommodate information of other documents and recommended labels in the multi-dimensional vector representation.

[0060] Figure 4 An example of a data structure is illustrated in accordance with aspects of the present disclosure. Example 400 illustrates an example activity log from a sponsored search application on how web ads (i.e., targeted content) are associated with queries and web pages. In aspects, the sponsored search activity log 402 is a table that includes activities and results. Graph 408 is a joint graph based on the sponsored search activity log 402.

[0061] The sponsored search activity log 402 includes two columns. The activity 404 can correspond to a web query or input on a web page used to query or traverse, and the result 406 can correspond to a result or destination web page. In aspects, the activity 404 can also include web traversals from a web page to another web page to capture relationships between web pages as documents. In aspects, the activity 404 in the left column of the table illustrates web search queries received from a user or a starting web page from which the user traversed. For example, various tracking tools including cookies can be used to track website traversals. The result 406 on the right column of the table includes directed content (web ads) in response to a given search query or web traversal operation. The right column also includes destination web pages to which the user traversed when following a link from the starting web page. Within each parenthesis, an identifier is provided for each query or web page. For example, Dl is an identifier for a document and LI is an identifier for a label. In aspects, an operation of traversing from a web page to another web page or following a link from a web page to another web page results in a document in the left column and another document in the right column. In some other aspects, an operation of traversing from a result page of a web query to another result page of a web query can result in the left column having a label and the right column having another label.

[0062] The present technology includes generating a joint graph based on the activity log. For example, the activity 404 and the result 406 can correspond to nodes of the joint graph. For example, a search query “What is a banana?” on a query page (Dl) results in directed content or a web ad for a screenshot of “Fresh Fruit Market - Banana Special! (LI)” for a fruit. The directed content is a recommended label for the received query as a document.

[0063] Thus, there is a relationship between Dl and LI. A task of sponsored search based on Dl annotates Dl with LI by connecting the document node Dl with the label node LI. Similarly, a classification task annotates Dl with L2. A web search based on Dl as a query produces another directed content, a web ad for a vitamin store: “Vitamin Supplement Store - Vitamin C (L2).” The activity log also includes an activity of the user selecting “Jump to image search page via image search page” on the query page Dl, causing a web page traversal to another query page “Image Search Top Page (D2).” Thus, Dl is associated with D2.

[0064] The training process of the XC process can include generating a joint graph (e.g., joint graph 408) based on the activity log. The joint graph 408 indicates that node Dl connects to recommended label LI, recommended label L2, and document D2. LI connects to D3. D3 connects to recommended labels L4 and L5. L2 connects to documents D4 and D5. D4 connects to recommended label L6. D5 connects to link L7. D2 connects to link L3. And, link L3 connects to L8.

[0065] Figure 5 FIG. illustrates an example data structure for generating a joint graph according to the present disclosure. In particular, example 500 illustrates an example of joining two different graphs. A label (or class) graph 502 includes labels that are nodes linked as self-rooted label nodes. A candidate label graph 502 depicts relationships between labels. For example, label graph 502 can illustrate relationships between directed content. A document graph 504 includes documents that are nodes linked from a document (e.g., Dl). For example, document graph 502 can depict relationships between web pages in a web search or sponsored search application. An arrow "merge 506" points to a joint graph 508, where a process of merging label graph 502 and document 504 generates joint graph 508 based on document Dl.

[0066] In label graph 502, root node is connected to candidate labels LI and L2. Label LI is connected to candidate labels L4 and L5. Label L2 is connected to labels L3, L6, and L7. Label L3 is connected to label L8. Edges in label graph 502 illustrate relationships between labels that are nodes. For example, labels LI and L2 are head labels. Labels L4, L5, L6, L7, and L8 are tail labels because tail labels do not have an outgoing edge to another label. In aspects, in a label graph, head labels are more "data rich" or contextually rich than tail labels. For example, L8 can be a child label of L3, which is a child label of L2.

[0067] In document graph 504, document Dl (514) is connected to documents D2, D3, D4, D5, and D6. Edges in document graph 504 illustrate relationships between documents that are nodes. Relationships can be based on, but are not limited to, semantic relationships, bibliographic relationships, temporal relationships, etc., based on content of the respective documents.

[0068] Edges 510 and 512 illustrate edges that connect nodes in label graph 502 and document graph 512. Edge 510 connects document Dl in document graph 504 to candidate label L2 in label graph 502 based on a relationship between document Dl and candidate label L2. For example, a previous classification task can have annotated document Dl with label L2 as a recommended label. Edge 512 connects candidate label L2 in candidate label graph 502 to document D5 in document graph 504. For example, web ad L2 can be associated with web page D5 in a sponsored search application. Edges 510 and 512 can be part of a set of other edges that connect between the two graphs. In some aspects, one or more edges can be bidirectional.

[0069] Merge 506 indicates merging the label graph 502 and the document graph 504 to generate a joint graph 508 of the two independent graphs. In the joint table 508, the document Dl is the head node and is connected to, for example, the candidate label LI, the candidate label L2, and the document D3. The example joint table 508 includes candidate labels and documents from two different graphs while maintaining the relationships between the candidate labels and the documents. The present technology uses the joint graph for generating a document representation to train a classifier for candidate labels.

[0070] Figure 6 is an example of a method for training a classifier for candidate labels in XC processing in accordance with aspects of the present disclosure. Figure 6 A general order of the operations for the method 600 is shown in FIG. 6. Generally, the method 600 starts with a start operation 602 and ends with an end operation 624. The method 600 can include more or fewer steps or can Figure 6 The order of the steps can be rearranged differently than shown in FIG. 6. The method 600 can be performed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer readable medium. Further, the method 600 can be performed by gates or circuits associated with a processor, an ASIC, an FPGA, a SOC or other hardware device. Hereinafter, the method 600 shall be explained in reference to the systems, components, devices, modules, software, data structures, data property representations, signaling diagrams, methods, etc., described in conjunction with Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 7 and Figures 8A-8B The method 600 shall be explained hereinafter in reference to the systems, components, devices, modules, software, data structures, data property representations, signaling diagrams, methods, etc., described in conjunction with

[0071] After the start operation 602, the method 600 starts with a receive operation 604 that receives a joint graph of documents and candidate labels. The joint graph can be based on an activity log of an application including, but not limited to, a web search, a sponsored search for providing web advertisements or targeted content, a product recommendation, a movie recommendation, etc. In aspects, the joint graph can be generated based on a combination of a graph indicating relationships between documents and another graph illustrating relationships between candidate labels.

[0072] The generating operation 608 generates document representations based on graph convolution of the joint graph. The document representations are multi-dimensional vector representations of the documents. In aspects, generating the document representations according to the present technology does not require vectorization of the words or content from the documents. Rather, the document representations can include attributes of the documents. In aspects, the generating operation 610 generates a set of document representations for a document by graph convolution based on a varying range of neighboring nodes (e.g., number of hops from the document node). For example, one document representation can be generated based on graph convolution using one hop. Another document representation can be generated based on graph convolution using two hops. Yet another document representation is based on three hops.

[0073] The generating operation 610 generates feature-transformed document representations. The generating operation 610 takes as input the set of graph convolution-based document representations. The generating operation 610 then performs a feature transformation of the document representations using a residual layer (e.g., neural network) with skip connections of the document representations. The generating operation 610 generates a set of feature-transformed document representations for each of the respective levels (or hops) of neighboring nodes in the joint graph.

[0074] The generating operation 612 generates per-label document representations based on label attention. The label attention determines different weights for focusing on each document representation, each document representation based on a particular range (i.e., hop) of neighboring nodes in the joint graph. For example, a candidate label that is two hops away from the document incurs label attention on the feature-transformed document representation based on two hops. In this way, information about the document or candidate label in the subsequent layers (or hops) up to two hops away from the document node is captured in the vector representation. Thus, different amounts of information are considered based on the location of the candidate label in the joint graph when generating the per-label document representations. For example, a head label in the graph is less focused on the hops than a tail label. The head label inherently has rich data due to the outgoing edges from the head label. In contrast, the tail label has less information due to being at the end of the graph.

[0075] The learning operation 614 learns the classifiers. In aspects, there is one classifier for each candidate label. The learning process can be joint learning based on training data and the per-label document representations. By incorporating the per-label document representations for training the label classifiers, the learning operation 614 enables learning of the respective classifiers to be more distinguishable from one another.

[0076] In aspects, the training process of the XC processing according to the present technology outperforms some conventional systems in performance. Unlike some conventional systems that use individual words or content from documents to learn classifiers for candidate labels, the present technology learns classifiers based on a joint graph (including both documents and candidate labels) rather than by analyzing the text or content of the documents. The amount of data processed for training millions of candidate labels based on document representations from the joint graph can be less than the amount of data required to analyze the content for each document. The training process based on the joint graph provides better scalability to train an extremely large number of candidate labels (i.e., millions of candidate labels) compared to training processes based on multiple different graphs.

[0077] The deployment operation 616 deploys the trained classifiers for use in the prediction process of the XC processing system. In aspects, the set of classifiers can be installed in an extreme classification server. There can be one classifier for each candidate label.

[0078] The receiving operation 618 receives a query as a document in the prediction process of the XC processing. The document can be in different forms for different applications. For example, the document can represent a web search query or a web page in a web search application or a sponsored search application. Additionally or alternatively, the document can represent a product identifier or product information in a product recommendation application. Further, the document can be a movie title in a movie recommendation application. The common purpose of receiving the document is for the XC processing to determine one or more labels as the most relevant recommendations based on the millions of candidate labels in response to the received document.

[0079] The generating operation 620 generates a set of recommended labels from the millions of candidate labels using the trained classifiers. In the case of cold start, where the received document is not in the joint graph used to train the classifiers, the generating operation 620 can identify a set of documents by using ANNS on top of existing nodes in the joint graph. To efficiently process the evaluation of the trained classifiers, the generating operation 620 can use ANNS on the graph convolution-based document representation and recall the highest predictions of recommended labels for a given document. In aspects, the graph convolution-based document representation can be an average multi-dimensional vector representation of the document representation across multiple ranges of neighboring nodes (e.g., one hop, two hops, and three hops).

[0080] In aspects, the generating operation 620 can also determine the most likely set of recommended labels for a given document by performing an approximate nearest neighbor search (ANNS) over the set of recommendation candidates (e.g., candidate labels). The ANNS can be performed by generating a multi-dimensional vector representation of the candidate labels using graph convolution (i.e., residual layers). That is, the ANNS uses the multi-dimensional vector representation of the candidate labels to recall the highest predictions of recommended labels by forming a shortlist of possible labels. In this way, the prediction process does not need to use all millions of classifiers to annotate a given document with recommended labels. The shortlist technique of selecting classes effectively improves performance, especially at cold start when the given document is not in the union tree used to train the classifiers.

[0081] In aspects, the generating operation 620 generates the highest predicted set of recommendations for a label based on a classifier trained according to the training data and per-label document representation joint from the joint graph without needing to analyze the verbatim content of the document and the candidate labels to determine the recommended labels. The prediction process that does not utilize verbatim analysis of content but rather the relationships between the document and the candidate labels makes the prediction process scalable without a significant loss in performance.

[0082] The sending operation 622 sends the results of the prediction process of the XC process in response to the received document. The sending operation 622 can send the results to a client device for viewing. The results of the prediction process can be one or more recommended labels. Depending on the application, the one or more recommended labels can take different forms. For example, the results of a web search can be a set of websites in response to the received web search. The results of a sponsored search can include one or more web advertisements or targeted content in response to a web search query. The results of a product recommendation query or a given product can include a set of products (each corresponding to a recommended label) related to the product recommendation query or the given product. The results of a movie recommendation can include a set of movie titles related to a given movie query or movie title. In aspects, the method 600 can end with an ending operation 624.

[0083] As should be appreciated, the operations 602-624 are described for purposes of illustrating the present methods and systems and are not intended to limit the disclosure to a particular sequence of steps, e.g., the steps can be performed in different order, additional steps can be performed, and disclosed steps can be excluded without departing from the present disclosure.

[0084] Figure 7is a block diagram illustrating the physical components (e.g., hardware) of a computing device 700 with which aspects of the disclosure can be practiced. The computing device components described below can be suitable for the computing devices described above. In a basic configuration, computing device 700 can include at least one processing unit 702 and system memory 704. Depending on the configuration and type of computing device, system memory 704 can comprise, but is not limited to, volatile (e.g., random access memory (RAM)), non-volatile (e.g., read-only memory (ROM)), flash memory, or any combination. System memory 704 can include operating system 705 and one or more program tools 706 suitable for implementing aspects disclosed herein. For example, operating system 705 can be suitable for controlling the operation of computing device 700. Additionally, aspects of the disclosure can be implemented in conjunction with a graphics library, other operating systems, or any other application program and are not limited to any particular application or system. This basic configuration is illustrated in Figure 7 by those components within dashed line 708 in FIG. 7. Computing device 700 can have additional features or functionality. For example, computing device 700 can also include additional data storage devices (removable and / or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated in Figure 7 by removable storage 709 and non-removable storage 710.

[0085] As stated above, a number of program tools and data files can be stored in the system memory 704. While executing on the at least one processing unit 702, the program tools 706 (e.g., an application 720) can perform processes including, for example, aspects described herein. Aspects of the disclosure can also be practiced in conjunction with a user and / or web TV, which can be Figure 1 described in more detail below. The application 720 includes a graph convolutional generator 722, a feature transformer 724, a label attention generator 726, a classifier learner 728, and a document classifier 730, as described in more detail below. Other program tools that can be used in accordance with aspects of the disclosure can include electronic mail and contacts applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided applications, etc.

[0086] Additionally, aspects of the disclosure can be practiced in an electrical circuit comprising discrete electronic elements, packaged or integrated electronic chips containing logic gates, a circuit utilizing a microprocessor, or by combinations of them. For example, aspects of the disclosure can be practiced via a system-on-a-chip (SOC) where Figure 7Each of the components or groups of components illustrated in the diagram can be integrated on a single integrated circuit. Such a SOC device can include one or more processing units, graphics units, communications units, system virtualization units and various application functionality all of which are integrated (or "burned") onto the chip substrate as a single integrated circuit. When operating via a SOC, the functionality described herein with regard to the capabilities of the client switching protocol can be operated via application-specific logic integrated with other components of the computing device 700 on the single integrated circuit (chip). Other technologies can also be used to implement aspects of the present application such as, for example, mechanical, optical, fluidic, quantum and electro-magnetic technologies; new technology can still be developed that will change the use of such technology. Still further, aspects of the present application can be implemented in

[0087] The computing device 700 can also have one or more input device(s) 712 such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or swipe input device, etc. One or more output device(s) 714 such as a display, speakers, a printer, etc. can also be included. The aforementioned devices are examples and others can be used. The computing device 700 can include one or more communication connections 716 allowing communications with other computing devices 750. Examples of suitable communication connections 716 include, but are not limited to, a

[0088] The term computer readable media as used herein can include computer storage media. Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, or program tools. The system memory 704, the removable storage device 709, and the non-removable storage device 710 are all computer storage media examples (e.g., memory storage). Computer storage media can include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other article of manufacture which can be used to store information and which can be accessed by the computing device 700. Any such computer storage media can be part of the computing device 700. Computer storage media does not include a carrier wave or other propagated or modulated data signal.

[0089] Communication media can be embodied by computer readable instructions, data structures, program tools or other data in modulated data signals, such as carrier waves or other transport mechanisms, and includes any information delivery media. The term "modulated data signal" can describe a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0090] Figure 8A and Figure 8B A computing device or mobile computing device 800, such as a mobile telephone, a smart phone, a wearable computer (e.g., a smart watch), a tablet computer, a laptop computer, etc., that can implement aspects of the present disclosure is illustrated. In some aspects, a client utilized by a user (e.g., a user of the client device 102 in Figure 1 FIG. 1, can be a mobile computing device. Referring to Figure 8A , one aspect of a mobile computing device 800 for implementing aspects is illustrated. In a basic configuration, the mobile computing device 800 is a handheld computer having both input elements and output elements. The mobile computing device 800 typically includes a display 805 and one or more input buttons 810 that allow the user to enter information into the mobile computing device 800. The display 805 of the mobile computing device 800 can also function as an input device (e.g., a touch screen display). If included, an edge input element 815 provides further user input. The edge input element 815 can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, mobile computing device 800 can incorporate more or less of each of the input elements. For example, in some aspects, the display 805 can not be a touch screen in

[0091] Figure 8Bis a block diagram illustrating an architecture of one aspect of a computing device, a server (e.g., application server 110 and classifier generator 140 in FIG. 1), a mobile computing device, and the like. That is, mobile computing device 800 can incorporate a system 802 (e.g., a system architecture) to implement some aspects. System 802 can be implemented as a "smart phone" capable of running one or more applications (e.g., browser, e-mail, calendaring, contact managers, messaging clients, games, and media clients / playback Figure 1

[0092] One or more application programs 866 can be loaded into the memory 862 and run on or in association with the operating system 864. Examples of the application programs include phone dialer programs, e-mail programs, information management (PIM) programs, word processing programs, spreadsheet programs, Internet browser programs, messaging programs, and so forth. The system 802 also includes a non-volatile storage area 868 within the memory 862. The non-volatile storage area 868 can be used to store persistent information that should not be lost if the system 802 is powered down. The application programs 866 can use and store information in the non-volatile storage area 868, such as e-mail or other messages used by an e-mail application, and the like. A synchronization application (not shown) also resides on the system 802 and is programmed to interact with a corresponding synchronization application resident on a host computer to keep the information stored in the non-volatile storage area 868 synchronized with the information stored on the host computer. As should be appreciated, other applications can be loaded into the memory 862 and run on the mobile computing device 800 described herein.

[0093] The system 802 has a power supply 870, which can be implemented as one or more batteries. The power supply 870 might further include an external power source, such as an AC adapter or a powered docking cradle that supplements or recharges the batteries.

[0094] The system 802 can also include a radio interface layer 872 that works with the radio interface 854 to facilitate wireless communication between the system 802 and the "outside world." The radio interface layer 872 facilitates wireless communication between the system 802 and devices other than the network 850. This enables the system 802 to serve as a terminal for wireless devices. Such devices

[0095] ​A visual indicator 820 (e.g., an LED) can be used to provide visual notifications, and / or an audio interface 874 can be used to generate audible notifications via an audio transducer 825. In the illustrated configuration, the visual indicator 820 is a light-emitting diode (LED), and the audio transducer 825 is a speaker. These devices can be directly coupled to a power supply 870 such that when activated, they remain on for a duration specified by the notification mechanism, even if the processor 860 and other components may be turned off to conserve battery power. The LED can be programmed to remain on indefinitely until the user takes action to indicate the device's power-on status. The audio interface 874 is used to provide and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 825, the audio interface 874 can also be coupled to a microphone to receive audible input, such as to support telephone conversations. According to various aspects of this disclosure, the microphone can also be used as an audio sensor to facilitate control of notifications, as described below. System 802 may also include a video interface 876, which enables the operation of the vehicle camera 830 to record still images, video streams, etc.

[0096] The mobile computing device 800 implementing system 802 may have additional features or functions. For example, the mobile computing device 800 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. Such additional storage... Figure 8B The diagram shows a non-volatile storage region 868.

[0097] As described above, data / information generated or captured by mobile computing device 800 and stored via system 802 can be stored locally on mobile computing device 800, or the data can be stored on any number of storage media that can be accessed by the device via radio interface layer 872 or via a wired connection between mobile computing device 800 and a separate computing device associated with mobile computing device 800 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information can be accessed via mobile computing device 800 via radio interface layer 872 or via a distributed computing network. Similarly, such data / information can be easily transferred between computing devices for storage and use according to known data / information transmission and storage means (including email and collaborative data / information sharing systems).

[0098] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure as claimed in any way. Such aspects, examples and specifics provided in this application are intended to be illustrative rather than restrictive. The described and illustrated aspects, examples and specifics were chosen and described to explain the principles of the disclosure and its best mode of operation. The scope of the disclosure is not to be construed as limited to the aspects, examples and specifics provided in this application. Various features (structural and methodological) are intended to be selectively included or omitted in various embodiments of the claimed disclosure. Having thus described the application in terms of its aspects and implementations, what is claimed is intended to cover variations, modifications and equivalent structures and processes falling within the spirit and scope of the application, as more fully defined in the following claims.

[0099] The present disclosure relates to systems and methods for training a classifier according to at least the examples provided in the following sections:

[0100] (A1) In one aspect, some embodiments include a computer-implemented method (e.g., 600) of training a classifier, the method comprising: receiving (604) a joint graph (e.g., 144, 308), wherein the joint graph comprises nodes (e.g., 310) and directed edges (e.g., 312), and wherein each node is either a document type or a label type; generating (608), based on a graph convolution of the joint graph, a plurality of multi-dimensional vector representations (e.g., 210A-C) of a document (e.g., 310), wherein the document is represented by a document node in the joint graph (e.g., 144, 308), and wherein each multi-dimensional vector representation (e.g., 210A) of the document corresponds to a different level of neighboring nodes from the document node in the joint graph; and generating (610), using a residual layer of a neural network (e.g., 220A-C), a transformed multi-dimensional vector representation (e.g., 230A-C) of the document based on the generated plurality of multi-dimensional vector representations (e.g., 210A-C) of the document, wherein the residual layer comprises a residual network and a skip connection (e.g., 220A-C). The method further comprises generating (612), using a label attention (e.g., 242A-C, 244A-C, 246A-C), a plurality of per-label multi-dimensional vector representations (e.g., 245A-C) of the document, wherein each per-label multi-dimensional vector representation of the document corresponds to a label of the plurality of labels, and wherein the label attention applies a variable frit (e.g., 242A-C, 244A-C, 246A-C) to one or more of the transformed multi-dimensional vector representations (e.g., 230A-C) of the document to generate the per-label multi-dimensional vector representations (e.g., 245A-C); training (614) a classifier (e.g., 252A-D) for each label of the plurality of labels; and sending (616) the plurality of trained classifiers (e.g., 252A-D) for deployment in a label prediction process.

[0101] (A2) In some embodiments of the method of Al, the level of neighboring nodes from the document node corresponds to a hop number from the document node of the joint graph, and the label attention for a tail label applies higher weighting to a multi-dimensional vector representation of the document that is generated based on a higher level of neighboring nodes.

[0102] (A3) In some embodiments of the method of Al or A2, the plurality of multi-dimensional vector representations of the document includes a first multi-dimensional vector representation (e.g., vector 210A) based on nodes within one hop from the document node, a second multi-dimensional vector representation (e.g., vector 210B) based on nodes within two hops from the document node, a third multi-dimensional vector representation (e.g., vector 210C) based on nodes within three hops from the document node.

[0103] (A4) In some embodiments of the method of Al to A3, the graph convolution includes vectorizing the document and features of the labels associated with nodes within a predetermined number of hops from the document node to generate a multi-dimensional vector representation of the document.

[0104] (A5) In some embodiments of the method of Al to A4, the number of the plurality of labels is at least one million, and the number of the plurality of per-label multi-dimensional vector representations of the document is at least one million.

[0105] (A6) In some embodiments of the method of Al to A5, the method further includes deploying (616) the plurality of trained classifiers for use in a label prediction process, receiving a document, generating (620) a set of recommended candidate labels for the received document using a subset of the plurality of trained classifiers, and sending the generated recommended candidate labels as a response (622) to the document.

[0106] (A7) In some embodiments of the method of Al to A6, the document includes at least one of: a query in a web search application, a query in a sponsored search application, a product in a product recommendation application, or a content title in a content recommendation application.

[0107] (A8) In some embodiments of the method of Al to A7, each label in the plurality of labels includes one of: a web page as a search result in a web search application, a targeted content snippet in a sponsored search application, another product in a product recommendation application, another content title in a content recommendation application.

[0108] In another aspect, some embodiments include a system (e.g., system 802) having one or more processors and memory storing computer-executable instructions that, when executed by the processor, cause the system to perform any of the methods described herein (e.g., methods Al to A8).

[0109] In another aspect, some embodiments include a computer-readable non-transitory recording medium (e.g., system memory 704) storing computer-executable instructions that, when executed by a processor, cause a computer system to perform any of the methods described herein (e.g., methods Al to A8).

[0110] Any of the one or more above aspects in combination with any of the other aspects. Any of the one or more aspects as described herein.

Claims

1. A computer-implemented method of training a plurality of classifiers, the method comprising: receiving a joint graph, wherein the joint graph comprises nodes and directed edges, and wherein each node is either a document type or a label type, and the graph comprises document nodes and label nodes; generating a plurality of multi-dimensional vector representations of a document based on graph convolution of the joint graph, wherein the document is represented by a document node in the joint graph, and wherein each multi-dimensional vector representation of the document corresponds to a different level of neighboring nodes from the document node in the joint graph, wherein the plurality of multi-dimensional vector representations of the document comprises a first multi-dimensional vector representation based on nodes within one hop from the document node, a second multi-dimensional vector representation based on nodes within two hops from the document node, a third multi-dimensional vector representation based on nodes within three hops from the document node; generating a transformed multi-dimensional vector representation of the document based on the generated plurality of multi-dimensional vector representations of the document using a residual layer of a neural network, wherein the residual layer comprises a residual network and a skip connection; generating a plurality of per-label multi-dimensional vector representations of the document using label attention, wherein each per-label multi-dimensional vector representation of the document corresponds to a label of a plurality of labels, and wherein the label attention applies variable weighting to one or more of the transformed multi-dimensional vector representations of the document to generate the per-label multi-dimensional vector representations; training a plurality of classifiers, each classifier of the plurality of classifiers for one label of the plurality of labels; and sending the trained plurality of classifiers for deployment in a label prediction process.

2. The computer-implemented method of claim 1, wherein the levels of neighboring nodes from the document node correspond to a number of hops from the document node in the joint graph, and wherein the label attention for a tail label applies higher weighting to multi-dimensional vector representations of the document generated based on higher levels of neighboring nodes. multiplying each multi-dimensional vector representation by a weight output of a residual layer and adding to the multi-dimensional vector representation for each multi-dimensional vector representation.

3. The computer-implemented method of claim 1, wherein generating the transformed multi-dimensional vector representation comprises: vectorizing features of a document and labels associated with nodes within a predetermined number of hops from the document node to generate the multi-dimensional vector representations of the document.

4. The computer-implemented method of claim 1, wherein the graph convolution comprises:

5. The computer-implemented method of claim 1, wherein a number of the plurality of labels is at least one million, and wherein a number of the plurality of per-label multi-dimensional vector representations of the document is at least one million.

6. The computer-implemented method of claim 1, further comprising: deploying the trained plurality of classifiers for a label prediction process; receiving a document; using a subset of the trained plurality of classifiers to generate a set of recommended candidate labels for the received document; and sending the generated recommended candidate labels as a response to the document.

7. The computer-implemented method of claim 1, wherein the document comprises at least one of: a query in a web search application, ​ a query in a sponsored search application, a product in a product recommendation application, or a content title in a content recommendation application.

8. The computer-implemented method of claim 7, wherein each tag of the plurality of tags comprises one of: a web page that is a search result in the web search application, a targeted content snippet in the sponsored search application, another product in the product recommendation application, or another content title in a content recommendation application.

9. A system for learning classifiers for tags in extreme classification processing, the system comprising: a processor; and a memory storing computer-executable instructions that, when executed by the processor, cause the system to: receive a joint graph, wherein the joint graph comprises nodes and directed edges, and wherein each node is either a document type or a tag type, and the graph comprises document nodes and tag nodes; generate a plurality of multi-dimensional vector representations of a document based on graph convolution of the joint graph, wherein the document is represented by a document node in the joint graph, and wherein each multi-dimensional vector representation of the document corresponds to a different level of neighboring nodes from the document node in the joint graph, wherein the plurality of multi-dimensional vector representations of the document comprises a first multi-dimensional vector representation based on nodes within one hop from the document node, a second multi-dimensional vector representation based on nodes within two hops from the document node, a third multi-dimensional vector representation based on nodes within three hops from the document node; generate a transformed multi-dimensional vector representation of the document based on the generated plurality of multi-dimensional vector representations of the document using a residual layer of a neural network, wherein the residual layer comprises a residual network and a skip connection; generate a plurality of per-tag multi-dimensional vector representations of the document using tag attention, wherein each per-tag multi-dimensional vector representation of the document corresponds to a tag of a plurality of tags, and wherein the tag attention applies variable weighting to one or more of the transformed multi-dimensional vector representations of the document to generate the per-tag multi-dimensional vector representations; train a plurality of classifiers, each classifier of the plurality of classifiers for one tag of the plurality of tags; and send the trained plurality of classifiers for deployment in a tag prediction process.

10. The system of claim 9, wherein the levels of neighboring nodes from the document node correspond to a number of hops from the document node of the joint graph, and wherein the tag attention for a tail tag applies higher weighting to multi-dimensional vector representations of the document generated based on higher levels of neighboring nodes.

11. The system of claim 9, the computer-executable instructions, when executed, further cause the system to: deploy the trained plurality of classifiers for a tag prediction process; receive a document; use a subset of the trained plurality of classifiers to generate a set of recommended candidate tags for the received document; and send the generated recommended candidate tags as a response to the document. ​ ​ 12. The system of claim 9, wherein the document comprises at least one of: a web page as a search result in a web search application, a targeted content snippet in a sponsored search application, another product in a product recommendation application, or another content title in a content recommendation application, and wherein the tag comprises at least one of: a query in the web search application, a query in the sponsored search application, a product in the product recommendation application, or a content title in the content recommendation application.

13. A computer-readable non-transitory recording medium storing computer- executable instructions that, when executed by a processor, cause a computer system to: receive a joint graph, wherein the joint graph comprises nodes and directed edges, and wherein each node is either a document type or a tag type, and the graph comprises document nodes and tag nodes; generate a plurality of multi-dimensional vector representations of a document based on graph convolution of the joint graph, wherein the document is represented by a document node in the joint graph, and wherein each multi-dimensional vector representation of the document corresponds to a different level of neighboring nodes from the document node of the joint graph, wherein the plurality of multi-dimensional vector representations of the document comprises a first multi-dimensional vector representation based on nodes within one hop from the document node, a second multi-dimensional vector representation based on nodes within two hops from the document node, a third multi-dimensional vector representation based on nodes within three hops from the document node; generate a transformed multi-dimensional vector representation of the document based on the generated plurality of multi-dimensional vector representations of the document using a residual layer of a neural network, wherein the residual layer comprises a residual network and a skip connection; generate a plurality of per-tag multi-dimensional vector representations of the document using tag attention, wherein each per-tag multi-dimensional vector representation of the document corresponds to a tag of a plurality of tags, and wherein the tag attention applies variable weighting to one or more of the transformed multi-dimensional vector representations of the document to generate the per-tag multi-dimensional vector representations; train a plurality of classifiers, each classifier of the plurality of classifiers for one tag of the plurality of tags; and send the trained plurality of classifiers for deployment in a tag prediction process.

14. The computer-readable non-transitory recording medium of claim 13, wherein the levels of neighboring nodes from the document node correspond to a number of hops from the document node of the joint graph, and wherein the tag attention for a tail tag applies higher weighting to a multi-dimensional vector representation of the document generated based on a higher level of neighboring nodes.

15. The computer-readable non-transitory recording medium of claim 13, the computer- executable instructions, when executed, further cause the system to: deploy the trained plurality of classifiers for a tag prediction process; receive a document; generate a set of recommended candidate tags for the received document using a subset of the trained plurality of classifiers; send the generated recommended candidate tags as a response to the document. ​ ​

Images