Classifying Data Objects Using Neighborhood Representations

Abstract

Methods, systems, and apparatus for classifying data objects, including a computer program encoded on a computer storage medium. One method includes maintaining a dataset including reference data objects, each having one or more labels, one or more features, or both, receiving a request to add a new data object to the dataset, the new data object having the one or more features but missing one or more labels, selecting N neighboring data objects based on similarity scores of the neighboring data objects with respect to the new data object, generating a neighborhood feature vector for the new data object, processing the neighborhood feature vector using a machine learning model to predict one or more labels for the new data object, and updating the dataset to include the new data object and to associate the one or more predicted labels with the new data object.

Description

[Technical field]

[0001] This specification relates to classification of data objects using machine learning and neural networks. [Background technology]

[0002] A machine learning model receives inputs and generates outputs based on the received inputs and the values ​​of the parameters of the model. For example, a machine learning model receives an image and generates a score for each set of classes. The score for a given class represents the probability that the image contains an image of an object belonging to that class.

[0003] A machine learning model may, for example, be composed of a single level of linear or nonlinear operations, or may be a deep network, e.g., a machine learning model composed of multiple levels, one or more of which may be layers of nonlinear operations. One example of a deep network is a neural network with one or more hidden layers. Summary of the Invention

[0004] This specification describes a data object classification system that receives a data object and generates a classification for the data object. In particular, the system accesses a dataset that stores reference data objects associated with known categories, and generates a classification for the data object using the known categories of similar reference data objects in the vicinity of the data object.

[0005] In general, one innovative aspect of the subject matter described herein includes maintaining a dataset including reference data objects each having one or more labels, one or more features, or both, where each label of the reference data objects defines a respective category of the reference data object and each feature of the reference data object describes a characteristic of the reference data object; receiving a request to add a new data object to the dataset, the new data object (i) having the one or more features but (ii) missing one or more labels; and selecting N neighboring data objects from the reference data objects based on a similarity score of the neighboring data objects with respect to the new data object, where the similarity score of each neighboring data object is greater than or equal to the similarity score of the new data object. The method may be embodied in a method including the operations of: selecting, based on one or more features of the new data object and one or more features of the neighboring data objects, N being a natural number greater than or equal to one; for each neighboring data object in the N neighboring data objects, generating a neighborhood feature vector for the new data object using (i) the one or more labels of the neighboring data objects and (ii) a similarity score of the neighboring data object to the new data object; processing the neighborhood feature vector using a machine learning model to predict one or more labels that are missing for the new data object; and updating the dataset to include the new data object and to associate the one or more predicted labels with the new data object.

[0006] Other implementations of the present aspect include corresponding apparatus, systems, and computer programs encoded on a computer storage device and configured to perform aspects of the method. Each of these and other implementations may optionally include one or more of the following features.

[0007] In some aspects, maintaining a dataset including the reference data objects includes maintaining data describing a heterogeneous graph including nodes connected by edges, where each node corresponds to a different reference data object in the reference data object, and where each edge represents a relationship between two nodes connected by the edge.

[0008] In some aspects, maintaining the dataset including the reference data objects includes maintaining the reference data objects and associated labels and features of the reference data objects in a relational dataset.

[0009] In some aspects, the operations further include determining a similarity score for each neighboring data object with respect to the new data object based on one of Euclidean distance or cosine similarity in the embedding space.

[0010] In some aspects, the operations further include determining a similarity score for each neighboring data object with respect to the new data object based on one of a Proper Mutual Information (PMI) score or a bipartite score.

[0011] In some embodiments, the machine learning model comprises one of a neural network, a logistic regression model, a support vector machine (SVM), or a decision tree or random forest model.

[0012] In some aspects, generating a neighborhood feature vector for the new data object includes determining a concatenation of respective similarity scores of a subset of N neighboring data objects having a particular category among the respective categories defined by the one or more labels.

[0013] In some aspects, a subset of the N neighboring data objects includes neighboring data objects that each have a first label that defines a trusted category, eg, a positive label.

[0014] In some aspects, a subset of the N neighboring data objects includes neighboring data objects that each have a second label that defines an untrusted category, eg, a negative label.

[0015] As used herein, a data object having a positive label that defines a trusted category refers to a data object that corresponds to a category that satisfies a trusted condition (e.g., satisfies a trust threshold). A data object having a positive label may be "allowed-listed", whereby a computer system is permitted to use, e.g., manipulate or otherwise process, the data object. A data object having a negative label that defines an untrusted category refers to a data object that corresponds to a category that does not satisfy a trusted condition (e.g., does not satisfy a trust threshold). An untrusted data object may be "block-listed", whereby a computer system may not be permitted to use the data object.

[0016] In some aspects, each data object represents an image, video, audio, text, or a web page.

[0017] In some embodiments, the value of N depends on the total number of labels that each reference data object has.

[0018] Certain embodiments of the subject matter described herein may be implemented to achieve one or more of the following advantages: By configuring a data object classification system to predict the label of an input data object based on a neighborhood representation of the input data object determined from a dataset of reference data objects maintained by the system, both the accuracy of the label prediction and the inference speed of the prediction process may be improved. In particular, label prediction may be performed in real time, even for dynamic data objects that are frequently modified over time, and at a very early stage in the entire life cycle of the data object (e.g., within seconds or milliseconds of the data object becoming available). Label prediction may also be performed using fewer memory resources than other approaches, since it avoids the processing of high-dimensional categorical features, which are typically stored as sparse high-dimensional vectors using one-hot encoding. Instead, the label may be predicted based on a neighborhood representation maintained in the form of a dense vector that can be efficiently stored and retrieved from memory.

[0019] The neighborhood representation of the input data object may be generated by encoding the similarity scores (for the input data object), known labels, and feature information, etc., of a subset of the neighboring reference data objects of the input data object selected from the dataset into a structured dense vector format, which is a data-efficient and information-rich representation that facilitates classification of the input data object using any of a variety of known classifiers, as well as other downstream tasks involving processing of the representation. Furthermore, the interpretability of the data object classification system is improved since the generation process of the neighborhood representation can be traced, thereby making it easier to examine the context of the classification. In the scope of many real-world production environments, this allows for understanding how a particular label was assigned to an input data object, for example, how the system predicted that a web page was a phishing or malicious web page, or how the system predicted that an image contained forbidden content. This allows for addressing potential reliability issues that manifest as concerns about the decisions that led to a particular outcome.

[0020] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0021] [Figure 1] 1 illustrates an example of a data object classification system. [Diagram 2] 1 illustrates an example of a heterogeneous graph representing reference data objects. [Diagram 3] FIG. 1 is a flow diagram of an example process for generating one or more labels for an input data object. [Figure 4] FIG. 1 is a block diagram of an example computer system that can be used to perform the described operations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Like reference symbols and designations in the various drawings indicate like elements.

[0023] This specification describes a data object classification system that receives a data object and generates a classification for the data object.

[0024] 1 illustrates an example of a data object classification system 100. Data object classification system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below may be implemented.

[0025] The data object classification system 100 generates label data for a given data object, e.g., label data 142 for a new data object 102. Each data object has one or more features that each describe a respective characteristic, attribute, or property of the data object. The label data for a given data object identifies one or more category labels for the data object. Each of the category labels is a label for a category within a distinct (e.g., finite) set of possible categories to which the data object classification system 100 has determined the data object belongs. Generally, the category labels for a given category are terms that identify or describe the category. For example, the category of an image may include the topic of the content depicted in the image, or the object detected in the image.

[0026] The data object classification system 100 may be configured to generate label data for various data objects, for example, any type of data object that can be classified as belonging to one or more categories, each of which is associated with a respective label. Once the label data is generated, the data object classification system 100 may use the label data in any of a variety of ways.

[0027] For example, if the data object is an image, the data object classification system 100 may be a visual recognition system that determines whether an input image contains an image of an object that belongs to an object category from a predefined set of object categories. In this example, the label data for a given input image identifies one or more labels for the input image, each label labeling a respective object category to which an object depicted in the image belongs. In this example, the features of the data object may include pixel intensity values, edges, corners, ridges, interest points, and color histograms of the image, possible metadata associated with the image, etc.

[0028] As another example, where the data object is a video or a portion of a video, the data object classification system 100 may be a video classification system that determines which topic or topics an input video or portion of a video is related to. In this example, the label data for a given video or video portion identifies one or more labels for the video or video portion, each label identifying a respective topic to which the video or video portion is related. In this example, the features of the data object may include intensity values ​​of pixels for each frame of the video, visual features of the frames, and possible metadata associated with the video.

[0029] As another example, if the data object is audio data, the data object classification system 100 may be a speech recognition system that determines, for a given utterance, the term that the utterance represents. In this example, the label data for the given utterance identifies one or more labels, each label being a term represented by the given utterance. In this example, the features of the data object may include time domain and frequency domain features, such as energy envelope and distribution, frequency content, harmonicity, pitch, etc.

[0030] As another example, if the data object is text data, the data object classification system 100 may be a text classification system that determines to which topic or topics an input text segment is associated. In this example, the label data for a given text segment identifies one or more labels for the text segment, each label identifying a respective topic to which the text or text segment is associated. In this example, the features of the data object may include text features obtained through any suitable natural language processing technique, such as lexical analysis, pattern recognition, sentiment analysis, etc.

[0031] As another example, if the data objects are web pages, the data object classification system 100 may be a web page classification system that determines which category a given web page (or the content of a given web page) belongs to. In this example, the label data for a given web page identifies one or more labels, each label identifying a respective category to which the web page belongs. In this example, the data object features may include URL features (e.g., hostname, usage, subdomains, top level domain, URL length, and presence of HTTPS), domain features (e.g., domain registration period, age of the domain, domain name owner, associated domain registrar, website traffic, and DNS records), website code features (e.g., presence or absence of website redirection and popup window usage), website content features (e.g., hyperlinks to target website, hyperlinks to non-target websites, media content, and text input fields), website visitor features (e.g., IP address, visit time, and session duration).

[0032] In these examples, appropriate actions may be taken by the system on the data object, such as, for example, providing a label to a user of the system (e.g., in response to a request / search query submitted by the user), forwarding the data object, the label, or both to another system for additional processing, flagging for deeper review in a human review process, using the labeled data object in a map / navigation context, etc.

[0033] The data object classification system 100 includes a nearby data object selection engine 110 , a dataset of reference data objects 120 , a nearby feature vector generation engine 130 , and a classification subsystem 140 .

[0034] The dataset 120 of reference data objects may be a labeled data object database (or other suitable data structure) that stores the reference data objects in association with the known label data of the reference data objects, e.g., in a data storage device and / or cloud-based storage. The dataset 120 of reference data objects may be any data store suitable for storing a plurality of reference data objects and the labels and features associated with the plurality of reference data objects. In some implementations, the dataset 120 may be a relational database having a multi-dimensional structure, where each feature of a data object is stored in a cell of the relational database defined by rows and columns. In an example where each data object represents a web page, the relational database may store relationships between features of web pages with different labels. For example, the relational database may store relationships between the geographic locations of web pages with a particular label, the content contained in web pages with a particular label, the layout of web pages with a particular label, and the features available on web pages with a particular label, and / or combinations thereof.

[0035] In some implementations, the dataset 120 may be a heterogeneous graph including multiple nodes and edges connecting the nodes, both of which are of heterogeneous types. The heterogeneous graph may represent relationships between reference data objects, and the heterogeneous graph may be stored in one or more data stores. Each node in the heterogeneous graph represents a different reference data object (nodes of the same type represent reference data objects of the same type), and pairs of reference data objects in the heterogeneous graph are connected by edges that indicate a relationship between the two reference data objects represented by the pair of nodes.

[0036] 2 illustrates an example of a heterogeneous graph 200 that represents reference data objects. The heterogeneous graph 200 identifies relationships between the reference data objects.

[0037] For example, the heterogeneous graph 200 includes nodes 202 and 204 representing a first reference data object, nodes 206 and 208 representing a second reference data object, and node 210 representing a third reference data object. The first, second, and third reference data objects may be of different types from each other. For example, nodes 202 and 204 may each represent a web domain, and nodes 206 and 208 may each represent an IP address. An edge connecting a pair of nodes indicates that some measure of association exists between the reference data objects represented by the connected pair of nodes. For example, nodes 202 and 206 are connected by edge 212, indicating that the web domain represented by node 202 is hosted at the IP address represented by node 206.

[0038] Returning to FIG. 1, to classify the new data object 102, the nearby data object selection engine 110 selects a suitable subset of nearby data objects 122 from the dataset of reference data objects 120 based on the similarity score of each of the nearby data objects 122 to the new data object 102. The suitable subset includes at least one data object in the dataset, but not all of the data objects in the dataset. That is, the nearby data object selection engine 110 selects a relatively small number of reference data objects from the dataset that are most similar to the new data object 102 as the nearby data objects 122. Although a large number of reference data objects, e.g., a million, ten million, a billion or more, may be maintained in the dataset 120 by the system 100, only a small number of them may actually need to be selected, e.g., only the top 50, 100, or 200 nearby data objects having the highest similarity scores or meeting a similarity threshold may need to be selected.

[0039] For example, the nearby data object selection engine 110 may calculate the similarity score by evaluating a vector space based similarity measure, such as pairwise cosine similarity or Euclidean distance, of vectors or some other embedding (i.e., structured representation in an embedding space) that respectively represent the features of the nearby data objects 122 and the features of the new data object 102. As another example, the nearby data object selection engine 110 may calculate the similarity score by evaluating a graph based distance measure between a pair of nodes in a heterogeneous graph or another information graph that respectively represent the nearby data objects 122 and the new data object 102. Other similarity scores may be used.

[0040] The neighborhood feature vector generation engine 130 uses the selected subset of the neighborhood data objects 122 to generate one or more neighborhood feature vectors 132 for the new data object 102. In particular, the neighborhood feature vector generation engine 130 may generate each neighborhood feature vector 132 using (i) one or more labels, (ii) a similarity score (to the new data object), or both (i) and (ii) of each neighborhood data object 122 in the selected subset.

[0041] The classification subsystem 140 receives one or more neighborhood feature vectors 132 and processes the neighborhood feature vectors 132 to generate label data 142 that identifies one or more category labels for the new data object 102. The classification subsystem 140 may include one or more machine learning models. Each classification machine learning model may be configured as a neural network model (e.g., a multi-layer perceptron model), a naive Bayes model, a support vector machine model, a linear regression model, a logistic regression model, or a k-nearest neighbor model, just to name a few. Each model may be configured to process model inputs including the neighborhood feature vectors 132 according to model parameters to generate model outputs including respective classification scores for a predetermined set of possible categories of the new data object 102.

[0042] In some implementations involving multiple classification machine learning models, each model may correspond to a respective partition (or subset) of a predefined set of possible categories of the new data object 102. In some of these implementations, each model may process a different neighborhood feature vector 132 than each of the other models. Each model may be configured to generate, for each category in each partition, a respective classification score for that category. For example, assuming the new data object 102 represents an image having a total of four possible categories (each representing a possible object that may be present in the image), including (i) horse, (ii) dog, (iii) golden retriever, and (iv) German shepherd, a first partition may include the more general categories, i.e., category (i) and category (ii), and a second partition may include the more specific categories, i.e., category (iii) and category (iv). In this example, a first model included in the classification subsystem 140 may generate a respective classification score for each of categories (i) and (ii), and a second model included in the classification subsystem 140 may generate a respective classification score for each of categories (iii) and (iv), with the classification score for a given category representing the likelihood that the new data object 102 belongs to the given category. Within each partition, the classification subsystem 140 may then select the category with the higher (or highest) classification score as the category determined by the classification subsystem for the new data object 102.

[0043] Once the label data 142 for the new data object 102 is generated, the data object classification system 100 may store the new data object 102 in the dataset of reference data objects 120. For example, the system may store the new data object in association with the label data for the new data object, e.g., in association with data identifying the label generated for the new data object. In some implementations, instead of or in addition to storing the new data object in the dataset of reference data objects 120, the data object classification system 100 may associate a label or labels with the new data object and provide the labeled data object for use for some immediate purpose. For example, the system may provide the label to another system, e.g., a client device whose application has requested a connection to a web page, and the client device may then use the label to approve or deny the application's connection request.

[0044] 3 is a flow diagram of an exemplary process 300 for generating one or more labels for an input data object. For convenience, the process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a suitably programmed data object classification system, such as the data object classification system 100 of FIG. 1, may perform the process 300.

[0045] The system maintains a dataset including a plurality of reference data objects (step 302). The dataset is a labeled dataset that stores the reference data objects in association with data specifying one or more labels and one or more features of the data objects. Each label of the reference data objects in the dataset defines a respective known category of the reference data object. Each feature of the reference data object describes a characteristic of the reference data object.

[0046] The label data, feature data, or both of the reference data objects in the dataset may be made available to the system in any of a variety of ways. For example, the label data may be provided by a user of the system, the same or different from another user who provided the feature data of the reference data objects. As another example, the feature data may be provided by a data collection system, and the label data may be determined from the feature data by a separate data classification system. As another example, the label data may be pre-determined by the system itself from the available feature data of the reference data objects.

[0047] As a specific illustrative example, each data object may represent a web page, and the one or more labels of each data object may be selected from one or more of: (i) activity state labels that define an activity state of the web page (e.g., including an active state label, a dormant state label, an expired state label, etc.); (ii) validity state labels that define a validity state of the web page (e.g., including a phishing label, a trusted label, etc.); and the one or more characteristics of each data object may describe one or more of: (i) the geographic location of the web page (e.g., defined by an IP address or area code); (ii) the content contained in the web page; (iii) the layout of the web page; (iv) the functionality of the web page; (v) the age of the web page or the number of visits the web page has received; and the like.

[0048] The system receives a request to add a new data object to the dataset (step 304). The new data object may have one or more features, but may be missing one or more labels. The request may be submitted by a user of the system, for example, via a wired or wireless network. The request may reference the new data object, for example, by providing data that identifies the new data object and defines one or more features of the new data object through a user interface made available by the system. Unlike the feature data that is made available to the system when a request to add a new data object is received, the one or more labels are typically not specified by the request and therefore are not immediately available to the system. To determine the one or more missing labels of the new data object, the system accesses reference data objects stored in a dataset maintained by the system.

[0049] Specifically, the system selects N neighboring data objects from the multiple reference data objects based on the similarity scores of the neighboring data objects to the new data object (step 306). In particular, the system determines a similarity score for each reference data object in the dataset based on (i) one or more features of the new data object and (ii) one or more features of the reference data object. Once determined, the system sorts these similarity scores in descending order of score value and selects (as N neighboring data objects) the N reference data objects having the highest score values ​​correspondingly, where N is a predetermined natural number equal to or greater than 1, e.g., 10, 50, 100, etc. In some cases, N may have a predetermined fixed value. In other cases, the exact value of N may depend on the total number of features each reference data object has. Thus, the system generally selects more neighboring data objects when each reference data object has more labels that define more categories of the reference data object.

[0050] Depending on how the reference data object is stored in the dataset, the system may calculate the similarity score in any of a variety of ways. For example, the similarity score may be calculated by evaluating a vector space-based similarity measure, such as pairwise cosine similarity or Euclidean distance, of vectors or some other embeddings that represent the features of the reference data object and the features of the new data object, respectively. As another example, the similarity score may be calculated by evaluating a graph-based distance measure between a pair of nodes in an information graph that represent the reference data object and the new data object, respectively.

[0051] As a specific example, in embodiments where the dataset maintains multiple reference data objects in a heterogeneous graph format, the system may calculate the similarity score as a self-mutual information (PMI) similarity score by using techniques described in commonly owned PCT Patent Application No. WO2021173114A1, which is incorporated herein by reference. As another specific example, in these embodiments, the system may alternatively calculate the similarity score as a bipartite similarity score by using techniques described in commonly owned U.S. Patent No. US10152557B2, which is incorporated herein by reference.

[0052] The system generates one or more neighborhood feature vectors for the new data object (step 308). In particular, the system generates each neighborhood feature vector using (i) one or more labels of the neighborhood data objects, (ii) the neighborhood data object's similarity score to the new data object for each of the neighborhood data objects in the N neighborhood data objects, or both (i) and (ii).

[0053] For example, the system can generate a neighborhood feature vector by concatenating some or all of the similarity scores of N neighborhood data objects, concatenating some or all of the labels of N neighborhood data objects, or both. As another example, the system can generate a neighborhood feature vector by determining a concatenation of similarity scores, labels, or other values that can be derived from both, such as binary values or other numerical values.

[0054] In a specific example, the system can generate a neighborhood feature vector by determining a concatenation of the similarity scores of each subset of N neighborhood data objects having one or more specific categories out of all of the categories defined by one or more labels. In this specific example, the neighborhood feature vector can be calculated as follows. x1x2···x L x L+1 ···x KL x KL+1 ···x KL+N Here, K << N, and L can be any natural number less than or equal to the total number of labels of each data object, as follows. ● x1 = the highest similarity score among all the similarity scores of N neighborhood data objects having the first specific label. ... ● x L = the highest similarity score among all the similarity scores of N neighborhood data objects having the L-th specific label. ... ● x KL = the K-th highest similarity score among all the similarity scores of N neighborhood data objects having the L-th specific label. ● x KL+1 = the highest similarity score among all the similarity scores of N neighborhood data objects. ... ● x KL+N = the N-th highest similarity score among all the similarity scores of N neighborhood data objects.

[0055] The system processes the one or more neighborhood feature vectors using one or more machine learning models to predict one or more missing labels for the new data object, step 310. Because each of the neighborhood feature vectors generated in step 308 is a vector representation that includes or is derived from information about the similarity scores and labels of the new data object's N neighboring data objects, the use of the neighborhood feature vectors may facilitate faster and more accurate identification of one or more missing labels for the new data object.

[0056] Generally, each of the one or more machine learning models is a model configured to generate a classification score for a new data object. In some implementations in which multiple machine learning models are included, the different models may process different model inputs to generate different classification scores. For example, a first model may process a first model input including a first neighborhood feature vector according to first model parameters to generate a respective classification score for each category in a first partition of a predetermined category set, and a second model may process a second model input including a second neighborhood feature vector of the new data object according to second model parameters to generate a respective classification score for each category in a second partition of a predetermined category set. The system may select each of the one or more labels based on the classification scores, for example, by selecting a label corresponding to the highest classification score (either within the predetermined category set or within a partition of the predetermined category set) or by selecting a label corresponding to a classification score above a predetermined threshold.

[0057] Once the label data for the new data object has been generated, the system updates the dataset to include the new data object and to associate one or more predicted labels with the new data object (step 312). That is, the system adds the new data object to the dataset (as a reference data object) by adding to the dataset data specifying (i) the one or more features defined by the received request and (ii) the one or more labels that the system predicted from the neighborhood feature vectors.

[0058] The process 300 may be performed for each input data object to generate one or more labels for the input data object. The input data object may be a data object for which the desired label data, i.e., the label to be generated by the system for the data object, is not known. The system may also perform the process 300 for input data objects in a set of training data, i.e., a set of input data objects for which the label to be predicted by the system is known, to train the system, i.e., to determine trained values ​​for parameters of a machine learning model(s) configured to process the neighborhood feature vectors. In some implementations, the process 300 may be repeatedly performed for input data objects selected from the set of training data as part of a conventional machine learning training technique, e.g., a steepest descent method with backpropagation training technique, to train the model.

[0059] 4 is a block diagram of an example computer system 400 that may be used to perform the operations described above. System 400 includes a processor 410, a memory 420, a storage device 430, and an input / output device 440. Each of the components 410, 420, 430, and 440 may be interconnected using, for example, a system bus 450. Processor 410 is capable of processing instructions for execution within system 400. In one embodiment, processor 410 is a single-threaded processor. In another embodiment, processor 410 is a multi-threaded processor. Processor 410 is capable of processing instructions stored in memory 420 or storage device 430.

[0060] The memory 420 stores information within the system 400. In one implementation, the memory 420 is a computer-readable medium. In one implementation, the memory 420 is a volatile memory unit. In another implementation, the memory 420 is a non-volatile memory unit.

[0061] The storage device 430 is capable of providing mass storage for the system 400. In one implementation, the storage device 430 is a computer-readable medium. In various different implementations, the storage device 430 may include, for example, a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large capacity storage device.

[0062] The input / output device 440 provides input / output operations for the system 400. In one embodiment, the input / output device 440 may include one or more of a network interface device, such as an Ethernet card, a serial communication device, such as an RS-232 port, and / or a wireless interface device, such as an 802.11 card. In another embodiment, the input / output device may include a driver device configured to receive input data and send output data to other input / output devices, such as keyboards, printers, and display devices 370. However, other implementations, such as mobile computing devices, mobile communication devices, set-top boxes, televisions, client devices, etc., may also be used.

[0063] Although an example processing system is described in FIG. 4, implementations of the subject matter and functional operations described herein can be implemented in other types of digital electronic circuitry, or computer software, firmware, or hardware, including the structures disclosed herein and structural equivalents thereof, or any combination or combinations thereof.

[0064] An electronic document (for simplicity we will simply call it a document) does not necessarily correspond to a file: a document may be stored as part of a file that holds other documents, in a single file dedicated to that document, or in multiple coordinated files.

[0065] The embodiments of the subject matter and operations described herein may be implemented in digital electronic circuitry, or computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. The embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a computer storage medium (or media) for execution by or for controlling the operation of a data processing apparatus. Alternatively, or in addition, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to an appropriate receiver apparatus for execution by the data processing apparatus. The computer storage medium may be, or may be included in, a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or one or more combinations thereof. Furthermore, although a computer storage medium is not a propagating signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium may also be, or be contained in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

[0066] The operations described herein may be implemented as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices or received from other sources.

[0067] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system on a chip, or a combination or combinations of the foregoing. An apparatus may include special purpose logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). In addition to hardware, an apparatus may also include code that creates an execution environment for the computer program, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform execution environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0068] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted, declarative or procedural, and a computer program may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may correspond to a file in a file system, but this is not necessarily the case. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files that store one or more modules, subprograms, or code portions). A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0069] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by manipulating input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0070] Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors. Generally, a processor receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor that performs actions according to instructions, and one or more memory devices that store instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to, or both. However, a computer need not have such devices. Furthermore, a computer may be incorporated in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Suitable devices for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0071] To provide for interaction with a user, embodiments of the subject matter described herein may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and pointing device, such as a mouse or trackball, through which the user may provide input to the computer. Other types of devices may also be used to provide interaction with a user. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, spoken language, or tactile input. Additionally, the computer may interact with a user by sending documents to and receiving documents from a device used by the user, such as by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0072] An embodiment of the subject matter described herein may be implemented in a computing system, including a back-end component, e.g., as a data server, or including a middleware component, e.g., an application server, or including a front-end component, e.g., a client computer having a graphical user interface or a web browser through which a user may interact with an embodiment of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0073] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of clients and servers arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to a client device (e.g., for the purpose of displaying the data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., a result of user interaction) may be received from the client device at the server.

[0074] Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the context of separate embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as functioning in a combination, and may even be initially claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0075] Similarly, although the figures depict operations in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequential order depicted, or that all of the operations depicted be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0076] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A method implemented by one or more computers, comprising: maintaining a dataset including a plurality of reference data objects each having one or more labels, one or more features, or both, each label of the reference data objects defining a respective category of the reference data object, and each feature of the reference data objects describing a characteristic of the reference data object; receiving a request to add a new data object to the dataset, the new data object (i) having one or more features but (ii) missing one or more labels; selecting N neighboring data objects from the plurality of reference data objects based on similarity scores of the neighboring data objects with respect to the new data object, wherein the similarity scores of each neighboring data object are determined based on the one or more features of the new data object and the one or more features of the neighboring data objects, and N is a natural number equal to or greater than 1; for each neighboring data object in the N neighboring data objects, generating a neighborhood feature vector of the new data object using (i) the one or more labels of the neighboring data object and (ii) the similarity score of the neighboring data object to the new data object; processing the neighborhood feature vector using a machine learning model to predict the one or more labels missing in the new data object; updating the dataset to include the new data object and to associate the one or more predicted labels with the new data object; A method comprising:

2. 2. The method of claim 1, wherein maintaining the dataset including a plurality of reference data objects includes maintaining data describing a heterogeneous graph including a plurality of nodes connected by edges, each node corresponding to a different one of the reference data objects in the plurality of reference data objects, and each edge representing a relationship between two nodes connected by the edge.

3. 3. The method of claim 1 or 2, wherein maintaining the dataset including a plurality of reference data objects comprises maintaining the plurality of reference data objects and associated labels and features of the plurality of reference data objects in a relational dataset.

4. The method of any one of claims 1 to 3, further comprising determining the similarity score of each neighbouring data object with respect to the new data object based on one of Euclidean distance or cosine similarity in an embedding space.

5. The method of any one of claims 1 to 3, further comprising determining the similarity score of each neighbouring data object with respect to the new data object based on one of a Proof Mutual Information (PMI) score or a bipartite score.

6. The method of any one of claims 1 to 5, wherein the machine learning model comprises one of a neural network, a logistic regression model, a support vector machine (SVM), or a decision tree or random forest model.

7. 7. The method of claim 1, wherein generating the neighborhood feature vector of the new data object comprises determining a concatenation of similarity scores of each of a subset of the N neighborhood data objects having a particular category among the respective categories defined by one or more labels.

8. The method of claim 7 , wherein the subset of the N neighboring data objects comprises neighboring data objects each having a positive label that defines a trusted category.

9. The method according to any one of claims 7 to 8, wherein said subset of said N proximate data objects comprises proximate data objects each having a negative label defining an untrusted category.

10. The method of any one of claims 1 to 9, wherein each data object represents an image, a video, an audio, a text, or a web page.

11. The method according to any one of claims 1 to 10, wherein the value of N depends on the total number of said labels that each reference data object has.

12. 1. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations, the operations comprising: maintaining a dataset including a plurality of reference data objects each having one or more labels, one or more features, or both, each label of the reference data objects defining a respective category of the reference data object, and each feature of the reference data objects describing a characteristic of the reference data object; receiving a request to add a new data object to the dataset, the new data object (i) having one or more features but (ii) missing one or more labels; selecting N neighboring data objects from the plurality of reference data objects based on similarity scores of the neighboring data objects with respect to the new data object, wherein the similarity scores of each neighboring data object are determined based on the one or more features of the new data object and the one or more features of the neighboring data objects, and N is a natural number equal to or greater than 1; for each neighboring data object in the N neighboring data objects, generating a neighborhood feature vector of the new data object using (i) the one or more labels of the neighboring data object and (ii) the similarity score of the neighboring data object to the new data object; processing the neighborhood feature vector using a machine learning model to predict the one or more labels missing in the new data object; updating the dataset to include the new data object and to associate the one or more predicted labels with the new data object; Including, the system.

13. 13. The system of claim 12, wherein maintaining the dataset including a plurality of reference data objects includes maintaining data describing a heterogeneous graph including a plurality of nodes connected by edges, each node corresponding to a different one of the reference data objects in the plurality of reference data objects, and each edge representing a relationship between two nodes connected by the edge.

14. 14. The system of claim 12 or 13, wherein maintaining the dataset including a plurality of reference data objects comprises maintaining the plurality of reference data objects and associated labels and features of the plurality of reference data objects in a relational dataset.

15. The system of any one of claims 12 to 14, further comprising determining the similarity score of each neighbouring data object with respect to the new data object based on one of Euclidean distance or cosine similarity in an embedding space.

16. The system of any one of claims 12 to 14, further comprising determining the similarity score of each neighbouring data object with respect to the new data object based on one of a Proof Mutual Information (PMI) score or a bipartite score.

17. 17. The system of claim 12, wherein the machine learning model comprises one of a neural network, a logistic regression model, a support vector machine (SVM), or a decision tree or random forest model.

18. 18. The system of claim 12, wherein generating the neighborhood feature vector of the new data object comprises determining a concatenation of similarity scores of each of a subset of the N neighborhood data objects having a particular category among the respective categories defined by one or more labels.

19. A system according to any one of claims 12 to 18, wherein each data object represents an image, video, audio, text or a web page.

20. 1. A computer storage medium encoded with a computer program, the computer program comprising instructions that, when executed by one or more computers, cause the one or more computers to perform operations, the operations including: maintaining a dataset including a plurality of reference data objects each having one or more labels, one or more features, or both, each label of the reference data objects defining a respective category of the reference data object, and each feature of the reference data objects describing a characteristic of the reference data object; receiving a request to add a new data object to the dataset, the new data object (i) having one or more features but (ii) missing one or more labels; selecting N neighboring data objects from the plurality of reference data objects based on similarity scores of the neighboring data objects with respect to the new data object, wherein the similarity scores of each neighboring data object are determined based on the one or more features of the new data object and the one or more features of the neighboring data objects, and N is a natural number equal to or greater than 1; for each neighboring data object in the N neighboring data objects, generating a neighborhood feature vector of the new data object using (i) the one or more labels of the neighboring data object and (ii) the similarity score of the neighboring data object to the new data object; processing the neighborhood feature vector using a machine learning model to predict the one or more labels missing in the new data object; updating the dataset to include the new data object and to associate the one or more predicted labels with the new data object; A computer storage medium.

Images