System and method for using knowledge representation with machine learning classifier

Abstract

To provide a system and method for using knowledge representation with a machine learning classifier.SOLUTION: Systems and methods are provided for generating training data for a machine-learning classifier. A knowledge representation synthesized based on an object of interest is used to assign labels to content items. The labeled content items can be used as training data for training a machine learning classifier. The labeled content items can also be used as validation data for the classifier.SELECTED DRAWING: Figure 10A

Description

[Technical Field]

[0001] The teachings described in this disclosure relate to the fields of machine learning and information retrieval. In particular, the teachings disclosed in this application relate to the deployment in a digital information system environment of a method for using information associated with one or more datasets represented as a knowledge representation to classify information, train a classifier model for classifying information, and / or use the results of the classifier to refine the knowledge representation. [Background technology]

[0002] Information technology is often used to provide users with various types of information, such as text, audio, video, and any other suitable type of information. In some cases, information is provided to a user in response to an action taken by the user. For example, information may be provided to a user in response to a search query entered by the user or in response to a user subscribing to content such as email alerts or electronic newsletters. In other cases, information is provided or "pushed" to a user even if the user has not explicitly requested such information. For example, advertisements or solicitations may be presented to a user from time to time.

[0003] There is a vast amount of content and information that can be provided to users via information technology. Indeed, because of the vast amount of information available via the Internet, the World Wide Web (WWW), and any other suitable information source, and because the available information is distributed across a vast number of independently owned and operated networks and servers, identifying information of interest to a user can be challenging. Similar challenges exist when the information of interest is distributed across large private networks. Summary of the Invention

[0004] While the inventive concepts presented in this disclosure are illustrated in multiple different embodiments, each illustrating one or more of the concepts, it should be understood that, in general, these concepts are not mutually exclusive and may be used in combination even when not shown.

[0005] Some embodiments provide a method of generating training data for a machine learning classifier, the method including: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving a first set of content items, the first set including one or more content items without labels, the labels classifying the content items into one or more categories; determining one or more scores for one or more respective content items in the first set, the score for each content item based on the knowledge representation and content of the respective content item; and generating the training data for the machine learning classifier by assigning a label to each of the one or more content items in the first set based on the score associated with each of the one or more content items in the first set.

[0006] Some embodiments provide a system for generating training data for a machine learning classifier, the system having at least one processor configured to perform a method including receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving a first set of content items, the first set including one or more content items without labels, the labels classifying the content items into one or more categories; determining one or more scores for one or more respective content items in the first set, the score for each content item being based on the knowledge representation and content of the respective content item; and generating the training data for the machine learning classifier by assigning a label to each of the one or more content items in the first set based on the score associated with each of the one or more content items in the first set.

[0007] Some embodiments provide at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method of generating training data for a machine-learning classifier, the method including: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving a first set of content items, the first set including one or more content items without labels, the labels classifying the content items into one or more categories; determining one or more scores for one or more respective content items in the first set, the score for each content item being based on the knowledge representation and content of the respective content item; and generating the training data for the machine-learning classifier by assigning a label to each of the one or more content items in the first set based on the score associated with each of the one or more content items in the first set.

[0008] Some embodiments provide a method of classifying at least one unlabeled content item with a machine learning classifier, the method including: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving training data including a first set of one or more labeled content items having labels that classify each content item into one or more categories; training the machine learning classifier using at least one feature of the labeled content items based on one or more attributes derived from the knowledge representation; and classifying, with the machine learning classifier, the at least one feature of the at least one unlabeled content item based on the one or more attributes derived from the knowledge representation into the one or more categories.

[0009] Some embodiments provide a system for classifying at least one unlabeled content item with a machine learning classifier, the system having at least one processor configured to perform a method including receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving training data including a first set of one or more labeled content items having labels that classify each content item into one or more categories; training the machine learning classifier using at least one feature of the labeled content items based on one or more attributes derived from the knowledge representation; and classifying, with the machine learning classifier, the at least one feature of the at least one unlabeled content item based on the one or more attributes derived from the knowledge representation into the one or more categories.

[0010] Some embodiments provide at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method of classifying at least one unlabeled content item with a machine learning classifier, the method comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving training data, the training data including a first set of one or more labeled content items having labels that classify each content item into one or more categories; training the machine learning classifier using at least one feature of the labeled content items based on one or more attributes derived from the knowledge representation; and classifying, by the machine learning classifier, the at least one feature of the at least one unlabeled content item based on the one or more attributes derived from the knowledge representation.

[0011] Some embodiments provide a method of modifying a knowledge representation based on a machine learning classifier, the method including: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving validation data including a first set of one or more labeled content items having labels that classify each content item into one or more categories, including a first category known to be relevant to the object of interest and a second category known to be unrelated to the object of interest; predicting each of the one or more labeled content items as one of a) relevant to the object of interest and b) unrelated to the object of interest with a machine learning classifier using at least one attribute derived from the knowledge representation as a feature; and modifying the knowledge representation based on a comparison of the prediction by the machine learning classifier for each content item of the first set to the label of each content item.

[0012] Some embodiments provide a system for modifying a knowledge representation based on a machine learning classifier, the system having at least one processor configured to perform a method including receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving validation data including a first set of one or more labeled content items having labels that classify each content item into one or more categories, including a first category known to be relevant to the object of interest and a second category known to be unrelated to the object of interest; predicting each of the one or more labeled content items as one of a) relevant to the object of interest and b) unrelated to the object of interest with a machine learning classifier using at least one attribute derived from the knowledge representation as a feature; and modifying the knowledge representation based on a comparison of the prediction by the machine learning classifier for each content item of the first set to the label of each content item.

[0013] Some embodiments include at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method of modifying a knowledge representation based on a machine-learning classifier, the method comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or relationship between two or more concepts; and receiving validation data, the validation data classifying each content item into a first category known to be associated with the object of interest and a second category known to be associated with the object of interest. and a second category for which the content is known to be irrelevant; predicting, with a machine learning classifier using at least one attribute derived from the knowledge representation as a feature, each of the one or more labeled content items as one of a) relevant to the object of interest and b) irrelevant to the object of interest; and modifying the knowledge representation based on a comparison of the prediction by the machine learning classifier for each content item of the first set to the label of each content item.

[0014] The above is a non-limiting summary of the invention as defined by the claims; it is understood that this summary does not necessarily describe the subject matter of each claim, and that each claim may relate to one or more, but not all, embodiments. [Brief explanation of the drawings]

[0015] The accompanying drawings are not intended to be drawn to scale. Where practical, like elements are identified by the same or similar reference numerals. For clarity, not every component may be labeled in every drawing. [Figure 1] FIG. 1 is a block diagram illustrating an exemplary computing system for use in implementing some embodiments of the present invention. [Figure 2] 1 is a flowchart illustrating an exemplary workflow for a supervised learning technique, according to some embodiments of the present invention. [Figure 3] 1 is a flowchart illustrating an exemplary workflow for synthesizing a knowledge representation based on an object of interest, according to some embodiments of the present invention. [Figure 4] 1 is a diagram of an exemplary object of interest, according to some embodiments of the present invention. [Figure 5] 1 is a diagram of an exemplary workflow for extracting topical uniform resource identifiers based on objects of interest, according to some embodiments of the present invention. [Figure 6] 1 is a diagram of an exemplary workflow for synthesizing a knowledge representation based on topical uniform resource identifiers extracted from objects of interest, according to some embodiments. [Figure 7] 1 is a flowchart illustrating an exemplary method for labeling content items according to some embodiments of the present invention. [Figure 8A] FIG. 1 is a diagram of an exemplary workflow for training a classifier model, according to some embodiments of the present invention. [Figure 8B] 8B is a diagram of an exemplary workflow for labeling content items using the classifier model of FIG. 8A, according to some embodiments of the present invention. [Figure 9] 1 is a diagram of an exemplary workflow for analyzing content items based on features derived from an object-of-interest-based knowledge representation. [Figure 10A]FIG. 1 is a diagram of an exemplary workflow for training a classifier model using features from a knowledge representation derived from an object of interest, according to some embodiments of the present invention. [Figure 10B] FIG. 10B is a diagram of an exemplary workflow for labeling content items using the classifier model of FIG. 10A, according to some embodiments of the present invention. [Figure 11] 1 is a diagram of an exemplary table showing attribute-based features associated with content items, the known relevance of the content items, and predictions by a classifier, according to some embodiments of the present invention. [Figure 12] FIG. 1 is an exemplary table illustrating attribute-based features associated with a content item, the known relevance of the content item, and predictions by a first classifier and a second classifier, according to some embodiments of the present invention. [Figure 13] FIG. 1 is an exemplary table illustrating the results of predictions made by multiple classifier models, according to some embodiments of the present invention. [Figure 14A] FIG. 10 is an exemplary table showing the accuracy of labels predicted by a classifier, according to some embodiments of the present invention. [Figure 14B] 1 is a graphical illustration of classifier accuracy, according to some embodiments of the present invention. [Figure 15A] FIG. 10 is an exemplary table illustrating values ​​of attributes associated with a modified knowledge representation and the accuracy of labels predicted by a classifier, according to some embodiments of the present invention. [Figure 15B] 15B is a graphical illustration of the accuracy of the classifier of FIG. 15A, according to some embodiments of the present invention. [Figure 16] FIG. 1 is a diagram of an exemplary flowchart for modifying a knowledge representation based on classifier results, according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] The vast amount of content accessible through digital information systems presents several challenges for information retrieval. One difficult problem is how to determine what information in a large set of content may be of interest to a user so that such information can be presented to the user without overwhelming the user with irrelevant information. Yet another difficult problem is how to identify potentially interesting information in a computationally efficient manner.

[0017] The systems and methods described in this disclosure may be implemented in a variety of ways: Figure 1 is a block diagram illustrating an exemplary computing system that may be used in implementing some embodiments of the invention described in this disclosure.

[0018] Computing device 100 may include one or more processors (e.g., microprocessors) 110 and one or more non-transitory, tangible, computer-readable storage media (e.g., memory 120). Memory 120 may store computer instructions that implement any of the functions described in this disclosure. One or more processors 110 may be coupled to memory 120 and may execute such computer instructions to realize and perform functions. Computing device 100 may also include a network input / output (I / O) interface 130, via which computing device 100 can communicate with other computers (e.g., over a network). In some embodiments, computing device 100 may also include one or more user I / O interfaces 140, via which computing device 100 can provide output to and receive input from users. User I / O interface 140 may include devices such as a keyboard, a mouse, a microphone, a display device (e.g., a monitor or touch screen), speakers, a camera, and / or various other types of I / O devices.

[0019] Some embodiments of the present invention use machine learning algorithms, particularly supervised learning, to train a classifier. Supervised learning is a branch of machine learning that involves the task of inferring a function from labeled training data. Such training data includes a set of training examples. In supervised learning, each training example may include an object (e.g., a content item) and a label (e.g., that may classify the content item into one or more categories). The supervised learning algorithm can analyze the training data and result in an inferred function (e.g., a classifier model), which can then be used to predict labels for unlabeled content items (or to classify unlabeled content items).

[0020] For example, a machine learning classifier trained using supervised learning can be used to assign labels to unlabeled content items. However, a significant amount of training data is required to train the classifier. In some embodiments, the training data includes content items with labels that classify each content item into a category. Without sufficient training data to train the classifier, it is difficult for the machine learning algorithm to generate a model that provides accurate results.

[0021] 2 is a flowchart illustrating an exemplary workflow for a supervised learning technique according to some embodiments of the present invention. The process begins with raw data 210. The raw data 210 may include one or more content items. To be useful for machine learning purposes, the raw data 210 is converted into labeled characterization data 220 by assigning a label to each content item in the raw data 210. The labels may be binary in nature (e.g., a "yes" label or a "no" label, or a "1" label or a "0" label). However, there may be more than two possible labels. Labels may be assigned to the raw data 210 in any number of ways. For example, a human operator may manually review each item in the raw data 210 and assign a label.

[0022] Once labeled, the characterization data 220 may be split into a training set 230 and a validation set 240. The characterization data may be split in any ratio between the training set 230 and the validation set 240. The training set 230 is then used as the basis for building (hereinafter referred to as training) a model 250. The model 250 may be conceptualized as a function inferred from correlations between known labels for content items and various features of the content items. One of the purposes of the model 250 is for use in assigning labels to unlabeled content items based on features of the unlabeled content items that are consistent with the labels assigned to the training set 230 and the features associated with the training set 230.

[0023] Once the model 250 has been trained, it may be validated at 260 using a validation set 240. One way to validate the model 250 is to apply the model 250 to the validation set 240 and obtain a label for each content item in the validation set 240. Because the validation set 240 already contains content items with known labels, the known labels may then be compared to the labels generated by the model 250 to determine how accurate the model 250 is in correctly predicting labels.

[0024] The validated model 250 can then be applied to unlabeled content items and used to generate labels. In this way, large amounts of data can be labeled and / or classified by the model within a reasonable range of accuracy. However, as noted above, large amounts of training data are generally required to train the model. It would be desirable to obtain the training data in a manner that is computationally efficient and does not require manual labeling of content items.

[0025] 3 is a flowchart illustrating an exemplary workflow for synthesizing a knowledge representation based on an object of interest, according to some embodiments of the present invention. An object of interest ("OOI") 310 is analyzed, and terms and / or other characteristics contained in the OOI 310 are used to generate one or more topic uniform resource identifiers (topic URIs) 320. The topic URIs 320 are then used as a basis for synthesizing a knowledge representation 330 that includes several additional concepts and / or relationships between concepts, some of which are not found in the OOI 310.

[0026] FIG. 4 is a diagram of an exemplary OOI according to some embodiments of the present invention. While the exemplary OOI 310 in FIG. 4 is a single webpage, it should be understood that the OOI may span one website covering multiple webpages or multiple websites and may include any suitable type of unstructured data comprised of one or more corpora presenting items of interest. In some embodiments, the OOI 310 may include topics, tweets, webpages, websites, documents, document titles, messages, advertisements, and / or search queries. In some embodiments, the OOI 310 may be a single instance of the foregoing objects (e.g., a simple topic, a single document, etc.) or may include a collection of the foregoing objects (e.g., a collection of topics relevant to an individual's interests, a group of documents relevant to an individual's interests, etc.). Furthermore, the OOI 310 may include various types of items relevant to a particular user (e.g., a user's tweets and documents of interest to that same user) and may range from items across various users (e.g., items relevant to a particular interest market or demographic segment). The OOI 310 may include both online and offline objects.

[0027] As will be appreciated, varying the scope of content contained within the OOI 310 will vary the amount and depth of topic URIs 320 extracted therefrom, and the resulting synthesized knowledge representation 330. Thus, a balance can be achieved between the degree of focus and the amount of concepts and / or relationships contained within the knowledge representation 330. For example, an OOI 310 that contains a broader range of data represents a larger domain and provides greater scope for generating topic URIs 320 and synthesizing data. In contrast, when using a topic as the OOI 310, such as a phrase consisting of a few words, the topic will be the only lens for synthesizing the data.

[0028] Figure 5 is a diagram of an exemplary workflow for extracting topic URIs based on objects of interest, according to some embodiments of the present invention. For clarity, the OOI 310 shown in Figure 5 is the same web page as shown in Figure 4. The topic URIs 320 generated from the OOI 310 are shown in both linear and hierarchical form. As can be seen, the term "skin" is identified, and the subtopics conditions, symptoms, and clogs are shown as related to skin. Note that the hierarchical and linear representations of topic URIs 320 are two different ways of expressing the same relationship.

[0029] 6 is a diagram of an exemplary workflow for synthesizing a knowledge representation 330 based on topic URIs 320 generated from objects of interest 310, according to some embodiments. As can be seen, knowledge representation 330 includes additional concepts, namely, "organ," "brain," "cancer," and "redness," which are identified as having relationships to other concepts in topic URI 320. These additional concepts and relationships between concepts may be obtained, for example, from sources external to the objects of interest. For example, a thesaurus, a dictionary, other content items, or various tautologies may be used to identify the additional concepts and relationships. As discussed above, it should be understood that knowledge representation 330 may be the product of various types of objects of interest, including a wide range of unstructured data, such as, for example, a collection of documents known to be of interest to a user.

[0030] While specific examples are provided in this disclosure, it should be noted that there are many ways to synthesize knowledge representations 330. Further examples can be found, for example, in U.S. Pat. No. 9,378,203 and U.S. patent application Ser. No. 13 / 345,637, which are incorporated by reference in their entireties. In some embodiments, an atomic knowledge representation model (AKRM) may include one or more elemental data structures and one or more knowledge processing rules. In some embodiments, the rules may be used by a system to decompose (analyze) one or more complex knowledge representations (KRs) to generate elemental data structures. For example, the system may include one or more computer processors and one or more computer memory hardware components, and the memory may be encoded with computer-executable instructions that, when executed by the one or more computer processors, cause the one or more computer processors to use the rules in analyzing one or more complex KRs to generate elemental data structures for the AKRMs. The memory may also be encoded with instructions that program the one or more computer processors to synthesize new complex KRs from the elemental data structures using the rules. In some embodiments, computer memory may be embodied as one or more non-transitory, tangible, computer-readable storage media encoded with computer-executable instructions that, when executed, cause one or more processors to perform any of the functions described in this disclosure.

[0031] The composed knowledge representation 330 can be used to evaluate the relevance of other content items. For example, using the example from FIG. 6 , other content items can be analyzed for the presence of one or more of the concepts found in the composed knowledge representation. A score can be generated based on such analysis, which can be used as an indicator of the potential relevance of the content item to the OOI 310. Additionally, weights can be assigned to different concepts in the composed knowledge representation 330. Such weights can be used to indicate that a particular concept in the knowledge representation 330 is more likely to be correlated with relevance compared to another concept in the knowledge representation. A score for a content item can be determined, for example, by taking a weighted sum of the concepts. In some embodiments, a score for a content item can be determined based on the intersection of one or more concepts in the knowledge representation 330 with the content of the content item.

[0032] 7 is a flowchart illustrating an exemplary method for labeling content items according to scores, according to some embodiments of the present invention. At 710, one or more scored content items are obtained. In some embodiments, the content items are scored based on a knowledge representation 330 based on an object of interest 310. At 720, each content item is compared to a predetermined score threshold. If the score for the content item is greater than the threshold, the content item is labeled as "recommended" at 730. If the score for the content item is equal to or less than the score threshold, the content item is labeled as "do not recommend." Content items with a "recommended" label and content items with a "do not recommend" label form a set 750 of labeled content items. In some embodiments, content items may be ranked by score, and labels may be assigned based on the ranking of the content items within the set, rather than purely based on score. For example, for ten scored content items received at 710, it may be determined at 720 whether the content item was ranked in the top three out of the total ten content items.

[0033] While Figure 7 shows two possible labels for a content item, the present invention contemplates the use of more than two labels, rather than binary labels (e.g., "yes" or "no"), and tiered labels (e.g., multiple label options). For example, a label may include "recommended," with the next lower tier being "suggested," followed by "not recommended." The tiers may be expanded into any suitable number of labels and / or categories. Furthermore, labels need not be hierarchical, but may establish associative categories (e.g., sports, politics, technology, etc.).

[0034] As described above, the set of labeled content items 750 includes one or more content items having associated labels. Figure 8A is a diagram of an exemplary workflow for training a classification model 820 according to some embodiments of the present invention. The set of labeled content items 810 can be used as training data (e.g., the training set 230 described in this disclosure) to build the machine learning classifier model 820. In some embodiments, the set of labeled content items 810 is generated by the process described in Figure 7.

[0035] In some embodiments, training the classification model 820 based on the labeled content items 810 includes, for each labeled content item, comparing various features of the content item with the label associated with that content item. In some embodiments, these features may include standard features, examples of which include title, length, author, word frequency, and / or inverse document frequency. In some embodiments, these features may include one or more attributes of a knowledge representation. The classification model 820 includes an inferred function that can be used to determine which label should be assigned to a particular content item based on the features of the particular content item.

[0036] FIG. 8B is a diagram of an exemplary workflow for labeling content items using the classification model 820 of FIG. 8A , according to some embodiments of the present invention. Once the classification model 820 has been trained and, optionally, validated with additional labeled validation data, the classification model 820 can be used to assign labels to a set 830 of one or more unlabeled content items. Such classification is based on the same features used to train the classification model 820. It should be understood that such features (e.g., title, length, author, word frequency, and / or inverse document frequency) can be easily ascertained from the content items with minimal or no analysis required by a human operator. Thus, labeling the unlabeled content items 830 using the classification model 820 can be performed relatively quickly and efficiently. Applying the classification model 820 to the unlabeled content items 830 results in a second set 820 of labeled content items. The labeled content items 850, which have been assigned appropriate labels (e.g., “recommended” or “recommended”), can then be sent to a user or user group as content that is likely to be of interest.

[0037] The above process can be considered similar to unsupervised learning techniques because the classifier does not require any teachers (i.e., training data or unlabeled content) to produce useful classifications. Unsupervised learning is a machine learning task that infers functions to describe hidden structure from unlabeled data. By identifying intersections between the knowledge representation 330 and the unlabeled content items 830, useful representations / features of the unlabeled content items 830 are generated (i.e., hidden structure containing semantic representations specific to the object of interest 310). Furthermore, the process of FIG. 7 can be extended to techniques for clustering content items around features derived from attributes of the knowledge representation (described in more detail below).

[0038] 9 is a diagram of an exemplary workflow for classifying unlabeled content items with a machine learning classifier using attributes derived from a knowledge representation based on objects of interest. As described above with reference to FIGS. 3-6, the objects of interest 310 can be used to synthesize a knowledge representation 330 that includes at least one concept and / or relationships between two or more concepts. In some embodiments, the knowledge representation is encoded as a non-transitory, computer-readable data structure.

[0039] As shown in Figure 9, a collection of labeled content items may be used as training data for a machine learning algorithm. It may be possible to use the knowledge representation 330 as a source of features for a machine learning classification model. Examples of content item features based on attributes derived from the knowledge representation may include the total number of concept intersections between the knowledge representation and the content item, the number of narrower concept intersections between the knowledge representation and the content item, or the sum of the weights of the intersections. Those skilled in the art will appreciate that many combinations and weighted sums of these attributes may also be used as features for a machine learning classification model. Further examples are described in more detail below.

[0040] 10A is a diagram of an exemplary workflow for training a classifier model using features from a knowledge representation derived from an object of interest, according to some embodiments of the present invention. The training data includes a first set 1010 of one or more labeled content items. In some embodiments, each label classifies the content items into one or more categories. In some embodiments, there may also be a set of validation data including a second set of one or more labeled content items.

[0041] The machine learning classifier 1020 is trained from features based on one or more attributes derived from the knowledge representation and the labeled content items 1010. Training the classification model 1020 may include inferring one or more functions using at least one feature of the content items in the training dataset based on one or more of the attributes derived from the knowledge representation and the labels associated with the content items in the training dataset.

[0042] 10B is a diagram of an exemplary workflow for classifying unlabeled content items using the classifier model of FIG. 10A, according to some embodiments of the present invention. A set of at least one unlabeled content item 1030 is analyzed by a classification model 1020, which classifies the at least one unlabeled content item 1030 into a second set of at least one labeled content item 1050. In some embodiments, this classifying uses at least one feature based on one or more attributes derived from a knowledge representation and the content of the unlabeled content item.

[0043] 11 is a diagram of an exemplary table showing attributes associated with content items, the known relevance of the content items, and predictions by a classifier, according to some embodiments of the present invention. In FIG. 11, a training set of labeled content items includes content items 1-50. In this example, the labels classify the content items as either relevant to the object of interest or not relevant to the object of interest (denoted in the table by Y for "yes" and N for "no"). Additionally, there is a validation data set that includes a second set of labeled content items, denoted by content items 51-100. Content items 101-n are unlabeled content items whose labels (and therefore their relevance to the OOI) are not known.

[0044] FIG. 11 further illustrates corresponding values ​​for features of the training and validation data items based on a first and second attribute (shown as “Attribute 1” and “Attribute 2,” respectively) of the knowledge representation. While values ​​for features based on two attributes are shown in FIG. 11 , some embodiments may use features based on one attribute to train a classifier model. Some embodiments may also use two or more features based on multiple attributes to train a classifier model. While the labels in the example of FIG. 11 are binary (i.e., the only options are Y or N), this need not necessarily be the case; content items may also be classified into more than two groups using the systems and methods described in this disclosure.

[0045] The values ​​for the features based on Attribute 1 and Attribute 2 for each content item in the table of Figure 11 may be based on the intersection of these attributes with the content of that content item. The attributes are derived from the combined knowledge representation and may include, for example, the total number of concepts common between the knowledge representation and the content item, the number of broader concepts common between the knowledge representation and the content item, the number of narrower concepts common between the knowledge representation and the content item, and the sum of the weights of the concepts common between the knowledge representation and the content item.

[0046] Using the exemplary OOI 310 shown in Figure 4 and the combined knowledge representation 330 shown in Figure 6 as an example, it can be seen that knowledge representation 330 includes a hierarchy of concepts with different breadths. For example, the concept of "organ" is broader than the concepts of "brain" and "skin" in knowledge representation 330. Similarly, the concept of "redness" is narrower than the concept of "symptoms." Each of these concepts in knowledge representation 330 may also have an associated weight.

[0047] Continuing with this example, attribute 1 in FIG. 11 is the "number of broader concepts" and attribute 2 is the "number of narrower concepts." Content item 1 may be compared to knowledge representation 330 and / or examined for intersection with knowledge representation 330 to determine that content item 1 contains seven instances of the word "organ," for an attribute 1 score of 7, and 30 instances of the word "redness," for an attribute 2 score of 30. Similarly, content item 2 may contain five instances of the word "brain" and ten instances of the word "skin," for an attribute 1 score of 15. Content item 2 may also contain 15 instances of the word "cancer," and no instances of the word "redness," for an attribute 2 score of 15. Those skilled in the art will understand that while attribute 1 and attribute 2 are characteristics of knowledge representation 330, the scores in the table of FIG. 11 are based on the features of the content item that are in common with the attributes derived from knowledge representation 330.

[0048] Because the "known relevance" labels are already known for content items 1 and 2 (and more generally for content items 1-50), a classifier model can be built based on the detected relationships between the values ​​of Attribute 1 and Attribute 2 on which the features are based and the resulting labels. It should be appreciated that when attributes derived from the synthesized knowledge representation 330 are used as features for training a machine learning classifier, fewer training data items may be required to build a classifier model with a pre-defined level of accuracy compared to using so-called "standard features" of content items (e.g., title, length, author, etc.) for training a machine learning classifier.

[0049] Utilizing synthetic means and the resulting attributes as features can help avoid the problem of data sparsity when the representation of data in either the OOI 310 or the unlabeled content items is too limited to yield a useful classification using only "standard features." For example, a single tweet in the OOI 310 or the title of an unlabeled content item 1030 simply cannot provide enough information for a classifier to accurately classify the content item (in some cases, the literal representation of the titles in several unlabeled content items 1030 may not be sufficiently distinguishable from one another). Instead, utilizing feature-based attributes sourced from knowledge representations provides the system with more data, which can avoid the drawbacks of relying on features applied in traditional machine learning methods.

[0050] Once the machine learning classifier is trained using the training data, the classifier model may be validated using validation data items (e.g., content items 51-100 in FIG. 11 ). As can be seen, values ​​for features based on Attribute 1 and Attribute 2 are determined for each validation data item, and predictions by the classifier based on the values ​​for features based on Attribute 1 and Attribute 2 are shown in the right-most column of FIG. 11 . The labels assigned to content items 51 and 100 are correct, while the label predicted by the classifier for content item 99 is incorrect. In some embodiments, a minimum level of accuracy for the classifier model may be required. If the rate of correct predictions for the classifier model is below an acceptable threshold, the system may further train the machine learning classifier using additional training data. For example, in some embodiments, if a classifier based solely on the training dataset does not achieve a required level of accuracy, the machine learning classifier may be trained using both the training dataset and the validation dataset.

[0051] Once the machine learning classifier is trained, it can be used to classify content items that do not have a label. As shown in Figure 11, content items 101 and 102 contain a "?" in the "Known Relevance" column, and the machine learning classifier predicts the "N" and "Y" labels, respectively.

[0052] In some embodiments, more than one machine learning classifier may be trained. For example, with reference to FIG. 12 , a first classifier (“Classifier 1”) may be trained using features of training data content items based on Attribute 1 and Attribute 2. A second classifier (“Classifier 2”) may be trained using features based on a different subset of these attributes. For example, the second classifier may be trained using only features based on Attribute 1. As can be seen, the labels predicted by the first and second classifiers may differ for the same content item depending on the correlation between a particular attribute or set of attributes and the label associated with it for a particular content item.

[0053] FIG. 13 is an exemplary table illustrating the results of predictions made by multiple classifier models (i.e., a collection of classifier models) according to some embodiments of the present invention. As shown, predicted labels for unlabeled content items 101-105 are shown for a first classifier, a second classifier, and a third classifier trained using different sets or subsets of features based on different attributes. In some embodiments, an overall label may be generated according to a weighted sum of predictions from the different classifiers. In the example shown in FIG. 13, each classifier is binary (meaning it assigns a yes or no label), and each classifier is weighted equally. Thus, a "yes" prediction may be given a value of 1, and a "no" prediction may be given a value of 0. An average score may then be taken for each content item for the overall label. In the example shown in FIG. 13, content items with an average score above 0.5 are given the "yes" label, while content items with a score below 0.5 are given the "no" label. In some embodiments, assigning a label to a content item is similar to classifying the content item into one of several possible categories (e.g., "relevant to OOI" or "not relevant to OOI").

[0054] Note that the thresholds and weights assigned to a particular classifier can be selected to achieve a desired result. The equally weighted system shown in FIG. 13 with binary classifiers is merely an example. In some embodiments, the classifiers are not equally weighted, and a classifier may optionally generate more than two possible labels (e.g., "recommend," "suggest," and "not recommend"). Another example classification is "duplicate" or "not duplicate," where a classifier can be trained to classify a content item as too similar to an object of interest to recommend (e.g., an article contains the same text as the object of interest but is released by a different publishing source).

[0055] As described above, in some embodiments, a knowledge representation is synthesized based on the content of the object of interest. In some embodiments, synthesizing includes generating at least one concept and / or a relationship between two or more concepts, where the concept and / or relationship is not enumerated in the object of interest. In some embodiments, the synthesized knowledge representation may apply generation rules to the elemental data structures to synthesize a new composite knowledge representation. The knowledge representation may also include a weight associated with at least one concept. Furthermore, the knowledge representation may take into account characteristics of a particular user or a particular user group. For example, if a particular user is known to be interested in tennis, this may be taken into account in the knowledge representation, e.g., by assigning a higher weight to concepts related to tennis or by generating additional concepts related to tennis and adding the additional concepts to the knowledge representation. Thus, a machine learning classifier trained using attributes from a knowledge representation customized for a particular user or user group may result in different classifications for the same content item compared to a classifier trained using attributes from a knowledge representation that is not specific to the particular user or user group.

[0056] In some embodiments, the training data used to train the machine learning classification model is generated according to the process described above with reference to FIG.

[0057] As noted above, the function inferred by a supervised learning algorithm may be based on observed or detected correlations between various features of the training data content items and the labels assigned to the training data content items. It is possible that some features of the content items may bear little or no relation to the resulting label, while other features may have a very close correlation with the resulting label. For example, a feature based on an attribute related to the number of narrower concepts in the knowledge representation contained in the content item may have a stronger correlation with the label than a feature based on an attribute related to the number of broader concepts.

[0058] It may be desirable to identify features of training data content items and / or attributes of knowledge representations that tend to have a higher correlation with the labels assigned to training data objects. Training a machine learning classifier with a greater emphasis on such features or attributes may result in better overall accuracy for the classifier, or potentially a reduction in the amount of training data items required to train the classifier to an acceptable accuracy range. Furthermore, identifying such features and / or attributes may also reduce the number of attributes or features that need to be evaluated for each training data item. This may reduce the computational complexity of the machine learning algorithm and its associated processing costs.

[0059] 14A is a diagram of an exemplary table showing the accuracy of labels predicted by a classifier, according to some embodiments of the present invention. As can be seen, the machine learning classifier correctly classifies validation content items 1, 2, 4, and 6, while incorrectly classifying validation content items 3, 5, 7, and 8. Thus, the classifier has a 50% accuracy rate for these eight validation content items.

[0060] Figure 14B is a graphical illustration of the accuracy of the results shown in Figure 14A. As can be seen, content items 7 and 8 are relevant but are "missed" by the machine learning classifier and lie outside the circle shown. Content items 1, 2, and 4 are correctly classified. Content items 3 and 5 are irrelevant and are incorrectly classified as relevant. Content item 6 is irrelevant and is correctly classified.

[0061] One metric for evaluating the accuracy of a classifier is the concept of precision, which is defined as the ratio of the number of correctly labeled relevant items divided by the total number of items labeled as relevant by the classifier. In the example of Figure 14B, the precision of the classifier is 3 / 5 (because content items 1, 2, and 4 are correctly recommended, while content items 3 and 5 are incorrectly recommended).

[0062] Another metric for evaluating the accuracy of a classifier is the concept of recall, which is defined as the ratio of the number of correctly labeled relevant items divided by the total number of relevant items. In the example of Figure 14B, the recall of the classifier is 3 / 5 (because content items 1, 2, and 4 are recommended, while content items 3 and 5 are not).

[0063] Precision and recall are often conflicting goals for a classifier, because the focus of recall is to identify all relevant items even if some are irrelevant, whereas the focus of precision is to identify only relevant items while filtering out irrelevant items. In other words, recall can be prioritized over precision when avoiding false negatives is prioritized over avoiding false positives, and precision can be prioritized over recall when avoiding false positives is prioritized over avoiding false negatives.

[0064] In some embodiments, predictions made by the machine learning classifier may be compared to the labels assigned to each verification content data item, and the knowledge representation may then be modified based on this comparison.

[0065] For example, it can be observed in FIG. 14A that higher scores for features based on attribute 2 correlate with the “Y” label, while values ​​for features based on attribute 1 correlate relatively less. In light of this trend, the knowledge representation for the OOI can be modified to reflect such correlation. For example, if the feature based on attribute 2 in the example of FIG. 14A is the number of narrower concepts in the knowledge representation that are common to each validation content item, the knowledge representation (e.g., FIG. 6) can be modified to include additional narrower concepts. For example, the concept “sunburn” can be added to the “conditions” hierarchy, and the concept “itching” can be added to the “symptoms” hierarchy. As another example, weights associated with particular concepts (i.e., narrower concepts) in the knowledge representation can be increased and / or weights associated with particular concepts (i.e., broader concepts) can be decreased.

[0066] The modified knowledge representation can then be used to generate values ​​for the features of the content items based on each attribute. Figure 15A reflects the updated values ​​for each attribute-based feature in the knowledge representation for validation content items 1-8. As can be seen, the value for the feature based on attribute 2 (which in this example corresponds to the number of narrower concepts) is larger compared to the value in Figure 14A. While the value for attribute 1 remains unchanged in Figure 15A, this is not necessarily the case, and the values ​​shown in this disclosure are merely examples.

[0067] Using the values ​​for the features based on Attribute 1 and Attribute 2 using the modified knowledge representation, the machine learning classifier can then newly predict a label for the validation content item. As described above, the attributes may include at least one of the following: the total number of concepts common between the knowledge representation and one or more labeled content items; the number of broader concepts common between the knowledge representation and one or more labeled content items; the number of narrower concepts common between the knowledge representation and one or more labeled content items; and the sum of the weights of the concepts common between the knowledge representation and one or more labeled content items. As can be seen in Figure 15A, the predicted relevance for a particular content item is different using the modified knowledge representation.

[0068] FIG. 15B provides a graphical illustration of the results of the classifier using the modified knowledge representation along with the machine learning classifier. As can be seen, content items 1, 2, 4, and 7 are now correctly recommended, while content item 8 remains "missed" by the classifier. Content item 3 remains incorrectly recommended, while content item 5 is now correctly labeled as irrelevant. Content item 6 remains correctly labeled as irrelevant. Thus, the precision of the classifier using the modified knowledge representation in this example is now 4 / 5 (four content items are correctly recommended, for a total of five content items). The recall of the classifier using the modified knowledge representation is now 4 / 5 (four content items are correctly recommended, and one relevant content item is missed). Thus, both the precision and recall of the machine learning classifier are improved in this example. Note that some embodiments may improve only one of precision and recall by modifying the knowledge representation.

[0069] In some embodiments, the knowledge representation may be iteratively modified until a target precision or recall threshold is achieved. For example, if the target precision is 9 / 10 (i.e., 90%), the results in Figures 15A and 15B would require further modifications to the knowledge representation to label content item 8 as relevant or content item 3 as irrelevant, given that Figures 15A and 15B show a precision of 4 / 5 (i.e., 80%).

[0070] 16 is a diagram of an exemplary flowchart for modifying a knowledge representation based on classifier results, according to some embodiments of the present invention. At 1610, a machine learning classifier predicts the relevance of a set of labeled content items (e.g., validation content items). At 1620, the prediction by the machine learning classifier is compared to the known labels of the validation data items (which correspond to relevance in this example). At 1630, it is determined whether the goal was achieved by the machine learning classifier. In some embodiments, the goal is a precision goal. In some embodiments, the goal is a recall goal. In some embodiments, the goal is a combination of a precision goal and a recall goal.

[0071] If the goal is not achieved, the knowledge representation is modified at 1640. The predicted label for the validation content item is removed at 1650. The process then returns to 1610, where the relevance of the validation content item is predicted.

[0072] At 1630, if the desired results have been achieved, the process ends and the machine learning classifiers and knowledge representations can be used to predict the relevance of unlabeled content items.

[0073] Accordingly, some embodiments of the methods, systems, and computer-readable media described herein may be useful in one or more of generating training data for training a machine learning classifier to identify content relevant to an object of interest, using feature attributes from a synthesized knowledge representation based on the object of interest to train the machine learning classifier, and modifying the synthesized knowledge representation based on observed precision and / or recall values ​​associated with the trained machine learning classifier.

[0074] The present disclosure also contemplates applying the aforementioned feedback loop to modify the knowledge representation so that new relationships and / or concepts can be formed within the knowledge representation. For example, if the occurrence of a particular concept or relationship within the knowledge representation results in improved performance of a machine learning classifier, the method can recognize that improved performance and modify the knowledge representation to include this concept or relationship. By considering portions of the content that do or do not have a particular concept or relationship, patterns within the content can be identified that affect the organization of elements in the knowledge representation.

[0075] Machine learning classifiers of the type described in this disclosure can also be used to determine whether two independent concepts already contained within a knowledge representation can be combined (or alternatively linked by some type of relationship) to form a new concept, for example, by identifying that two independent, unlinked concepts (e.g., "fire" and "hose") should be linked by a relationship (e.g., "fire" is-related-to "hose") or should be combined to form a new concept (e.g., "firehose").

[0076] It should be noted that various inventive concepts may be embodied as one or more methods, several examples of which are provided in this disclosure. The actions performed as part of a method may be ordered in any suitable manner. Thus, while actions are shown in an example embodiment as sequential, embodiments may be constructed in which actions are performed in a different order than shown, which may include performing some actions simultaneously, or vice versa.

[0077] The above-described embodiments of the present invention may be implemented in any of numerous ways. For example, embodiments may be implemented using hardware, software, or a combination thereof. If implemented in software, the software code may be embodied as stored program instructions that may be executed on any suitable processor or collection of processors (e.g., one microprocessor or multiple microprocessors), whether provided in a single computer or distributed among multiple computers.

[0078] It should be understood that a computer may be embodied in any of numerous forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, a computer may be embodied in devices not generally considered to be computers but having suitable processing capabilities, including personal digital assistants (PDAs), smartphones, tablets, readers, or any other suitable portable or fixed electronic device.

[0079] A computer may also have one or more input and output devices. These devices may be used, among other things, to present a user interface. Examples of output devices that may be used to provide a user interface include a printer or display screen for visual presentation of output, and a speaker or other sound-generating device for audible presentation of output. Examples of input devices that may be used for a user interface include a keyboard, a microphone, and a pointing device such as a mouse, touchpad, or digital tablet.

[0080] Such computers may be interconnected by one or more networks of any suitable form, including networks such as a local area network (LAN) or a wide area network (WAN) such as an enterprise network, an intelligent network (IN), the Internet, etc. Such networks may be based on any suitable technology and may operate according to any suitable protocol, and may include wireless networks, wired networks, and / or fiber optic networks.

[0081] The various methods or processes outlined in this disclosure may be coded as software executable on one or more processors utilizing any one of a variety of operating systems or platforms. Further, such software may be written using a number of suitable programming languages ​​and / or programming or scripting tools, and compiled as executable machine code or intermediate code that runs on a virtual machine or suitable framework.

[0082] In this regard, various inventive concepts may be embodied as an article of manufacture of at least one non-transitory, tangible computer-readable storage medium (e.g., computer memory, one or more floppy disks, compact disks, optical disks, magnetic tapes, flash memories, circuit configurations in field programmable gate arrays, or other semiconductor devices, etc.) encoded with one or more programs that, when executed on one or more computers or other processors, implement various process embodiments of the present invention. The at least one non-transitory computer-readable medium may be portable such that the one or more programs stored thereon can be loaded onto any suitable computer resource to implement various aspects of the present invention as described above.

[0083] The terms "program" or "software" are used generically in this disclosure to refer to any type of computer code or set of computer-executable instructions that can be used to program a computer or other processor to implement various aspects of the above-described embodiments. Furthermore, it should be understood that, according to one aspect, one or more computer programs that, when executed, perform the methods of the present invention need not reside on a single computer or processor, but may be distributed in a modular manner among different computers or processors to implement various aspects of the present invention.

[0084] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, items, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0085] Additionally, the data structures may be stored in any suitable form of non-transitory, tangible, computer-readable storage medium product. For ease of explanation, the data structures may be depicted as having fields that are related through their location within the data structure. Such relationships may equally be achieved by allocating storage for the fields with locations within the non-transitory, tangible, computer-readable medium that convey the relationship between the fields. However, any suitable mechanism may be used to establish relationships between information within fields of the data structure, including through the use of pointers, tags, or other mechanisms that establish relationships between data elements.

[0086] All definitions and definitions used in this disclosure should be understood to govern by dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0087] The indefinite articles "a" and "an," as used in this disclosure, unless expressly indicated otherwise, should be understood to mean "at least one."

[0088] As used in this disclosure, the phrase "at least one" in reference to a list of one or more elements should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but does not necessarily include at least one of each and every element specifically listed in the list of elements, and does not exclude any combinations of elements in the list of elements. This definition also allows for elements other than those specifically identified in the list of elements to which the phrase "at least one" refers, whether related or unrelated to the elements specifically identified, may optionally be present. Thus, as a non-limiting example, "at least one of A and B" (or "at least one of A or B," or "at least one of A and / or B") can refer in one embodiment to at least one (optionally including more than one) A with no B (optionally including elements other than B); in another embodiment to at least one (optionally including more than one) B with no A (optionally including elements other than A); in yet another embodiment to at least one (optionally including more than one) A and at least one (optionally including more than one) B (optionally including other elements); etc.

[0089] The phrase "and / or" as used in this disclosure should be understood to mean "either or both" of the elements so conjoined, i.e., elements that are present conjunctively in some cases and disjunctively in other cases. Multiple elements listed with "and / or" should be construed in the same manner, i.e., "one or more" of the elements so conjoined. Other elements other than the elements specifically identified by the "and / or" clause may optionally be present, whether related or unrelated to the elements specifically identified. Thus, as a non-limiting example, a reference to "A and / or B," when used with open-ended language such as "having," "comprising," or the like, can, in one embodiment, refer to A only (optionally including elements other than B); in another embodiment, it can refer to B only (optionally including elements other than A); in yet another embodiment, it can refer to both A and B (optionally including other elements); and so forth.

[0090] As used in this disclosure, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" should be interpreted as inclusive, i.e., including at least one (including more than one) of the elements or list of elements, and optionally including additional items not in the list.

[0091] The phraseology and terminology used in this disclosure are for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," and their derivatives is meant to encompass the items listed thereafter and additional items.

[0092] While several embodiments of the present invention have been described in detail, various modifications and improvements will readily occur to those skilled in the art. Such modifications and improvements are intended to be within the spirit and scope of the present invention. Accordingly, the foregoing description is by way of example only and is not intended to be limiting. The following notes are provided regarding the above embodiment. [Appendix 1] 1. A method for generating training data for a machine learning classifier, comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving a first set of content items, the first set including one or more content items without labels, the labels classifying the content items into one or more categories; determining one or more scores for one or more respective content items of the first set, the score for each content item being based on the knowledge representation and the content of the respective content item; generating the training data for the machine learning classifier by assigning a label to each of the one or more content items in the first set based on a score associated with each of the one or more content items in the first set; A method comprising: [Appendix 2] synthesizing the knowledge representation based on the content of the object of interest; 2. The method of claim 1, further comprising: [Appendix 3] The synthesizing step comprises: generating the at least one concept and / or relationships between two or more concepts, wherein the at least one concept and / or relationships between two or more concepts are not enumerated in the object of interest; 3. The method of claim 2, comprising: [Appendix 4] 10. The method of claim 1, wherein the knowledge representation includes a weight associated with the at least one concept. [Appendix 5] 10. The method of claim 1, wherein the score for each content item is based on an intersection of the at least one concept in the knowledge representation and the content of the each content item. [Appendix 6] 2. The method of claim 1, wherein the interest objects include topics, tweets, web pages, websites, documents, collections of documents, document titles, messages, advertisements, and / or search queries. [Appendix 7] training a classifier to predict labels for one or more unclassified content items based on the labels assigned to the first set of content items and one or more features associated with the first set of content items; 2. The method of claim 1, further comprising: [Appendix 8] receiving a second set of content items, the second set including one or more content items that do not have a label; assigning, by the classifier, a label to each of the one or more content items in the second set based on one or more features associated with the content items; 8. The method of claim 7, further comprising: [Appendix 9] assigning the label to the one or more respective content items of the first set comprises: assigning the label based on a score for each content item in the first set exceeding a predetermined threshold. 2. The method of claim 1, comprising: [Appendix 10] 2. The method of claim 1, wherein the label is one of "recommended" and "not recommended." [Appendix 11] 8. The method of claim 7, wherein the one or more features include at least one of title, length, author, word frequency, and inverse document frequency. [Appendix 12] 8. The method of claim 7, wherein the one or more features include one or more attributes of the knowledge representation. [Appendix 13] 1. A system for generating training data for a machine learning classifier, comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving a first set of content items, the first set including one or more content items without labels, the labels classifying the content items into one or more categories; determining one or more scores for one or more respective content items of the first set, the score for each content item being based on the knowledge representation and the content of the respective content item; generating the training data for the machine learning classifier by assigning a label to each of the one or more content items in the first set based on a score associated with each of the one or more content items in the first set; At least one processor configured to perform a method including A system having: [Appendix 14] The method comprises: synthesizing the knowledge representation based on the content of the object of interest; 14. The system of claim 13, further comprising: [Appendix 15] The synthesizing step comprises: generating the at least one concept and / or relationships between two or more concepts, wherein the at least one concept and / or relationships between two or more concepts are not enumerated in the object of interest; 15. The system of claim 14, comprising: [Appendix 16] 14. The system of claim 13, wherein the knowledge representation includes a weight associated with the at least one concept. [Appendix 17] 14. The system of claim 13, wherein the score for each content item is based on an intersection of the at least one concept in the knowledge representation and the content of the each content item. [Appendix 18] 14. The system of claim 13, wherein the objects of interest include topics, tweets, web pages, websites, documents, collections of documents, document titles, messages, advertisements, and / or search queries. [Appendix 19] The method comprises: training a classifier to predict labels for one or more unclassified content items based on the labels assigned to the first set of content items and one or more features associated with the first set of content items; 14. The system of claim 13, further comprising: [Appendix 20] The method comprises: receiving a second set of content items, the second set including one or more content items that do not have a label; assigning, by the classifier, a label to each of the one or more content items in the second set based on one or more features associated with the content items; 20. The system of claim 19, further comprising: [Appendix 21] assigning the label to the one or more respective content items of the first set comprises: assigning the label based on a score for each content item in the first set exceeding a predetermined threshold. 14. The system of claim 13, comprising: [Appendix 22] 14. The system of claim 13, wherein the label is one of "recommended" and "not recommended." [Appendix 23] 20. The system of claim 19, wherein the one or more features include at least one of title, length, author, word frequency, and inverse document frequency. [Appendix 24] 20. The system of claim 19, wherein the one or more features include one or more attributes of the knowledge representation. [Appendix 25] At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method for generating training data for a machine learning classifier, the method comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving a first set of content items, the first set including one or more content items without labels, the labels classifying the content items into one or more categories; determining one or more scores for one or more respective content items of the first set, the score for each content item being based on the knowledge representation and the content of the respective content item; generating the training data for the machine learning classifier by assigning a label to each of the one or more content items in the first set based on a score associated with each of the one or more content items in the first set; At least one non-transitory computer-readable storage medium, [Appendix 26] The method comprises: synthesizing the knowledge representation based on the content of the object of interest; 26. The at least one non-transitory computer-readable storage medium of claim 25, further comprising: [Appendix 27] The synthesizing step comprises: generating the at least one concept and / or relationships between two or more concepts, wherein the at least one concept and / or relationships between two or more concepts are not enumerated in the object of interest; 27. At least one non-transitory computer-readable storage medium according to claim 26, comprising: [Appendix 28] 26. The at least one non-transitory computer-readable storage medium of claim 25, wherein the knowledge representation includes a weight associated with the at least one concept. [Appendix 29] 26. The at least one non-transitory computer-readable storage medium of claim 25, wherein the score for each content item is based on an intersection of the at least one concept in the knowledge representation and the content of the each content item. [Appendix 30] 26. The at least one non-transitory computer-readable storage medium of claim 25, wherein the interest objects include topics, tweets, web pages, websites, documents, collections of documents, document titles, messages, advertisements, and / or search queries. [Appendix 31] The method comprises: training a classifier to predict labels for one or more unclassified content items based on the labels assigned to the first set of content items and one or more features associated with the first set of content items; 26. The at least one non-transitory computer-readable storage medium of claim 25, further comprising: [Appendix 32] The method comprises: receiving a second set of content items, the second set including one or more content items that do not have a label; assigning, by the classifier, a label to each of the one or more content items in the second set based on one or more features associated with the content items; 32. The at least one non-transitory computer-readable storage medium of claim 31, further comprising: [Appendix 33] assigning the label to the one or more respective content items of the first set comprises: assigning the label based on a score for each content item in the first set exceeding a predetermined threshold. 26. At least one non-transitory computer-readable storage medium as recited in claim 25, comprising: [Appendix 34] 26. The at least one non-transitory computer-readable storage medium of claim 25, wherein the label is one of “recommended” and “not recommended.” [Appendix 35] 32. The at least one non-transitory computer-readable storage medium of claim 31, wherein the one or more features include at least one of title, length, author, word frequency, and inverse document frequency. [Appendix 36] 32. The at least one non-transitory computer-readable storage medium of claim 31, wherein the one or more characteristics include one or more attributes of the knowledge representation. [Appendix 37] 1. A method of classifying at least one unlabeled content item with a machine learning classifier, comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving training data, the training data including a first set of one or more labeled content items having labels that classify each content item into one or more categories; training the machine learning classifier with at least one feature of the labeled content items based on one or more attributes derived from the knowledge representation; classifying, with the machine learning classifier, the at least one unlabeled content item into the one or more categories using at least one feature of the at least one unlabeled content item based on the one or more attributes derived from the knowledge representation; A method comprising: [Appendix 38] synthesizing the knowledge representation based on the content of the object of interest; 38. The method of claim 37, further comprising: [Appendix 39] The synthesizing step comprises: generating the at least one concept and / or relationships between two or more concepts, wherein the at least one concept and / or relationships between two or more concepts are not enumerated in the object of interest; 39. The method of claim 38, comprising: [Appendix 40] 38. The method of claim 37, wherein the knowledge representation includes a weight associated with the at least one concept. [Appendix 41] 38. The method of claim 37, wherein the at least one feature of the at least one unlabeled content item is based on an intersection of the one or more attributes derived from the knowledge representation. [Appendix 42] 38. The method of claim 37, wherein the interest objects include topics, tweets, web pages, websites, documents, collections of documents, document titles, messages, advertisements, and / or search queries. [Appendix 43] receiving verification data, the verification data including a second set of one or more labeled content items; classifying, with the machine learning classifier, the validation data into the one or more categories using at least one feature of the second set of one or more labeled content items based on the one or more attributes derived from the knowledge representation; 38. The method of claim 37, further comprising: [Appendix 44] comparing the classified category for each of the second set of one or more labeled content items to one or more labels for each of the second set of one or more labeled content items; determining the number of correct classifications and the number of incorrect classifications made by the machine learning classifier; retraining the machine learning classifier based on the number of correct classifications and the number of incorrect classifications; 44. The method of claim 43, further comprising: [Appendix 45] The at least one characteristic of the labeled content item is: a total number of concepts common between the knowledge representation and the at least one unlabeled content item; and the number of broader concepts common between the knowledge representation and the at least one unlabeled content item; and the number of narrower concepts common between the knowledge representation and the at least one unlabeled content item; and a sum of the weights of common concepts between the knowledge representation and the at least one unlabeled content item; 38. The method of claim 37, comprising at least one of: [Appendix 46] 38. The method of claim 37, wherein the knowledge representation is user-specific. [Appendix 47] 38. The method of claim 37, wherein the knowledge representation is group-specific. [Appendix 48] 38. The method of claim 37, wherein the label is one of "recommended" and "not recommended." [Appendix 49] 38. The method of claim 37, wherein the label is one of "is a duplicate" and "is not a duplicate." [Appendix 50] Training the machine learning classifier includes: training at least two machine learning classifiers, wherein a first of the at least two machine learning classifiers is trained using first features of the labeled content items based on the one or more attributes derived from the knowledge representation, and a second of the at least two machine learning classifiers is trained using second features of the labeled content items based on the one or more attributes derived from the knowledge representation, the first and second features being based on at least one different attribute; 38. The method of claim 37, comprising: [Appendix 51] 51. The method of claim 50, wherein the classified category for the one or more unlabeled content items is determined using the at least two machine learning classifiers. [Appendix 52] associating a weight with each of the at least two machine learning classifiers based on a respective number of correct classifications and a respective number of incorrect classifications by each of the at least two machine learning classifiers. 52. The method of claim 51, further comprising: [Appendix 53] 38. The method of claim 37, wherein the training data is generated based on scores for one or more unlabeled content items, the score for each unlabeled content item being based on the one or more attributes derived from the knowledge representation and content of the each unlabeled content item. [Appendix 54] 1. A system for classifying at least one unlabeled content item with a machine learning classifier, comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving training data, the training data including a first set of one or more labeled content items having labels that classify each content item into one or more categories; training the machine learning classifier with at least one feature of the labeled content items based on one or more attributes derived from the knowledge representation; classifying, with the machine learning classifier, the at least one unlabeled content item into the one or more categories using at least one feature of the at least one unlabeled content item based on the one or more attributes derived from the knowledge representation; At least one processor configured to perform a method including A system having: [Appendix 55] The method comprises: synthesizing the knowledge representation based on the content of the object of interest; 55. The system of claim 54, further comprising: [Appendix 56] The synthesizing step comprises: generating the at least one concept and / or relationships between two or more concepts, wherein the at least one concept and / or relationships between two or more concepts are not enumerated in the object of interest; 56. The system of claim 55, comprising: [Appendix 57] 55. The system of claim 54, wherein the knowledge representation includes a weight associated with the at least one concept. [Appendix 58] 55. The system of claim 54, wherein the at least one feature of the at least one unlabeled content item is based on an intersection of the one or more attributes derived from the knowledge representation. [Appendix 59] 55. The system of claim 54, wherein the interest objects include topics, tweets, web pages, websites, documents, collections of documents, document titles, messages, advertisements, and / or search queries. [Appendix 60] The method comprises: receiving verification data, the verification data including a second set of one or more labeled content items; classifying, with the machine learning classifier, the validation data into the one or more categories using at least one feature of the second set of one or more labeled content items based on the one or more attributes derived from the knowledge representation; 55. The system of claim 54, further comprising: [Appendix 61] The method comprises: comparing the classified category for each of the second set of one or more labeled content items to one or more labels for each of the second set of one or more labeled content items; determining the number of correct classifications and the number of incorrect classifications made by the machine learning classifier; retraining the machine learning classifier based on the number of correct classifications and the number of incorrect classifications; 61. The system of claim 60, further comprising: [Appendix 62] The at least one characteristic of the labeled content item is: a total number of concepts common between the knowledge representation and the at least one unlabeled content item; and the number of broader concepts common between the knowledge representation and the at least one unlabeled content item; and the number of narrower concepts common between the knowledge representation and the at least one unlabeled content item; and a sum of the weights of common concepts between the knowledge representation and the at least one unlabeled content item; 55. The system of claim 54, comprising at least one of: [Appendix 63] 55. The system of claim 54, wherein the knowledge representation is user-specific. [Appendix 64] 55. The system of claim 54, wherein the knowledge representation is group-specific. [Appendix 65] 55. The system of claim 54, wherein the label is one of "recommended" and "not recommended." [Appendix 66] 55. The system of claim 54, wherein the label is one of "is a duplicate" and "is not a duplicate." [Appendix 67] Training the machine learning classifier includes: training at least two machine learning classifiers, wherein a first of the at least two machine learning classifiers is trained using first features of the labeled content items based on the one or more attributes derived from the knowledge representation, and a second of the at least two machine learning classifiers is trained using second features of the labeled content items based on the one or more attributes derived from the knowledge representation, the first and second features being based on at least one different attribute; 55. The system of claim 54, comprising: [Appendix 68] 68. The system of claim 67, wherein the classified category for the one or more unlabeled content items is determined using the at least two machine learning classifiers. [Appendix 69] The method comprises: associating a weight with each of the at least two machine learning classifiers based on a respective number of correct classifications and a respective number of incorrect classifications by each of the at least two machine learning classifiers. 69. The system of claim 68, further comprising: [Appendix 70] 55. The system of claim 54, wherein the training data is generated based on scores for one or more unlabeled content items, the score for each unlabeled content item being based on the one or more attributes derived from the knowledge representation and content of the each unlabeled content item. [Appendix 71] At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method for classifying at least one unlabeled content item with a machine learning classifier, the method comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving training data, the training data including a first set of one or more labeled content items having labels that classify each content item into one or more categories; training the machine learning classifier with at least one feature of the labeled content items based on one or more attributes derived from the knowledge representation; classifying, with the machine learning classifier, the at least one unlabeled content item into the one or more categories using at least one feature of the at least one unlabeled content item based on the one or more attributes derived from the knowledge representation; At least one non-transitory computer-readable storage medium, [Appendix 72] The method comprises: synthesizing the knowledge representation based on the content of the object of interest; 72. The at least one non-transitory computer-readable storage medium of claim 71, further comprising: [Appendix 73] The synthesizing step comprises: generating the at least one concept and / or relationships between two or more concepts, wherein the at least one concept and / or relationships between two or more concepts are not enumerated in the object of interest; 73. At least one non-transitory computer-readable storage medium as recited in claim 72, comprising: [Appendix 74] 72. The at least one non-transitory computer-readable storage medium of claim 71, wherein the knowledge representation includes a weight associated with the at least one concept. [Appendix 75] 72. The at least one non-transitory computer-readable storage medium of claim 71, wherein the at least one feature of the at least one unlabeled content item is based on an intersection of the one or more attributes derived from the knowledge representation. [Appendix 76] At least one non-transitory computer-readable storage medium as described in Appendix 71, wherein the interest objects include topics, tweets, web pages, websites, documents, collections of documents, document titles, messages, advertisements, and / or search queries. [Appendix 77] The method comprises: receiving verification data, the verification data including a second set of one or more labeled content items; classifying, with the machine learning classifier, the validation data into the one or more categories using at least one feature of the second set of one or more labeled content items based on the one or more attributes derived from the knowledge representation; 72. The at least one non-transitory computer-readable storage medium of claim 71, further comprising: [Appendix 78] The method comprises: comparing the classified category for each of the second set of one or more labeled content items to one or more labels for each of the second set of one or more labeled content items; determining the number of correct classifications and the number of incorrect classifications made by the machine learning classifier; retraining the machine learning classifier based on the number of correct classifications and the number of incorrect classifications; 78. The at least one non-transitory computer-readable storage medium of claim 77, further comprising: [Appendix 79] The at least one characteristic of the labeled content item is: a total number of concepts common between the knowledge representation and the at least one unlabeled content item; and the number of broader concepts common between the knowledge representation and the at least one unlabeled content item; and the number of narrower concepts common between the knowledge representation and the at least one unlabeled content item; and a sum of the weights of common concepts between the knowledge representation and the at least one unlabeled content item; 72. At least one non-transitory computer-readable storage medium according to claim 71, comprising at least one of: [Appendix 80] 72. The at least one non-transitory computer-readable storage medium of claim 71, wherein the knowledge representation is user-specific. [Appendix 81] 72. The at least one non-transitory computer-readable storage medium of claim 71, wherein the knowledge representation is group-specific. [Appendix 82] 72. The at least one non-transitory computer-readable storage medium of claim 71, wherein the label is one of “recommended” and “not recommended.” [Appendix 83] 72. The at least one non-transitory computer-readable storage medium of claim 71, wherein the label is one of "is a copy" and "is not a copy." [Appendix 84] Training the machine learning classifier includes: training at least two machine learning classifiers, wherein a first of the at least two machine learning classifiers is trained using first features of the labeled content items based on the one or more attributes derived from the knowledge representation, and a second of the at least two machine learning classifiers is trained using second features of the labeled content items based on the one or more attributes derived from the knowledge representation, the first and second features being based on at least one different attribute; 72. At least one non-transitory computer-readable storage medium as recited in claim 71, comprising: [Appendix 85] 85. The at least one non-transitory computer-readable storage medium of Claim 84, wherein the classified category for the one or more unlabeled content items is determined using the at least two machine learning classifiers. [Appendix 86] The method comprises: associating a weight with each of the at least two machine learning classifiers based on a respective number of correct classifications and a respective number of incorrect classifications by each of the at least two machine learning classifiers. 86. The at least one non-transitory computer-readable storage medium of claim 85, further comprising: [Appendix 87] 72. The at least one non-transitory computer-readable storage medium of Claim 71, wherein the training data is generated based on scores for one or more unlabeled content items, the score for each unlabeled content item being based on the one or more attributes derived from the knowledge representation and content of the each unlabeled content item. [Appendix 88] 1. A method for modifying a knowledge representation based on a machine learning classifier, comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving verification data, the verification data including a first set of one or more labeled content items having labels that classify each content item into one or more categories, including a first category known to be relevant to the object of interest and a second category known not to be relevant to the object of interest; predicting each of the one or more labeled content items as one of: a) relevant to the object of interest; and b) not relevant to the object of interest with a machine learning classifier that uses at least one attribute derived from the knowledge representation as a feature; modifying the knowledge representation based on a comparison of the prediction by the machine learning classifier for each content item of the first set and a label for each content item; A method comprising: [Appendix 89] synthesizing the knowledge representation based on the content of the object of interest; 89. The method of claim 88, further comprising: [Appendix 90] The synthesizing step comprises: generating the at least one concept and / or relationships between two or more concepts, wherein the at least one concept and / or relationships between two or more concepts are not enumerated in the object of interest; 89. The method of claim 89, comprising: [Appendix 91] 89. The method of claim 88, wherein the knowledge representation includes a weight associated with the at least one concept. [Appendix 92] 89. The method of claim 88, wherein the predicting is based on an intersection of the one or more labeled content items with the features. [Appendix 93] 89. The method of claim 88, wherein the interest objects include topics, tweets, web pages, websites, documents, collections of documents, document titles, messages, advertisements, and / or search queries. [Appendix 94] After modifying the knowledge representation, re-predicting each of the first set of one or more labeled content items using the modified knowledge representation; modifying the knowledge representation based on a comparison of the prediction by the machine learning classifier for each content item in the first set and the label of each content item; 89. The method of claim 88, further comprising: [Appendix 95] 95. The method of claim 94, wherein the re-predicting and modifying are repeated until a ratio of the number of the one or more labeled content items correctly predicted as being relevant to the object of interest to the total number of labeled content items in the first category is greater than or equal to a precision threshold. [Appendix 96] 95. The method of claim 94, wherein the re-predicting and modifying are repeated until a ratio of the number of the one or more labeled content items correctly predicted as being relevant to the object of interest to the total number of the one or more labeled content items predicted as being relevant to the object of interest is greater than or equal to a recall threshold. [Appendix 97] Modifying the knowledge representation comprises: modifying a weight associated with said at least one concept in said knowledge representation; and / or Adding additional concepts to the knowledge representation 89. The method of claim 88, comprising: [Appendix 98] Modifying the knowledge representation based on the comparison comprises: modifying the knowledge representation if a ratio of the number of the one or more labeled content items that are correctly predicted as being relevant to the object of interest to a total number of the one or more labeled content items in the first category is less than a precision threshold. 89. The method of claim 88, comprising: [Appendix 99] Modifying the knowledge representation based on the comparison comprises: modifying the knowledge representation if a ratio of the number of the one or more labeled content items correctly predicted as being relevant to the object of interest to the total number of the one or more labeled content items predicted as being relevant to the object of interest is less than a recall threshold. 89. The method of claim 88, comprising: [Appendix 100] The at least one attribute is: the total number of concepts common between the knowledge representation and the one or more labeled content items; the number of broader concepts common between the knowledge representation and the one or more labeled content items; and the number of narrower concepts common between the knowledge representation and the one or more labeled content items; and a sum of the weights of common concepts between the knowledge representation and the one or more labeled content items; 89. The method of claim 88, comprising at least one of: [Appendix 101] 1. A system for modifying a knowledge representation based on a machine learning classifier, comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving verification data, the verification data including a first set of one or more labeled content items having labels that classify each content item into one or more categories, including a first category known to be relevant to the object of interest and a second category known not to be relevant to the object of interest; predicting each of the one or more labeled content items as one of: a) relevant to the object of interest; and b) not relevant to the object of interest with a machine learning classifier that uses at least one attribute derived from the knowledge representation as a feature; modifying the knowledge representation based on a comparison of the prediction by the machine learning classifier for each content item of the first set and a label for each content item; At least one processor configured to perform a method including A system having: [Appendix 102] The method comprises: synthesizing the knowledge representation based on the content of the object of interest; 102. The system of claim 101, further comprising: [Appendix 103] The synthesizing step comprises: generating the at least one concept and / or relationships between two or more concepts, wherein the at least one concept and / or relationships between two or more concepts are not enumerated in the object of interest; 103. The system of claim 102, comprising: [Appendix 104] 102. The system of claim 101, wherein the knowledge representation includes a weight associated with the at least one concept. [Appendix 105] 102. The system of claim 101, wherein the predicting is based on an intersection of the one or more labeled content items and the features. [Appendix 106] 102. The system of claim 101, wherein the interest objects include topics, tweets, web pages, websites, documents, collections of documents, document titles, messages, advertisements, and / or search queries. [Appendix 107] After modifying the knowledge representation, the method further comprises: re-predicting each of the first set of one or more labeled content items using the modified knowledge representation; modifying the knowledge representation based on a comparison of the prediction by the machine learning classifier for each content item in the first set and the label of each content item; 102. The system of claim 101, further comprising: [Appendix 108] 108. The system of claim 107, wherein the re-predicting and modifying are repeated until a ratio of the number of the one or more labeled content items correctly predicted as being relevant to the object of interest to the total number of labeled content items in the first category is greater than or equal to a precision threshold. [Appendix 109] 108. The system of claim 107, wherein the re-predicting and modifying are repeated until a ratio of the number of the one or more labeled content items correctly predicted as being relevant to the object of interest to the total number of the one or more labeled content items predicted as being relevant to the object of interest is greater than or equal to a recall threshold. [Appendix 110] Modifying the knowledge representation comprises: modifying a weight associated with said at least one concept in said knowledge representation; and / or Adding additional concepts to the knowledge representation 102. The system of claim 101, comprising: [Appendix 111] Modifying the knowledge representation based on the comparison comprises: modifying the knowledge representation if a ratio of the number of the one or more labeled content items that are correctly predicted as being relevant to the object of interest to a total number of the one or more labeled content items in the first category is less than a precision threshold. 102. The system of claim 101, comprising: [Appendix 112] Modifying the knowledge representation based on the comparison comprises: modifying the knowledge representation if a ratio of the number of the one or more labeled content items correctly predicted as being relevant to the object of interest to the total number of the one or more labeled content items predicted as being relevant to the object of interest is less than a recall threshold. 102. The system of claim 101, comprising: [Appendix 113] The at least one attribute is: the total number of concepts common between the knowledge representation and the one or more labeled content items; the number of broader concepts common between the knowledge representation and the one or more labeled content items; and the number of narrower concepts common between the knowledge representation and the one or more labeled content items; and a sum of the weights of common concepts between the knowledge representation and the one or more labeled content items; 102. The system of claim 101, comprising at least one of: [Appendix 114] At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method for modifying a knowledge representation based on a machine learning classifier, the method comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving verification data, the verification data including a first set of one or more labeled content items having labels that classify each content item into one or more categories, including a first category known to be relevant to the object of interest and a second category known not to be relevant to the object of interest; predicting each of the one or more labeled content items as one of: a) relevant to the object of interest; and b) not relevant to the object of interest with a machine learning classifier that uses at least one attribute derived from the knowledge representation as a feature; modifying the knowledge representation based on a comparison of the prediction by the machine learning classifier for each content item of the first set and a label for each content item; At least one non-transitory computer-readable storage medium, [Appendix 115] The method comprises: synthesizing the knowledge representation based on the content of the object of interest; 115. The at least one non-transitory computer-readable storage medium of claim 114, further comprising: [Appendix 116] The synthesizing step comprises: generating the at least one concept and / or relationships between two or more concepts, wherein the at least one concept and / or relationships between two or more concepts are not enumerated in the object of interest; 116. At least one non-transitory computer-readable storage medium as described in Clause 115, comprising: [Appendix 117] 115. The at least one non-transitory computer-readable storage medium of Clause 114, wherein the knowledge representation includes a weight associated with the at least one concept. [Appendix 118] 115. The at least one non-transitory computer-readable storage medium of Clause 114, wherein the predicting is based on an intersection of the one or more labeled content items and the features. [Appendix 119] At least one non-transitory computer-readable storage medium as described in Appendix 114, wherein the interest objects include topics, tweets, web pages, websites, documents, collections of documents, document titles, messages, advertisements, and / or search queries. [Appendix 120] After modifying the knowledge representation, the method further comprises: re-predicting each of the first set of one or more labeled content items using the modified knowledge representation; modifying the knowledge representation based on a comparison of the prediction by the machine learning classifier for each content item in the first set and the label of each content item; 115. The at least one non-transitory computer-readable storage medium of claim 114, further comprising: [Appendix 121] 121. The at least one non-transitory computer-readable storage medium of claim 120, wherein the re-predicting and modifying are repeated until a ratio of the number of the one or more labeled content items correctly predicted as being relevant to the object of interest to the total number of labeled content items in the first category is greater than or equal to a precision threshold. [Appendix 122] 121. The at least one non-transitory computer-readable storage medium of Claim 120, wherein the re-predicting and modifying are repeated until a ratio of the number of the one or more labeled content items correctly predicted as being relevant to the object of interest to the total number of the one or more labeled content items predicted as being relevant to the object of interest is greater than or equal to a recall threshold. [Appendix 123] Modifying the knowledge representation comprises: modifying a weight associated with said at least one concept in said knowledge representation; and / or Adding additional concepts to the knowledge representation 115. At least one non-transitory computer-readable storage medium as recited in Clause 114, comprising: [Appendix 124] Modifying the knowledge representation based on the comparison comprises: modifying the knowledge representation if a ratio of the number of the one or more labeled content items that are correctly predicted as being relevant to the object of interest to a total number of the one or more labeled content items in the first category is less than a precision threshold. 115. At least one non-transitory computer-readable storage medium as recited in Clause 114, comprising: [Appendix 125] Modifying the knowledge representation based on the comparison comprises: modifying the knowledge representation if a ratio of the number of the one or more labeled content items correctly predicted as being relevant to the object of interest to the total number of the one or more labeled content items predicted as being relevant to the object of interest is less than a recall threshold. 115. At least one non-transitory computer-readable storage medium as recited in Clause 114, comprising: [Appendix 126] The at least one attribute is: the total number of concepts common between the knowledge representation and the one or more labeled content items; the number of broader concepts common between the knowledge representation and the one or more labeled content items; and the number of narrower concepts common between the knowledge representation and the one or more labeled content items; and a sum of the weights of common concepts between the knowledge representation and the one or more labeled content items; 115. At least one non-transitory computer-readable storage medium as described in Clause 114, comprising at least one of:

Claims

[Claim 1] 1. A method for modifying a knowledge representation based on a machine learning classifier, comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation including at least one concept and / or a relationship between two or more concepts; receiving verification data, the verification data including a first set of one or more labeled content items having labels that classify each content item into one or more categories, including a first category known to be relevant to the object of interest and a second category known not to be relevant to the object of interest; predicting each of the one or more labeled content items as one of a) relevant to the object of interest, or b) not relevant to the object of interest, with a machine learning classifier that uses at least one attribute derived from the knowledge representation as a feature; modifying the knowledge representation based on a comparison of the prediction by the machine learning classifier for each content item of the first set and a label for each respective content item; A method comprising:

Images