Systems and methods for using knowledge representation with machine learning classifiers

Through machine learning classifiers and training data generation methods of knowledge representation, information overload and identification challenges in digital information systems are solved, accurate classification and labeling of information of users are realized, and information retrieval efficiency is improved.

CN110168579BActive Publication Date: 2024-12-06PRIMAL FUSION INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201680091990.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2016-11-23
Publication Date
2024-12-06
Estimated Expiration
2036-11-23

AI Technical Summary

Technical Problem

In digital information systems, it becomes particularly challenging to determine information of interest to the user and effectively identify other information to avoid the problem of information overload, especially when information is distributed across large networks.

Method used

Using a machine learning classifier to generate training data, using knowledge representation encoded as a non-transitory computer-readable data structure, a collection of content items is received and tagged to it, and training data is generated to classify information. The method includes receiving the encoded knowledge representation, determining the scores of the content items, and assigning tags based on the scores to generate training data.

Benefits of technology

It realizes accurate classification and marking of a large amount of content, improves the efficiency of information retrieval, reduces the presentation of irrelevant information, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110168579B_ABST
    Figure CN110168579B_ABST
Patent Text Reader

Abstract

A system and method for generating training data for a machine learning classifier is provided. A knowledge representation synthesized based on an object of interest is used to assign tags to content items. The labeled content items can be used as training data for training a machine learning classifier. The labeled content items can also be used as validation data for the classifier.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The teachings described herein relate to the fields of machine learning and information retrieval. Specifically, the teachings disclosed herein relate to deploying methods in a digital information system environment for using information associated with one or more data sets expressed as knowledge representations to classify information, training a classifier model to classify information and / or using the classifier results to refine the knowledge representations. Background Art

[0002] Information technology is often used to provide various types of information to users, such as text, audio, video, and any other suitable type of information. In some cases, information is provided to users in response to actions taken by users. For example, information may be provided to users in response to a search query entered by a user or in response to content such as email alerts or electronic newsletters that the user has subscribed to. In other cases, such information is provided or "pushed" to users without the user specifically requesting the information. For example, advertisements or solicitations may occasionally be presented to users.

[0003] There is a vast amount of content and information that can be provided to users via information technology. Indeed, because a vast amount of information is available via the Internet, the World Wide Web (WWW), and any other suitable information supply source, and because the available information is distributed across a large number of independently owned and operated networks and servers, locating information of interest to a user presents a challenge. Similar challenges exist when the information of interest is distributed across large private networks. Summary of the invention

[0004] The inventive concepts presented herein are illustrated in a number of different embodiments, each of which illustrates one or more concepts, but it should be understood that, in general, these concepts are not mutually exclusive and may be used in combination even when not illustrated.

[0005] Some embodiments provide a method for generating training data for a machine learning classifier, the method comprising: receiving, based on an object of interest, a knowledge representation encoded as a non-transitory computer-readable data structure, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving a first set of content items, wherein the first set comprises one or more content items without labels, wherein the labels classify the content items into one or more categories; determining one or more scores for one or more corresponding content items in the first set, wherein the scores of the corresponding content items are based on the knowledge representation and content of the corresponding content items; and generating training data for the machine learning classifier by assigning labels to the one or more corresponding content items in the first set based on the scores associated with the one or more corresponding content items in the first set.

[0006] Some embodiments provide a system for generating training data for a machine learning classifier, the system comprising: at least one processor configured to execute a method comprising: receiving, based on an object of interest, a knowledge representation encoded as a non-transitory computer-readable data structure, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving a first set of content items, wherein the first set comprises one or more content items without labels, wherein the labels classify the content items into one or more categories; determining one or more scores for one or more corresponding content items in the first set, wherein the scores of the corresponding content items are based on the knowledge representation and content of the corresponding content items; and generating training data for the machine learning classifier by assigning labels to the one or more corresponding content items in the first set based on the scores associated with the one or more corresponding content items in the first set.

[0007] Some embodiments provide at least one non-volatile computer-readable storage medium storing processor-executable instructions, which, when executed by at least one processor, cause the at least one processor to perform a method for generating training data for a machine learning classifier, the method comprising: receiving, based on an object of interest, a knowledge representation encoded as a non-volatile computer-readable data structure, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving a first set of content items, wherein the first set comprises one or more content items without labels, wherein the labels classify the content items into one or more categories; determining one or more scores for one or more corresponding content items in the first set, wherein the scores of the corresponding content items are based on the knowledge representation and the content of the corresponding content items; and generating training data for the machine learning classifier by assigning labels to the one or more corresponding content items in the first set based on the scores associated with the one or more corresponding content items in the first set.

[0008] Some embodiments provide a method for classifying at least one unlabeled content item using a machine learning classifier, the method comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving training data, the training data comprising a first set of one or more labeled content items with a label, the label classifying each content item into one or more categories; training a machine learning classifier using at least one feature of the labeled content item based on one or more attributes obtained from the knowledge representation; and classifying at least one unlabeled content item into one or more categories using the machine learning classifier using at least one feature of the unlabeled content item based on the one or more attributes obtained from the knowledge representation.

[0009] Some embodiments provide a system for classifying at least one unlabeled content item using a machine learning classifier, the system comprising: at least one processor configured to execute a method comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving training data comprising a first set of one or more labeled content items with a label, the label classifying each content item into one or more categories; training a machine learning classifier using at least one feature of the labeled content item based on one or more attributes obtained from the knowledge representation; and classifying at least one unlabeled content item into one or more categories using at least one feature of the unlabeled content item based on the one or more attributes obtained from the knowledge representation using the machine learning classifier.

[0010] Some embodiments provide at least one non-volatile computer-readable storage medium storing processor-executable instructions, which, when executed by at least one processor, cause the at least one processor to perform a method for classifying at least one unlabeled content item using a machine learning classifier, the method comprising: receiving a knowledge representation encoded as a non-volatile computer-readable data structure based on an object of interest, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving training data, the training data comprising a first set of one or more labeled content items with a label, the label classifying each content item into one or more categories; training a machine learning classifier using at least one feature of the labeled content item based on one or more attributes obtained from the knowledge representation; and classifying at least one unlabeled content item into one or more categories using at least one feature of the unlabeled content item based on the one or more attributes obtained from the knowledge representation using the machine learning classifier.

[0011] Some embodiments provide a method for modifying a knowledge representation based on a machine learning classifier, the method comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving verification data comprising a first set of one or more tagged content items having a label, the label classifying each content item into one or more categories, the one or more categories comprising a first category known to be related to the object of interest and a second category known to be unrelated to the object of interest; using a machine learning classifier that uses at least one attribute obtained from the knowledge representation as a feature to predict each of the one or more tagged content items as one of the following: a) related to the object of interest or b) unrelated to the object of interest; and modifying the knowledge representation based on a comparison of the prediction of the machine learning classifier for each content item in the first set with the label of each corresponding content item.

[0012] Some embodiments provide a system for modifying a knowledge representation based on a machine learning classifier, the system comprising: at least one processor configured to execute a method comprising: receiving a knowledge representation encoded as a non-transitory computer-readable data structure based on an object of interest, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving verification data comprising a first set of one or more tagged content items with a label, the label classifying each content item into one or more categories, the one or more categories comprising a first category known to be related to the object of interest and a second category known to be unrelated to the object of interest; using a machine learning classifier using at least one attribute obtained from the knowledge representation as a feature to predict each of the one or more tagged content items as one of the following: a) related to the object of interest or b) unrelated to the object of interest; and modifying the knowledge representation based on a comparison of the prediction of the machine learning classifier for each content item in the first set with the label of each corresponding content item.

[0013] Some embodiments provide at least one non-volatile computer-readable storage medium storing processor-executable instructions, which, when executed by at least one processor, cause the at least one processor to perform a method for modifying a knowledge representation based on a machine learning classifier, the method comprising: receiving a knowledge representation encoded as a non-volatile computer-readable data structure based on an object of interest, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving verification data, the verification data comprising a first set of one or more tagged content items with a label, the label classifying each content item into one or more categories, the one or more categories comprising a first category known to be related to the object of interest and a second category known to be unrelated to the object of interest; using a machine learning classifier that uses at least one attribute obtained from the knowledge representation as a feature to predict each of the one or more tagged content items as one of the following: a) related to the object of interest or b) unrelated to the object of interest; and modifying the knowledge representation based on a comparison of the prediction of the machine learning classifier for each content item in the first set with the label of each corresponding content item.

[0014] The above is a non-limiting summary of the present invention as defined by the appended claims, with the understanding that this summary does not necessarily describe the subject matter of each claim and that each claim is directed to one or some but not all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings are not intended to be drawn to scale. Where practical, similar elements are identified by the same or similar reference numerals. For the sake of clarity, not every component may be labeled in every drawing. In the drawings:

[0016] Figure 1 is a block diagram illustrating an exemplary computing system for use in practicing some embodiments of the present invention;

[0017] Figure 2 is a flow chart illustrating an exemplary workflow for supervised learning techniques according to some embodiments of the present invention;

[0018] Figure 3 is a flow chart illustrating an exemplary workflow for synthesizing a knowledge representation based on an object of interest according to some embodiments of the present invention;

[0019] Figure 4 is an illustration of an example object of interest according to some embodiments of the present invention;

[0020] Figure 5 is a diagram of an exemplary workflow for extracting a subject uniform resource identifier based on an object of interest according to some embodiments of the present invention;

[0021] Figure 6 is an illustration of an exemplary workflow for synthesizing a knowledge representation based on subject uniform resource identifiers extracted from an object of interest, according to some embodiments;

[0022] Figure 7 is a flow chart illustrating an exemplary method for marking content items according to some embodiments of the present invention;

[0023] Fig. 8A is a diagram of an exemplary workflow for training a classifier model according to some embodiments of the present invention;

[0024] Figure 8B According to some embodiments of the present invention, Fig. 8A an illustration of an exemplary workflow for labeling content items using a classifier model based on the

[0025] Fig. 9 is an illustration of an exemplary workflow for analyzing a content item based on features derived from a knowledge representation based on an object of interest;

[0026] Fig. 10A is an illustration of an exemplary workflow for training a classifier model using features from a knowledge representation derived from an object of interest according to some embodiments of the present invention;

[0027] Fig. 10B According to some embodiments of the present invention, Fig. 10A an illustration of an exemplary workflow for labeling content items using a classifier model based on the

[0028] Fig.11 is a diagram showing an example table of attribute-based features associated with content items, known relevance of the content items, and predictions by a classifier according to some embodiments of the present invention;

[0029] Fig.12 is a diagram showing an example table of attribute-based features associated with content items, known relevance of the content items, and predictions by a first classifier and a second classifier according to some embodiments of the present invention;

[0030] Fig.13 is a diagram showing an example table of prediction results made by multiple classifier models according to some embodiments of the present invention;

[0031] Fig.14A is a diagram of an example table showing the accuracy of labels predicted by a classifier according to some embodiments of the present invention;

[0032] Fig. 14B is a graphical illustration of the accuracy of a classifier according to some embodiments of the present invention;

[0033] Fig.15A is a diagram showing an example table of attribute values ​​associated with a modified knowledge representation and the accuracy of labels predicted by a classifier according to some embodiments of the present invention;

[0034] Fig. 15B According to some embodiments of the present invention Fig.15A A graphical illustration of the accuracy of the classifier; and

[0035] Fig.16 is an illustration of an exemplary flow chart for modifying a knowledge representation based on results of a classifier according to some embodiments of the invention. DETAILED DESCRIPTION

[0036] The vast amount of content accessible via digital information systems presents many information retrieval challenges. One challenging problem is how to determine what information among the vast amount of content may be of interest to a user so that such information can be presented to the user without overwhelming the user with irrelevant information. Moreover, another challenging problem is how to identify information of potential interest in a computationally efficient manner.

[0037] The systems and methods described herein can be implemented in various ways. Figure 1 is a block diagram illustrating an exemplary computing system for use in practicing some embodiments of the invention described herein.

[0038] The computing device 100 may include one or more processors (e.g., microprocessors) 110 and one or more tangible, non-transient computer-readable storage media (e.g., memory 120). The memory 120 may store computer instructions that implement any functionality described herein in a tangible, non-transient computer-readable storage medium. (Multiple) processors 110 may be coupled to the memory 120 and may execute such computer instructions so that the functionality is implemented and executed. The computing device 100 may also include a network input / output (I / O) interface 130, via which the computing device 100 may communicate with other computers (e.g., over a network). In some embodiments, the computing device may also include one or more user I / O interfaces 140, via which the computer may provide output to a user and receive input from a user. The user I / O interface 140 may include devices such as a keyboard, a mouse, a microphone, a display device (e.g., a monitor or a touch screen), a speaker, a camera, and / or various other types of I / O devices.

[0039] Some embodiments of the present invention use machine learning algorithms to train classifiers, in particular, supervised learning. Supervised learning is a branch of machine learning that involves the task of inferring functions from labeled training data. The training data includes a set of training examples. In supervised learning, each training example can include an object (e.g., a content item) and a label (e.g., it can classify the content item into one or more categories). The supervised learning algorithm can analyze the training data and generate an inference function (e.g., a classifier model), which can then be used to predict labels for unlabeled content items (or otherwise classify them).

[0040] A machine learning classifier trained using, for example, supervised learning can be used to assign labels to unlabeled content items. However, in order to train the classifier, a large amount of training data is required. In some embodiments, the training data includes content items with labels that classify each content item into a category. Without sufficient training data to train the classifier, it is difficult for the machine learning algorithm to produce a model that produces accurate results.

[0041] Figure 2 2 is a flowchart illustrating an exemplary workflow for supervised learning techniques according to some embodiments of the present invention. The process begins with raw data 210. Raw data 210 may include one or more content items. To facilitate machine learning purposes, raw data 210 is converted into characterization data with labels 220 by assigning labels to each content item in raw data 210. Labels may be binary in nature (e.g., "yes" or "no" or "1" or "0" labels). However, there may also be more than two possible labels. Labels may be assigned to raw data 210 in any number of ways. For example, a human operator may manually review each item in raw data 210 and assign labels.

[0042] Once labeled, the characterization data 220 may be divided into a training set 230 and a validation set 240. The characterization data may be divided between the training set 230 and the validation set 240 in any proportion. The training set 230 is then used as a basis for building (hereinafter referred to as training) a model 250. The model 250 may be conceptualized as a function inferred from the correlation between known labels for content items and various features of the content items. One of the purposes of the model 250 is to assign labels to unlabeled content items based on their features in a manner consistent with the labels assigned to the training set 230 and the features associated with the training set 230.

[0043] Once the model 250 is trained, the model 250 can be validated using the validation set 240 at 260. One way to validate the model 250 is to apply the model 250 to the validation set 240 and obtain labels for each content item in the validation set 240. Since the validation set 240 already contains content items with known labels, the known labels can then be compared to the labels generated by the model 250 to determine the accuracy of the model 250 in correctly predicting the labels.

[0044] The validated model 250 can then be applied to unlabeled content items and used to generate labels. In this way, a large amount of data can be labeled and / or classified by the model within a reasonable degree of accuracy. However, as mentioned above, a large amount of training data is generally required in order to train the model. It is desirable to obtain the training data in a manner that is computationally efficient and does not require manually labeled content items.

[0045] Figure 3 3 is a flow chart illustrating an exemplary workflow for synthesizing a knowledge representation based on an object of interest according to some embodiments of the present invention. An object of interest ("OOI") 310 is analyzed, and terms and / or other attributes contained in the OOI 310 are used to generate one or more subject uniform resource identifiers 320 (subject URIs). The subject URIs 320 are then used as a basis for synthesizing a knowledge representation 330 that includes a plurality of additional concepts and / or relationships between concepts, some of which are not found in the OOI 310.

[0046] Figure 4 is an illustration of an example OOI according to some embodiments of the present invention. Figure 4 The exemplary OOI 310 in is a single web page, but it should be understood that the OOI can be extended to a website or multiple websites covering multiple web pages, and can include any suitable type of unstructured data consisting of one / multiple corpora presenting items of interest. In some embodiments, the OOI 310 may include topics, tweets, web pages, websites, documents, document titles, messages, advertisements, and / or search queries; in some embodiments, the OOI 310 may be a single instance of the aforementioned objects (e.g., a simple topic, a single document, etc.), or may include a collection of the objects (e.g., a collection of topics related to personal interests, a set of documents related to personal interests, etc.). Moreover, the OOI 310 may include various types of items related to a particular user (e.g., a user's tweets and documents of interest to the same user), and may be extended to items across various users (e.g., items related to a particular market or demographic segment of interest). The OOI 310 may include both online objects and offline objects.

[0047] As will be appreciated, by varying the scope of content contained within an OOI 310, the number and depth of topic URIs 320 extracted therefrom will also vary, and the resulting synthesized knowledge representation 330 will in turn vary. Thus, a balance between the concentration and quantity of concepts and / or relationships contained within a knowledge representation 330 may be achieved. For example, an OOI 310 containing more extended data will represent a larger domain and provide a greater scope over which to generate topic URIs 320 and synthesize data. In contrast, when a topic (such as a phrase of a few words) is used as an OOI 310, that topic will be the only lens used to synthesize the data.

[0048] Figure 5 is a diagram of an exemplary workflow for extracting a subject URI based on an object of interest according to some embodiments of the present invention. For clarity, Figure 5 The OOI 310 shown is Figure 4 310. The topic URI 320 generated from the OOI 310 is shown in both a linear and hierarchical manner. As can be seen, the term "skin" is identified, and the sub-topics of conditions, symptoms, and blockages are shown as being related to skin. It should be noted that the hierarchical and linear representations of the topic URI 320 are two different ways of representing the same relationship.

[0049] Figure 6 is an illustration of an exemplary workflow for synthesizing a knowledge representation 330 based on a subject URI 320 generated from an object of interest 310, according to some embodiments. As can be seen, the knowledge representation 330 contains additional concepts, namely, "organ", "brain", "cancer", and "redness", which are identified as having relationships with other concepts in the subject URI 320. For example, these additional concepts and the relationships between concepts can be obtained from a source external to the object of interest. For example, a thesaurus, a dictionary, other content items, or various tautologies can be used to identify the additional concepts and relationships. As mentioned above, it should be appreciated that the knowledge representation 330 can be the result of various types of objects of interest, including a wide range of unstructured data, such as a collection of documents that are known to be of interest to a user.

[0050] It should be noted that although specific examples are provided herein, there are many ways to synthesize knowledge representation 330. For example, further examples can be found in U.S. Patent No. 9,378,203 and U.S. Patent Application No. 13 / 345,637, the contents of which are incorporated herein by reference in their entirety. In some embodiments, an atomic knowledge representation model (AKRM) may include one or more basic data structures and one or more knowledge processing rules. In some embodiments, the system can use these rules to deconstruct (analyze) one or more complex KRs to generate basic data structures. For example, the system may include one or more computer processors and one or more computer memory hardware components, and the memory may be encoded with computer executable instructions, which, when executed by one or more processors, cause one or more processors of the system to generate the basic data structure of the AKRM using these rules when analyzing one or more complex KRs. The memory may also be encoded with instructions for programming one or more processors to use these rules to synthesize new complex KRs from basic data structures. In some embodiments, the computer memory may be implemented as one or more tangible non-transitory computer-readable storage media encoded with computer executable instructions, which, when executed, cause one or more processors to perform any of the functions described herein.

[0051] The synthesized knowledge representation 330 can be used to evaluate the relevance of other content items. Figure 6 For example, other content items may be analyzed for the presence of one or more concepts found in the synthesized knowledge representation. A score may be generated based on this analysis, which may be used as an indicator of the potential relevance of the content item relative to OOI 310. Moreover, weights may be assigned to different concepts in synthesized knowledge representation 330. Such weights may be used to indicate that a particular concept in knowledge representation 330 is more likely to be associated with a relevance than another concept in the knowledge representation. For example, a score for a content item may be determined by obtaining a weighted sum of the concepts. In some embodiments, a score for a content item may be determined based on the intersection of one or more concepts in knowledge representation 330 with the content of the content item.

[0052] Figure 7is a flowchart illustrating an exemplary method for content items marked according to scores according to some embodiments of the present invention. At 710, one or more scored content items are obtained. In some embodiments, the items are scored based on the knowledge representation 330, which is based on the object of interest 310. At 720, each content item is compared with a predetermined threshold score. If the score for the content item is greater than the threshold, then the content item is marked as "recommended" at 730. If the score for the content item is less than or equal to the threshold score, then the content item is marked as "not recommended". Then, the content items with "recommended" and "not recommended" labels form a set of marked content items 750. In some embodiments, content items can be ranked by scores, and labels can be assigned based on the ranking of content items within the set, rather than purely based on scores. For example, for the ten scored content items received at 710, at 720, it can be determined whether the content item ranks in the top three of the total ten content items.

[0053] It should be noted that although Figure 7 Two possible labels for content items are shown, but the present invention also contemplates the use of more than two labels, and for multiple layers of labels (e.g., multiple label options), rather than binary labels (e.g., "yes" or "no"). For example, the labels may include "recommended", the next lowest layer being "suggested", followed by "not recommended". These layers may be extended to any suitable number of labels and / or categories. Furthermore, the labels need not be hierarchical, but associated categories may be established (e.g., sports, politics, technology, etc.).

[0054] As mentioned above, the set of tagged content items 750 includes one or more content items having tags associated therewith. Fig. 8A 8 is an illustration of an exemplary workflow for training a classification model 820 according to some embodiments of the present invention. A collection of tagged content items 810 may be used as training data (e.g., the training set 230 described herein) to build a machine learning classifier model 820. In some embodiments, by Figure 7 The collection of tagged content items 810 is generated using the process described in .

[0055] In some embodiments, for each tagged content item, training a classification model 820 based on the tagged content item 810 includes: a comparison of various features of the content item and the tags associated with the content item. In some embodiments, these features may include standard features, examples of which include title, length, author, term frequency, and / or inverse document frequency. In some embodiments, these features may include one or more attributes of knowledge representation. The classification model 820 includes an inference function that can be used to determine which tag should be assigned to a specific content item based on the features of the specific content item.

[0056] Figure 8B According to some embodiments of the present invention, Fig. 8A 820 to label content items. Once the classification model 820 has been trained, and optionally validated using additional labeled validation data, the classification model 820 can be used to assign labels to a collection of one or more unlabeled content items 830. This classification is based on the same features used to train the classification model 820. It should be appreciated that such features (e.g., title, length, author, term frequency, and / or inverse document frequency) can be easily determined from the content items, with minimal or no analysis required by a human operator. In this way, labeling unlabeled content items 830 using the classification model 820 can be performed relatively quickly and efficiently. Applying the classification model 820 to the unlabeled content items 830 produces a second set of labeled content items 850. The labeled content items 850, which have been assigned favorable labels (e.g., "recommended" or "suggested"), can then be sent to a user or user group as content that may be of interest.

[0057] The foregoing process can be considered similar to an unsupervised learning approach because the classifier does not require any supervision (i.e., training data or unlabeled content) to produce useful classifications. Unsupervised learning is a machine learning task that infers functions that describe hidden structures from unlabeled data. By identifying intersections between knowledge representations 330 and unlabeled content items 830, useful representations / features of unlabeled content items 830 (i.e., hidden structures that include semantic representations specific to the object of interest 310) are generated. Furthermore, Figure 7 The process can be extended to techniques for clustering content items around features of knowledge representation (described in further detail below).

[0058] Fig. 9 is an illustration of an exemplary workflow for classifying unlabeled content items using attributes derived from a knowledge representation based on an object of interest using a machine learning classifier. Figures 3 to 6As mentioned, the object of interest 310 can be used to synthesize a knowledge representation 330 that includes at least one concept and / or a relationship between two or more concepts. In some embodiments, the knowledge representation is encoded as a non-transitory computer-readable data structure.

[0059] like Fig. 9 As shown, the set of marked content items can be used as training data for machine learning algorithms. It is possible to use knowledge representation 330 as a feature source for a machine learning classification model. Examples of content item features based on the attributes obtained from the knowledge representation can include: the total number of concept intersections between the knowledge representation and the content items, the number of narrower concept intersections between the knowledge representation and the content items, or the weighted sum of the intersection concepts. Those skilled in the art will appreciate that many combinations and weighted sums of these attributes can also be used as features for a machine learning classification model. Further examples are described in more detail below.

[0060] Fig. 10A 1010 is a diagram of an exemplary workflow for training a classifier model using features from a knowledge representation obtained from an object of interest according to some embodiments of the present invention. The training data includes a first set of one or more tagged content items 1010. In some embodiments, each tag classifies the content item into one or more categories. In some embodiments, there may also be a validation data set including a second set of one or more tagged content items.

[0061] The machine learning classifier 1020 is trained based on one or more attributes obtained from the knowledge representation and features of the labeled content items 1010. Training the classification model 1020 may include inferring one or more functions using at least one feature of the content items in the training dataset based on the one or more attributes obtained from the knowledge representation and labels associated with the content items in the training dataset.

[0062] Fig. 10B According to some embodiments of the present invention, Fig. 10A 1050. The classification model 1020 is used to analyze a set of at least one unlabeled content item 1030, which is classified into a second set of at least one labeled content item 1050. In some embodiments, the classification uses at least one feature based on one or more attributes obtained from the knowledge representation and the content of the unlabeled content item.

[0063] Fig.11 is a diagram showing an example table of attributes associated with content items, known relevance of the content items, and predictions by a classifier according to some embodiments of the present invention. Fig.11 In the table, the training set of labeled content items includes content items 1 to 50. In this example, the labels classify the content items as being relevant to the object of interest or not relevant to the object of interest (in the table, "yes" is represented by Y and "no" is represented by N). In addition, there is a validation data set that includes a second set of labeled content items represented by content items 51 to 100. Content items 101 to n are unlabeled content items whose labels are unknown (and therefore, their relevance to the OOI is also unknown).

[0064] Fig.11 Further shown are the corresponding values ​​of the features of the training data and validation data items based on the first attribute and the second attribute (denoted as "attribute 1" and "attribute 2") of the knowledge representation. Fig.11 , but some embodiments may use features based on one attribute to train a classifier model. Some embodiments may also use two or more features based on an attribute to train a classifier model. It should be further noted that although Fig.11 The labels in the examples in are binary (ie, the only options are Y or N), but this is not necessarily the case, and the systems and methods described herein may be used to classify content items into more than two groups.

[0065] against Fig.11 The value of the attribute 1 and attribute 2 based features of each content item in the table in can be based on the intersection of these attributes with the content of the content item. The attribute is obtained from the synthesized knowledge representation and can include, for example, the total number of concepts that intersect between the knowledge representation and the content item, the number of broader concepts that intersect between the knowledge representation and the content item, the number of narrower concepts that intersect between the knowledge representation and the content item, and the sum of the weights of the concepts that intersect between the knowledge representation and the content item.

[0066] use Figure 4 The exemplary OOI 310 and Figure 6 The composite knowledge representation 330 is shown as an example, and it can be seen that the knowledge representation 300 includes concept hierarchies with different widths. For example, the "organ" concept is wider than the "brain" and "skin" concepts in the knowledge representation 330. Similarly, the "redness" concept is narrower than the "symptom" concept. Each of these concepts in the knowledge representation 330 can also have a weight associated with it.

[0067] Continuing with the example, Fig.11Attribute 1 in is the “number of broader concepts” and Attribute 2 is the “number of narrower concepts”. Content item 1 can be compared and / or intersected with knowledge representation 330, and it can be determined that content item 1 contains 7 instances of the word “organ” with a score of 7 for attribute 1 and 30 instances of the word “redness” with a score of 30 for attribute 2. Similarly, content item 2 may contain 5 instances of the word “brain” with a score of 15 for attribute 1 and 10 instances of the word “skin”. Content item 2 may also contain 15 instances of the word “cancer” with a score of 15 for attribute 2 and no instances of the word “redness”. Those skilled in the art will appreciate that although attributes 1 and 2 are characteristics of knowledge representation 300, Fig.11 The scores in the table of are based on features of the content item intersected with the attributes, the features being obtained from the knowledge representation 330 .

[0068] Since the "known relevance" labels for content items 1 and 2 (more generally, content items 1 to 50) are already known, a classifier model can be developed based on the detected relationship between the values ​​of feature-based attributes 1 and 2 and the generated labels. It should be appreciated that when the attributes derived from the synthesized knowledge representation 300 are used as features for training a machine learning classifier, fewer training data items may be required to develop a classifier model with a predefined level of accuracy compared to the use of so-called "standard features" of content items (e.g., title, length, author, etc.) for training a machine learning classifier.

[0069] Utilizing synthetic means and attributes generated as features can help avoid the problem of data sparsity, where the expression of the data in the OOI 310 or unlabeled content items is too limited to produce any useful classification using only "standard features". For example, a single tweet of the OOI 310 or the title of the unlabeled content item 1030 may not provide enough information for the classifier to accurately classify the content item (and in some cases, the textual expressions of the titles in multiple unlabeled content items 1030 may not be sufficiently distinguishable from each other). In contrast, by utilizing feature-based attributes from knowledge representation, a large amount of data is provided to the system, which can avoid the shortcomings of relying on features applied by traditional machine learning methods.

[0070] Once a machine learning classifier has been trained using the training data, it can then be trained using validation data items (e.g. Fig.11 As can be seen, for each validation data item, the value of the feature based on attribute 1 and attribute 2 is determined, and in Fig.11The rightmost column of shows the predictions made by the classifier based on the values ​​for the features based on attribute 1 and attribute 2. The labels assigned to content items 51 and 100 are correct, while the label predicted by the classifier for content item 99 is incorrect. In some embodiments, a minimum level of accuracy for the classifier model may be required. If the correct prediction rate for the classifier model is below an acceptable threshold, the system can use additional training data to further train the machine learning classifier. For example, in some embodiments, where a classifier based only on a training data set does not achieve the necessary level of accuracy, both the training data set and the validation data set can be used to train the machine learning classifier.

[0071] Once trained, the machine learning classifier can then be used to classify unlabeled content items. Fig.11 As shown, content items 101 and 102 contain a "?" in the "Known Relevance" column, and the machine learning classifier has predicted "N" and "Y" labels, respectively.

[0072] In some embodiments, more than one machine learning classifier may be trained. Fig.12 , a first classifier ("Classifier 1") may be trained using features of the training data content items based on Attribute 1 and Attribute 2. A second classifier ("Classifier 2") may be trained using features based on a different subset of the attributes. For example, the second classifier may be trained using only features based on Attribute 1. As can be seen, the labels predicted by the first and second classifiers for the same content item may be different, depending on the correlation between a particular attribute or set of attributes and the label associated therewith for the particular content item.

[0073] Fig.13 is a diagram showing an example table of prediction results made by multiple or all classifier models according to some embodiments of the present invention. As shown, predicted labels for unlabeled content items 101 to 105 are shown for first, second, and third classifiers trained using different sets or subsets of features based on different attributes. In some embodiments, an overall label can be generated based on a weighted sum of the predictions from the different classifiers. Fig.13 In the example shown, each classifier is binary (meaning a yes or no label is assigned), and each classifier is weighted equally. Accordingly, a 'yes' prediction may be assigned a value of 1, and a 'no' prediction may be assigned a value of 0. An average score may then be obtained for each content item for the overall label. Fig.13In the example shown, content items with an average score above 0.5 are assigned a 'yes' label, while content items with a score below 0.5 are assigned a 'no' label. In some embodiments, assigning labels to content items is similar to classifying content items into one of a plurality of possible categories (e.g., "related to OOI" or "not related to OOI").

[0074] It should be noted that the thresholds and weights assigned to a particular classifier can be selected to achieve desired results. Fig.13 The illustrated equal weighted system with binary classifiers is an example only. In some embodiments, the classifiers are not equally weighted, and the classifiers may optionally produce more than two possible labels (e.g., "recommended," "suggested," and "not recommended"). Another example classification is "duplicate" or "not duplicate," where a classifier may be trained to classify a content item as too similar to an Object of Interest to recommend (e.g., if the article contains the same text as the Object of Interest, but was published by a different publication source).

[0075] As mentioned above, in some embodiments, a knowledge representation is synthesized based on the content of an object of interest. In some embodiments, the synthesis includes generating at least one concept and / or a relationship between two or more concepts, wherein the concept and / or relationship is not listed in the object of interest. In some embodiments, the synthesized knowledge representation can apply generative rules to basic data structures to synthesize new complex knowledge representations. The knowledge representation can also include a weight associated with at least one concept. Moreover, the knowledge representation can take into account the characteristics of a specific user or a specific user group. For example, if it is known that a specific user is interested in tennis, this can be taken into account in the knowledge representation, for example, by assigning a greater weight to concepts related to tennis or by generating additional concepts related to tennis and adding them to the knowledge representation. Accordingly, a machine learning classifier trained using attributes from a knowledge representation customized for a particular user or user group can produce different classifications for the same content item, relative to a classifier trained using attributes from a knowledge representation that is not specific to a particular user or user group.

[0076] In some embodiments, according to the above Figure 7 The described process is used to generate training data that is used to train a machine learning classification model.

[0077] As mentioned above, the function inferred by the supervised learning algorithm can be based on observed or detected correlations between various features of the training data content items and the labels assigned to the training data content items. Some features of the content items may have little effect on the generated labels, while other features may have a very close correlation with the generated labels. For example, a feature based on an attribute related to the number of narrower concepts in the knowledge representation contained in the content item may have a stronger correlation with the label than a feature based on an attribute related to the number of broader concepts.

[0078] It is desirable to identify features of training data content items and / or attributes of knowledge representations that tend to have a higher correlation with the labels assigned to the training data objects. Training a machine learning classifier with a greater emphasis on such features or attributes can result in a higher overall accuracy for the classifier, or potentially result in a reduction in the number of training data items required to train the classifier to an acceptable accuracy range. Moreover, the identification of such features and / or attributes can also reduce the number of attributes or features to be evaluated for each training data item. This, in turn, can reduce the computational complexity of the machine learning algorithm and the processing costs associated therewith.

[0079] Fig.14A is a diagram of an example table showing the accuracy of labels predicted by a classifier according to some embodiments of the present invention. As can be seen, the machine learning classifier correctly classified verification content items 1, 2, 4, and 6, while verification content items 3, 5, 7, and 8 were incorrectly classified. Therefore, the classifier has an accuracy rate of 50% for these 8 verification content items.

[0080] Fig. 14B yes Fig.14A A graphical illustration of the accuracy of the results shown. As can be seen, content items 7 and 8 are relevant, but are "missed" by the machine learning classifier and fall outside the illustrated circle. Content items 1, 2, and 4 are correctly classified. Content items 3 and 5 are irrelevant and are incorrectly classified as relevant. Content item 6 is irrelevant and is correctly classified.

[0081] One metric used to evaluate the accuracy of a classifier is the concept of precision, which is defined as the ratio of the number of correctly labeled relevant items divided by the total number of items labeled as relevant by the classifier. Fig. 14B In the example of , the accuracy of the classifier would be 3 / 5 (because content items 1, 2, and 4 are recommended correctly, while content items 3 and 5 are recommended incorrectly).

[0082] Another metric used to evaluate the accuracy of a classifier is the concept of recall, which is defined as the ratio of the number of correctly labeled items divided by the total number of items. Fig. 14BIn the example of , the recall of the classifier would be 3 / 5 (because content items 1, 2, and 4 are recommended, while related items 7 and 8 are not recommended).

[0083] Precision and recall are often competing goals for classifiers, because recall focuses on identifying every relevant term, even if irrelevant terms are also included, while precision focuses on identifying only relevant terms while excluding irrelevant terms. In other words, recall may be better than precision when avoiding false negatives is better than avoiding false positives, and precision may be better than recall when avoiding false positives is better than avoiding false negatives.

[0084] In some embodiments, the comparison of the predictions made by the machine learning classifier can be compared with the labels assigned to each verification content data item. The knowledge representation can then be modified based on the comparison.

[0085] For example, in Fig.14A It is observed that higher scores of features based on attribute 2 are associated with the “Y” label, and the values ​​of features based on attribute 1 are relatively less relevant. Given this trend, the knowledge representation for OOI can be modified to reflect this correlation. For example, if Fig.14A In the example of , the feature based on attribute 2 is the number of narrower concepts in the knowledge representation that intersect each verification data item, then the knowledge representation can be modified (e.g., Figure 6 ) to include additional narrower concepts. For example, the concept of "sunburn" can be added to the "condition" level, and the concept of "itch" can be added to the "symptom" level. As another example, the weights associated with certain concepts (i.e., narrower concepts) in the knowledge representation can be increased, and / or the weights associated with certain concepts (i.e., broader concepts) can be decreased.

[0086] The modified knowledge representation may then be used to generate values ​​for features of the content item based on each attribute. Fig.15A reflects the updated value of each feature based on the attributes in the knowledge representation used to verify content items 1 to 8. As can be seen, relative to Fig.14A The values ​​in , based on the values ​​of the features of attribute 2 (which in this example corresponds to the number of narrower concepts) are exaggerated. Fig.15A The value of Property 1 is not changed in , but this may not necessarily be the case and the values ​​illustrated herein are merely examples.

[0087] Using the modified knowledge representation to use the values ​​of the features based on attribute 1 and attribute 2, the machine learning classifier can then re-predict the label for verifying the content item. As mentioned above, the attributes can include at least one of the total number of concepts that cross between the knowledge representation and one or more labeled content items, the number of broader concepts that cross between the knowledge representation and one or more labeled concept items, the sum of the weights of the concepts that cross between the knowledge representation and one or more labeled content items, and / or the number of narrower concepts that cross between the knowledge representation and one or more labeled content items. As can be seen from Fig.15A It is seen that the relevance predicted for certain content items using the modified knowledge representation is different.

[0088] Fig. 15B A graphical illustration of the results of a classifier using a knowledge representation modified using a machine learning classifier is provided. As can be seen, content items 1, 2, 4, and 7 are now correctly recommended, while content item 8 is still "missed" by the classifier. Content item 3 is still incorrectly recommended, while content item 5 is now correctly marked as irrelevant. Content item 6 is still correctly marked as irrelevant. Accordingly, the precision of the classifier using the modified knowledge representation in this example is now 4 / 5 (four content items are correctly recommended, and five content items are recommended in total). The recall of the classifier using the modified knowledge representation is now 4 / 5 (four content items are correctly recommended, and one relevant content item is omitted). Therefore, in this example, both the precision and recall of the machine learning classifier are improved. It should be noted that some embodiments can only improve one of the precision or recall by modifying the knowledge representation.

[0089] In some embodiments, the knowledge representation may be iteratively modified until a target precision or recall threshold has been achieved. For example, if the target precision is 9 / 10 (or 90%), then Fig.15A and Fig. 15B The diagram shows 4 / 5 (or 80%) accuracy, Fig.15A and Fig. 15B The results in will require further modification of the knowledge representation in an attempt to mark content item 8 as relevant or content item 3 as irrelevant.

[0090] Fig.16is an illustration of an exemplary flow chart for modifying a knowledge representation based on the results of a classifier according to some embodiments of the present invention. At 1610, a machine learning classifier predicts the relevance of a set of labeled content items (e.g., verification content items). At 1620, the predictions by the machine learning classifier are compared to the known labels of the verification data items (corresponding to the relevance in this example). At 1630, it is determined whether the machine learning classifier has achieved a goal. In some embodiments, the goal is a precision goal. In some embodiments, the goal is a recall goal. In some embodiments, the goal is a combination of precision and recall goals.

[0091] If the goal is not achieved, then at 1640, the knowledge representation is modified. At 1650, the predicted label for the verification content item is removed. The process then returns to 1610, where the relevance of the verification content item is predicted.

[0092] If at 1630, the target outcome has been achieved, then the process ends and the machine learning classifier and knowledge representation can be used to predict the relevance of unlabeled content items.

[0093] Accordingly, some embodiments of the methods, systems, and computer-readable media described herein can facilitate one or more of: generating training data for training a machine learning classifier for identifying content related to an object of interest, training a machine learning classifier based on properties of the object of interest using features from a synthetic knowledge representation, and / or for modifying a synthetic knowledge representation based on observed precision and / or recall values ​​associated with the trained machine learning classifier.

[0094] The present disclosure also contemplates applying the aforementioned feedback loop to modify the knowledge representation so that new relationships and / or concepts can be formed within the knowledge representation. For example, if the presence of a particular concept or relationship in the knowledge representation results in improved performance of a machine learning classifier, the method can identify the improved performance and modify the knowledge representation to include the concept or relationship. By considering content segments with or without a particular concept or relationship, patterns in the content can be identified to influence selections of the knowledge representation.

[0095] Machine learning classifiers of the type described herein can also be used to determine whether two independent concepts already contained within a knowledge representation can be combined to form a new concept (or alternatively, can be connected by some type of relationship), for example, by identifying that two independent and disconnected concepts (e.g., "fire" and "water pipe") should be connected by a relationship (e.g., "fire" is related to "water pipe") or alternatively combined to form a new concept (e.g., "fire belt").

[0096] It should be noted that various inventive concepts may be embodied as one or more methods, a number of examples of which are provided herein. The actions performed as part of the method may be ordered in any suitable manner. Accordingly, embodiments may be constructed that perform the actions in an order different from that illustrated, which may include performing some actions simultaneously, even though shown as sequential actions in illustrative embodiments, and vice versa.

[0097] The above-described embodiments of the present invention may be implemented in any of a variety of ways. For example, hardware, software, or a combination thereof may be used to implement these embodiments. When implemented in software, the software code may be embodied as stored program instructions that may be executed on any suitable processor or set of processors (e.g., one or more microprocessors), whether provided in a single computer or distributed among multiple computers.

[0098] It should be understood that the computer may be embodied in any of a variety of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, the computer may be embodied in a device that is not generally considered a computer but has suitable processing capabilities, including a personal digital assistant (PDA), a smart phone, a tablet computer, a reader, or any other suitable portable or fixed electronic device.

[0099] Moreover, a computer may have one or more input and output devices. Among other purposes, these devices may be used to present a user interface. Examples of output devices that may be used to provide a user interface include a printer or display screen for visual presentation of the output and a loudspeaker or other sound generating device for audible presentation of the output. Examples of input devices that may be used for a user interface include a keyboard, a microphone, and an indicating device, such as a mouse, a touch pad, and a digitized tablet computer.

[0100] Such computers may be interconnected in any suitable form via one or more networks, including networks such as local area networks (LANs) or wide area networks (WANs), such as enterprise networks, intelligent networks (INs), or the Internet. Such networks may be based on any suitable technology and may operate according to any suitable protocol, and may include wireless networks, wired networks, and / or fiber optic networks.

[0101] The various methods or processes outlined herein may be encoded as software executable on one or more processors that employ any of a variety of operating systems or platforms. Additionally, such software may be written using any of a variety of suitable programming languages ​​and / or programming or scripting tools, and may also be compiled into executable machine language code or intermediate code that is executed on a virtual machine or a suitable framework.

[0102] In this regard, various inventive concepts may be embodied as at least one non-transitory tangible computer-readable storage medium (e.g., computer memory, one or more floppy disks, compressed disks, optical disks, tapes, flash memory, circuit configurations in field programmable gate arrays or other semiconductor devices, etc.) article of manufacture encoded with one or more programs that, when executed on one or more computers or other processors, implement various process embodiments of the present invention. One or more non-transitory computer-readable media may be portable, such that one or more programs stored thereon may be loaded onto any suitable computer resource to implement various aspects of the present invention discussed above.

[0103] In a general sense, the term "program" or "software" is used herein to refer to any type of computer code or computer executable instruction set that can be used to program a computer or other processor to implement the various aspects of the embodiments discussed above. Additionally, it should be understood that, according to one aspect, one or more computer programs that perform the methods of the present invention when executed do not need to reside on a single computer or processor, but can be distributed in a modular manner between different computers or processors to implement various aspects of the present invention.

[0104] Computer executable instructions can be in many forms, such as program modules, that are executed by one or more computers or other devices. Generally, program modules include routines, programs, projects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Typically, the functionality of program modules can be combined or distributed in various embodiments as needed.

[0105] Moreover, the data structure can be stored in a non-transient tangible computer-readable storage medium article in any suitable form. To simplify the description, the data structure can be shown as having fields that are related by position in the data structure. Similarly, this relationship can be achieved by allocating storage for fields that have positions in a non-transient tangible computer-readable medium that conveys the relationship between the fields. However, any suitable mechanism can be used to establish a relationship between the information in the fields of the data structure, including by using pointers, tags, or other mechanisms that establish a relationship between data elements.

[0106] All definitions, used and defined herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0107] Unless expressly indicated to the contrary, the indefinite articles "a" and "an" as used herein should be understood to mean "at least one".

[0108] As used herein, with respect to a list of one or more elements, the phrase "at least one" should be understood to mean at least one element selected from any one or more elements in the list of elements, but does not necessarily include at least one of each element specifically listed in the list of elements, and does not exclude any combination of elements in the list of elements. This definition also allows that in addition to the elements specifically identified in the list of elements referred to by the phrase "at least one", other elements may optionally be present, whether related or unrelated to the specifically identified elements. Thus, as a non-limiting example, in one embodiment, "at least one of A and B" (or equivalently "at least one of A or B" or equivalently "at least one of A and / or B") may mean at least one A, optionally including more than one A, without B (and optionally including elements other than B); in another embodiment, means at least one B, optionally including more than one B, without A (and optionally including elements other than A); in yet another embodiment, means at least one A (optionally including more than one A) and at least one B (optionally including more than one B) (and optionally including other elements); and so on.

[0109] The term "and / or" as used herein should be understood to mean "one or both" of the elements so combined, i.e., elements that exist in combination in some cases and separately in other cases. Multiple elements listed with "and / or" should be interpreted in the same manner, i.e., "one or more" of the elements so combined. In addition to the elements specifically identified by the "and / or" clause, other elements may optionally be present, whether related or unrelated to these specifically identified elements. Therefore, as a non-limiting example, when used in conjunction with open-ended language such as "including", in one embodiment, a reference to "A and / or B" may refer only to A (optionally including elements other than B); in another embodiment, only to B (optionally including elements other than A); in yet another embodiment, to A and B (optionally including other elements); and so on.

[0110] As used herein, "or" should be understood to have the same meaning as "and / or" defined above. For example, when separating items in a list, "or" or "and / or" should be interpreted as inclusive, i.e., including at least one of a plurality of elements or a list of elements, but also including more than one, and optionally including additional unlisted items.

[0111] The phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "including," "involving," and variations thereof are meant to encompass the items listed thereafter and additional items.

[0112] Having described in detail a number of embodiments of the present invention, it will be readily apparent to those skilled in the art that various modifications and improvements will be within the spirit and scope of the present invention. Accordingly, the foregoing description is intended to be exemplary and non-restrictive.

Claims

1. A method for generating training data for a machine learning classifier, the method comprising: synthesizing, based on the content of the object of interest, a knowledge representation encoded as a non-transitory computer-readable data structure, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving a first set of content items, wherein the first set includes one or more content items without tags, wherein the tags categorize the content items into one or more categories; deriving a feature from at least one attribute in the synthesized knowledge representation for labeling the first set of content items; as well as generating the training data for the machine learning classifier by assigning labels to one or more corresponding content items in the first set based on the obtained features and the content of the corresponding content items, The features include: a number of broader concepts that intersect between the knowledge representation and the one or more content items, wherein a broader concept is a concept at a higher level in a concept hierarchy included in the knowledge representation, a number of narrower concepts that intersect between the knowledge representation and the one or more content items, wherein narrower concepts are concepts at a lower level in a concept hierarchy included in the knowledge representation, and The sum of weights of concepts that intersect between the knowledge representation and the one or more content items. 2 . The method according to claim 1 , wherein the synthesizing further comprises generating the at least one concept and / or the relationship between two or more concepts, wherein the concepts and / or the relationships are not listed in the object of interest.

3. The method of claim 1, wherein the object of interest comprises a topic, a tweet, a web page, a website, a document, a document collection, a document title, a message, an advertisement, and / or a search query.

4. The method according to claim 1, further comprising: A classifier is trained using the first set of labeled content items to predict labels for one or more unclassified content items.

5. The method according to claim 4, further comprising: receiving a second set of content items, wherein the second set includes one or more content items without tags; as well as A label is assigned by the classifier to one or more of the content items in the second set.

6. The method of claim 1, wherein the label is one of "recommended" or "not recommended".

7. The method according to claim 1, further comprising: One or more scores are determined for one or more respective content items in the first set, wherein the scores for respective content items are based on the knowledge representation and content of the respective content items.

8. The method of claim 7, wherein the score for the corresponding content item is based on an intersection of the at least one concept in the knowledge representation and the content of the corresponding content item.

9. The method of claim 7, wherein assigning the tag to the one or more corresponding content items in the first set comprises assigning the tag based on the score for the corresponding content item in the first set exceeding a predetermined threshold.

10. A system for generating training data for a machine learning classifier, the system comprising: At least one processor is configured to execute a method, the method comprising: synthesizing, based on the content of the object of interest, a knowledge representation encoded as a non-transitory computer-readable data structure, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving a first set of content items, wherein the first set includes one or more content items without tags, wherein the tags categorize the content items into one or more categories; deriving a feature from at least one attribute in the synthesized knowledge representation for use in tagging the first set of content items; and generating the training data for the machine learning classifier by assigning labels to one or more corresponding content items in the first set based on the obtained features and the content of the corresponding content items, The features include: a number of broader concepts that intersect between the knowledge representation and the one or more content items, wherein a broader concept is a concept at a higher level in a concept hierarchy included in the knowledge representation, a number of narrower concepts that intersect between the knowledge representation and the one or more content items, wherein narrower concepts are concepts at a lower level in a concept hierarchy included in the knowledge representation, and The sum of weights of concepts that intersect between the knowledge representation and the one or more content items. 11 . The system of claim 10 , wherein the synthesizing further comprises generating the at least one concept and / or the relationship between two or more concepts, wherein the concepts and / or the relationships are not listed in the object of interest.

12. The system of claim 10, wherein the object of interest comprises a topic, a tweet, a web page, a website, a document, a collection of documents, a document title, a message, an advertisement, and / or a search query.

13. The system of claim 10, wherein the method further comprises: A classifier is trained using the first set of labeled content items to predict labels for one or more unclassified content items.

14. The system of claim 13, wherein the method further comprises: receiving a second set of content items, wherein the second set includes one or more content items without tags; as well as A label is assigned by the classifier to one or more of the content items in the second set.

15. The system of claim 10, wherein the label is one of "recommended" or "not recommended".

16. The system of claim 10, further comprising: One or more scores are determined for one or more respective content items in the first set, wherein the scores for respective content items are based on the knowledge representation and content of the respective content items.

17. The system of claim 16, wherein the score for the corresponding content item is based on an intersection of the at least one concept in the knowledge representation and the content of the corresponding content item.

18. The system of claim 16, wherein assigning the tag to the one or more corresponding content items in the first set comprises assigning the tag based on the score for the corresponding content item in the first set exceeding a predetermined threshold.

19. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method of generating training data for a machine learning classifier, the method comprising: synthesizing, based on the content of the object of interest, a knowledge representation encoded as a non-transitory computer-readable data structure, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving a first set of content items, wherein the first set includes one or more content items without tags, wherein the tags categorize the content items into one or more categories; deriving a feature from at least one attribute in the synthesized knowledge representation for labeling the first set of content items; as well as generating the training data for the machine learning classifier by assigning labels to one or more corresponding content items in the first set based on the obtained features and the content of the corresponding content items, The features include: a number of broader concepts that intersect between the knowledge representation and the one or more content items, wherein a broader concept is a concept at a higher level in a concept hierarchy included in the knowledge representation, a number of narrower concepts that intersect between the knowledge representation and the one or more content items, wherein narrower concepts are concepts at a lower level in a concept hierarchy included in the knowledge representation, and The sum of weights of concepts that intersect between the knowledge representation and the one or more content items.

20. The at least one non-transitory computer-readable storage medium of claim 19, wherein the synthesizing further comprises generating the at least one concept and / or the relationship between two or more concepts, wherein the concepts and / or the relationships are not listed in the object of interest.

21. The at least one non-transitory computer-readable storage medium of claim 19, wherein the object of interest comprises a topic, a tweet, a web page, a website, a document, a collection of documents, a document title, a message, an advertisement, and / or a search query.

22. The at least one non-transitory computer-readable storage medium of claim 19, wherein the method further comprises: A classifier is trained using the first set of labeled content items to predict labels for one or more unclassified content items.

23. The at least one non-transitory computer-readable storage medium of claim 22, wherein the method further comprises: receiving a second set of content items, wherein the second set includes one or more content items without tags; as well as A label is assigned by the classifier to one or more of the content items in the second set.

24. The at least one non-transitory computer-readable storage medium of claim 19, wherein the label is one of "recommended" or "not recommended." 25. The at least one non-transitory computer-readable storage medium of claim 19, wherein the method further comprises: One or more scores are determined for one or more respective content items in the first set, wherein the scores for respective content items are based on the knowledge representation and content of the respective content items.

26. The at least one non-transitory computer-readable storage medium of claim 25, wherein the score for the respective content item is based on an intersection of the at least one concept in the knowledge representation with the content of the respective content item.

27. The at least one non-transitory computer-readable storage medium of claim 25, wherein assigning the tag to the one or more corresponding content items in the first set comprises assigning the tag based on the score for the corresponding content item in the first set exceeding a predetermined threshold.

28. A method for classifying at least one unlabeled content item using a machine learning classifier, the method comprising: receiving, based on the object of interest, a knowledge representation encoded as a non-transitory computer-readable data structure, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving training data, the training data comprising a first set of one or more tagged content items having a label that categorizes each content item into one or more categories, wherein the first set of one or more tagged content items is generated by assigning one or more labels to one or more respective content items in the first set based on features and content of the respective content items, and the features are derived from at least one attribute of the knowledge representation; training the machine learning classifier using at least one feature of the tagged content item based on one or more attributes obtained from the knowledge representation; as well as classifying, using the machine learning classifier, the at least one feature of the unlabeled content item into the one or more categories based on the one or more attributes obtained from the knowledge representation, The at least one characteristic of the marked content item comprises: a number of broader concepts that intersect between the knowledge representation and the at least one unlabeled content item, wherein a broader concept is a concept at a higher level in a concept hierarchy included in the knowledge representation, a number of narrower concepts that intersect between the knowledge representation and the at least one unlabeled content item, wherein narrower concepts are concepts at a lower level in a concept hierarchy included in the knowledge representation, and A sum of weights of concepts crossed between the knowledge representation and the at least one unlabeled content item.

29. The method of claim 28, further comprising synthesizing the knowledge representation based on content of the object of interest.

30. The method of claim 29, wherein the synthesizing further comprises generating the at least one concept and / or the relationship between two or more concepts, wherein the concepts and / or the relationships are not listed in the object of interest.

31. The method of claim 28, wherein the knowledge representation includes a weight associated with the at least one concept.

32. The method of claim 28, wherein the at least one feature of the one unlabeled content item is based on an intersection of the one or more attributes obtained from the knowledge representation.

33. The method of claim 28, wherein the object of interest comprises a topic, a tweet, a web page, a website, a document, a collection of documents, a document title, a message, an advertisement, and / or a search query.

34. The method of claim 28, further comprising: receiving verification data, the verification data comprising a second set of one or more marked content items; The verification data is classified into the one or more categories using, with the machine learning classifier, at least one feature of the second set of one or more tagged content items based on the one or more attributes derived from the knowledge representation.

35. The method of claim 34, further comprising: comparing the classified category for each tagged content item in the second set of one or more tagged content items to one or more tags for each tagged content item in the second set of one or more tagged content items; determining a number of correct classifications and a number of incorrect classifications made by the machine learning classifier; as well as The machine learning classifier is retrained based on the number of correct classifications and the number of incorrect classifications.

36. The method of claim 28, wherein the knowledge representation is user-specific.

37. The method of claim 28, wherein the knowledge representation is group-specific.

38. The method of claim 28, wherein the label comprises one of "recommended" and "not recommended".

39. The method of claim 28, wherein the label comprises one of "repeat" and "non-repeat".

40. The method of claim 28, wherein training the machine learning classifier comprises training at least two machine learning classifiers, wherein a first classifier of the at least two machine learning classifiers is trained using a first feature of the tagged content item based on the one or more attributes obtained from the knowledge representation, and wherein a second classifier of the at least two machine learning classifiers is trained using a second feature of the tagged content item based on the one or more attributes obtained from the knowledge representation, wherein the first feature and the second feature are based on at least one different attribute.

41. The method of claim 40, wherein the classified category for the at least one unlabeled content item is determined using the at least two machine learning classifiers.

42. The method of claim 41 , further comprising associating a weight with each of the at least two machine learning classifiers based on a respective number of correct classifications and incorrect classifications by each of the at least two machine learning classifiers.

43. A method according to claim 28, wherein the training data is generated based on scores for one or more unlabeled content items, wherein the scores for the corresponding unlabeled content items are based on the one or more attributes obtained from the knowledge representation and the content of the corresponding content items.

44. A system for classifying at least one unlabeled content item using a machine learning classifier, the system comprising: At least one processor is configured to execute a method, the method comprising: receiving, based on the object of interest, a knowledge representation encoded as a non-transitory computer-readable data structure, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving training data, the training data comprising a first set of one or more tagged content items having labels, the labels classifying each content item into one or more categories; training the machine learning classifier using at least one feature of the tagged content item based on one or more attributes derived from the knowledge representation; and classifying, with the machine learning classifier, the at least one unlabeled content item into the one or more categories using at least one feature of the unlabeled content item based on the one or more attributes derived from the knowledge representation, wherein the first set of one or more labeled content items is generated by assigning one or more tags to one or more respective content items in the first set based on the features and content of the respective content items, and the features are derived from the at least one attribute of the knowledge representation, The at least one characteristic of the marked content item comprises: a number of broader concepts that intersect between the knowledge representation and the at least one unlabeled content item, wherein a broader concept is a concept at a higher level in a concept hierarchy included in the knowledge representation, a number of narrower concepts that intersect between the knowledge representation and the at least one unlabeled content item, wherein narrower concepts are concepts at a lower level in a concept hierarchy included in the knowledge representation, and A sum of weights of concepts crossed between the knowledge representation and the at least one unlabeled content item.

45. The system of claim 44, wherein the method further comprises synthesizing the knowledge representation based on content of the object of interest.

46. ​​The system of claim 45, wherein the synthesizing further comprises generating the at least one concept and / or the relationship between two or more concepts, wherein the concepts and / or the relationships are not listed in the object of interest.

47. The system of claim 44, wherein the knowledge representation includes a weight associated with the at least one concept.

48. The system of claim 44, wherein the at least one feature of the one unlabeled content item is based on an intersection of the one or more attributes obtained from the knowledge representation.

49. The system of claim 44, wherein the objects of interest include topics, tweets, web pages, websites, documents, document collections, document titles, messages, advertisements, and / or search queries.

50. The system of claim 44, wherein the method further comprises: receiving verification data, the verification data comprising a second set of one or more marked content items; as well as The verification data is classified into the one or more categories using, with the machine learning classifier, at least one feature of the second set of one or more tagged content items based on the one or more attributes derived from the knowledge representation.

51. The system of claim 50, wherein the method further comprises: comparing the classified category for each tagged content item in the second set of one or more tagged content items to one or more tags for each tagged content item in the second set of one or more tagged content items; determining a number of correct classifications and a number of incorrect classifications made by the machine learning classifier; as well as The machine learning classifier is retrained based on the number of correct classifications and the number of incorrect classifications.

52. The system of claim 44, wherein the knowledge representation is user-specific.

53. The system of claim 44, wherein the knowledge representation is group-specific.

54. A system according to claim 44, wherein the label includes one of "recommended" and "not recommended".

55. A system according to claim 44, wherein the label includes one of "repeat" and "non-repeat".

56. A system according to claim 44, wherein training the machine learning classifier comprises training at least two machine learning classifiers, wherein a first classifier of the at least two machine learning classifiers is trained using a first feature of the tagged content item based on the one or more attributes obtained from the knowledge representation, and wherein a second classifier of the at least two machine learning classifiers is trained using a second feature of the tagged content item based on the one or more attributes obtained from the knowledge representation, wherein the first feature and the second feature are based on at least one different attribute.

57. The system of claim 56, wherein the classified category for the at least one unlabeled content item is determined using the at least two machine learning classifiers.

58. The system of claim 57, wherein the method further comprises associating a weight with each of the at least two machine learning classifiers based on a respective number of correct classifications and incorrect classifications by each of the at least two machine learning classifiers.

59. A system according to claim 44, wherein the training data is generated based on scores for one or more unlabeled content items, wherein the scores for the corresponding unlabeled content items are based on the one or more attributes obtained from the knowledge representation and the content of the corresponding content items.

60. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method of classifying at least one unlabeled content item using a machine learning classifier, the method comprising: receiving, based on the object of interest, a knowledge representation encoded as a non-transitory computer-readable data structure, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving training data, the training data comprising a first set of one or more tagged content items having labels, the labels classifying each content item into one or more categories; training the machine learning classifier using at least one feature of the tagged content item based on one or more attributes obtained from the knowledge representation; as well as classifying, with the machine learning classifier, the at least one unlabeled content item into the one or more categories using at least one feature of the unlabeled content item based on the one or more attributes derived from the knowledge representation, wherein the first set of one or more labeled content items is generated by assigning one or more tags to one or more respective content items in the first set based on the features and content of the respective content items, and the features are derived from the at least one attribute of the knowledge representation, The at least one characteristic of the marked content item comprises: a number of broader concepts that intersect between the knowledge representation and the at least one unlabeled content item, wherein a broader concept is a concept at a higher level in a concept hierarchy included in the knowledge representation, a number of narrower concepts that intersect between the knowledge representation and the at least one unlabeled content item, wherein narrower concepts are concepts at a lower level in a concept hierarchy included in the knowledge representation, and A sum of weights of concepts crossed between the knowledge representation and the at least one unlabeled content item.

61. The at least one non-transitory computer-readable storage medium of claim 60, wherein the method further comprises synthesizing the knowledge representation based on content of the object of interest.

62. At least one non-transitory computer-readable storage medium according to claim 61, wherein the synthesis further includes generating the at least one concept and / or the relationship between two or more concepts, wherein the concepts and / or the relationships are not listed in the object of interest.

63. The at least one non-transitory computer-readable storage medium of claim 60, wherein the knowledge representation comprises a weight associated with the at least one concept.

64. The at least one non-transitory computer-readable storage medium of claim 60, wherein the at least one feature of the unlabeled content item is based on an intersection of the one or more attributes obtained from the knowledge representation.

65. The at least one non-transitory computer-readable storage medium of claim 60, wherein the object of interest comprises a topic, a tweet, a web page, a website, a document, a collection of documents, a document title, a message, an advertisement, and / or a search query.

66. The at least one non-transitory computer-readable storage medium of claim 60, wherein the method further comprises: receiving verification data, the verification data comprising a second set of one or more marked content items; as well as The verification data is classified into the one or more categories using, with the machine learning classifier, at least one feature of the second set of one or more tagged content items based on the one or more attributes derived from the knowledge representation.

67. The at least one non-transitory computer-readable storage medium of claim 66, wherein the method further comprises: comparing the classified category for each tagged content item in the second set of one or more tagged content items to one or more tags for each tagged content item in the second set of one or more tagged content items; determining a number of correct classifications and a number of incorrect classifications made by the machine learning classifier; as well as The machine learning classifier is retrained based on the number of correct classifications and the number of incorrect classifications.

68. The at least one non-transitory computer-readable storage medium of claim 60, wherein the knowledge representation is user-specific.

69. The at least one non-transitory computer-readable storage medium of claim 60, wherein the knowledge representation is group-specific.

70. The at least one non-transitory computer-readable storage medium of claim 60, wherein the label comprises one of "recommended" and "not recommended." 71. At least one non-transitory computer-readable storage medium according to claim 60, wherein the label comprises one of "repeat" and "non-repeat".

72. At least one non-transitory computer-readable storage medium according to claim 60, wherein training the machine learning classifier comprises training at least two machine learning classifiers, wherein a first classifier of the at least two machine learning classifiers is trained using a first feature of the tagged content item based on the one or more attributes obtained from the knowledge representation, and wherein a second classifier of the at least two machine learning classifiers is trained using a second feature of the tagged content item based on the one or more attributes obtained from the knowledge representation, wherein the first feature and the second feature are based on at least one different attribute.

73. The at least one non-transitory computer-readable storage medium of claim 72, wherein the classified category for the at least one unlabeled content item is determined using the at least two machine learning classifiers.

74. At least one non-transitory computer-readable storage medium according to claim 73, wherein the method further comprises associating a weight with each of the at least two machine learning classifiers based on a respective number of correct classifications and incorrect classifications by each of the at least two machine learning classifiers.

75. At least one non-transitory computer-readable storage medium according to claim 60, wherein the training data is generated based on scores for one or more unlabeled content items, wherein the scores for the corresponding unlabeled content items are based on the one or more attributes obtained from the knowledge representation and the content of the corresponding content items.

76. A method for modifying knowledge representation based on a machine learning classifier, the method comprising: receiving, based on the object of interest, a knowledge representation encoded as a non-transitory computer-readable data structure, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving validation data, the validation data comprising a first set of one or more tagged content items having a label, the label classifying each content item into one or more categories, the one or more categories comprising a first category known to be relevant to the object of interest and a second category known to be unrelated to the object of interest; predicting each of the one or more tagged content items as one of: a) related to the object of interest or b) not related to the object of interest using a machine learning classifier using at least one attribute derived from the knowledge representation as a feature, wherein the machine learning classifier is trained based on a set of content items having the feature, and the feature comprises: a number of broader concepts that intersect between the knowledge representation and the one or more tagged content items, wherein a broader concept is a concept at a higher level in a concept hierarchy included in the knowledge representation, a number of narrower concepts that intersect between the knowledge representation and the one or more tagged content items, wherein narrower concepts are concepts at a lower level in a concept hierarchy included in the knowledge representation, and a sum of weights of concepts that intersect between the knowledge representation and the one or more tagged content items; as well as The knowledge representation is modified based on a comparison of the prediction made by the machine learning classifier for each content item in the first set with the label for each corresponding content item, wherein modifying the knowledge representation comprises adding one or more additional concepts to the knowledge representation based on the comparison.

77. The method of claim 76, further comprising synthesizing the knowledge representation based on content of the object of interest.

78. The method of claim 77, wherein the synthesizing further comprises generating the at least one concept and / or the relationship between two or more concepts, wherein the concepts and / or the relationships are not listed in the object of interest.

79. The method of claim 76, wherein the knowledge representation includes a weight associated with the at least one concept.

80. The method of claim 76, wherein the prediction is based on an intersection of the one or more tagged content items with the features.

81. The method of claim 76, wherein the object of interest comprises a topic, a tweet, a web page, a website, a document, a collection of documents, a document title, a message, an advertisement, and / or a search query.

82. The method of claim 76, further comprising: After modifying the knowledge representation: re-predicting each tagged content item in the first set of the one or more tagged content items using the modified knowledge representation; as well as The knowledge representation is modified based on a comparison of the prediction made by the machine learning classifier for each content item in the first collection and the label for each respective content item.

83. A method according to claim 82, wherein the re-prediction and the modification are repeated until the ratio of the number of the one or more tagged content items that are correctly predicted as being related to the object of interest to the total number of tagged content items in the first category equals or exceeds an accuracy threshold.

84. A method according to claim 82, wherein the re-prediction and the modification are repeated until the ratio of the number of the one or more tagged content items that are correctly predicted to be related to the object of interest to the total number of the one or more tagged content items predicted to be related to the object of interest equals or exceeds a recall threshold.

85. The method of claim 76, wherein modifying the knowledge representation comprises modifying a weight associated with the at least one concept in the knowledge representation.

86. A method according to claim 76, wherein modifying the knowledge representation based on the comparison includes modifying the knowledge representation when the ratio of the number of the one or more tagged content items that are correctly predicted to be related to the object of interest to the total number of the one or more tagged content items in the first category is less than a threshold accuracy value.

87. A method according to claim 76, wherein modifying the knowledge representation based on the comparison includes modifying the knowledge representation when a ratio of the number of the one or more tagged content items that are correctly predicted to be related to the object of interest to the total number of the one or more tagged content items that are predicted to be related to the object of interest is less than a threshold recall value.

88. A system for modifying a knowledge representation based on a machine learning classifier, the system comprising: At least one processor is configured to execute a method, the method comprising: receiving, based on the object of interest, a knowledge representation encoded as a non-transitory computer-readable data structure, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving validation data, the validation data comprising a first set of one or more tagged content items having a label, the label classifying each content item into one or more categories, the one or more categories comprising a first category known to be relevant to the object of interest and a second category known to be unrelated to the object of interest; predicting each of the one or more tagged content items as one of: a) related to the object of interest or b) not related to the object of interest using a machine learning classifier using at least one attribute derived from the knowledge representation as a feature, wherein the machine learning classifier is trained based on a set of content items having the feature, and the feature comprises: a number of broader concepts that intersect between the knowledge representation and the one or more tagged content items, wherein a broader concept is a concept at a higher level in a concept hierarchy included in the knowledge representation, a number of narrower concepts that intersect between the knowledge representation and the one or more tagged content items, wherein narrower concepts are concepts at a lower level in a concept hierarchy included in the knowledge representation, and a sum of weights of concepts that intersect between the knowledge representation and the one or more tagged content items; and modifying the knowledge representation based on a comparison of the prediction made by the machine learning classifier for each content item in the first collection with the label for each respective content item, Wherein modifying the knowledge representation comprises adding one or more additional concepts to the knowledge representation based on the comparison.

89. The system of claim 88, wherein the method further comprises synthesizing the knowledge representation based on content of the object of interest.

90. The system of claim 89, wherein the synthesizing further comprises generating the at least one concept and / or the relationship between two or more concepts, wherein the concepts and / or the relationships are not listed in the object of interest.

91. The system of claim 88, wherein the knowledge representation includes a weight associated with the at least one concept.

92. The system of claim 88, wherein the prediction is based on an intersection of the one or more tagged content items with the feature.

93. The system of claim 88, wherein the objects of interest include topics, tweets, web pages, websites, documents, document collections, document titles, messages, advertisements, and / or search queries.

94. The system of claim 88, wherein the method further comprises: After modifying the knowledge representation: re-predicting each tagged content item in the first set of one or more tagged content items using the modified knowledge representation; as well as The knowledge representation is modified based on a comparison of the prediction made by the machine learning classifier for each content item in the first collection and the label for each respective content item.

95. A system according to claim 94, wherein the re-prediction and the modification are repeated until the ratio of the number of the one or more tagged content items that are correctly predicted as being related to the object of interest to the total number of tagged content items in the first category equals or exceeds an accuracy threshold.

96. A system according to claim 94, wherein the re-prediction and the modification are repeated until the ratio of the number of the one or more tagged content items that are correctly predicted as being related to the object of interest to the total number of the one or more tagged content items predicted as being related to the object of interest equals or exceeds a recall threshold.

97. The system of claim 88, wherein modifying the knowledge representation comprises modifying a weight associated with the at least one concept in the knowledge representation.

98. A system according to claim 88, wherein modifying the knowledge representation based on the comparison includes modifying the knowledge representation when a ratio of the number of the one or more tagged content items that are correctly predicted to be related to the object of interest to the total number of the one or more tagged content items in the first category is less than a threshold accuracy value.

99. A system according to claim 88, wherein modifying the knowledge representation based on the comparison includes modifying the knowledge representation when a ratio of the number of the one or more tagged content items that are correctly predicted to be related to the object of interest to the total number of the one or more tagged content items predicted to be related to the object of interest is less than a threshold recall value.

100. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method of modifying a knowledge representation based on a machine learning classifier, the method comprising: receiving, based on the object of interest, a knowledge representation encoded as a non-transitory computer-readable data structure, the knowledge representation comprising at least one concept and / or a relationship between two or more concepts; receiving validation data, the validation data comprising a first set of one or more tagged content items having a label, the label classifying each content item into one or more categories, the one or more categories comprising a first category known to be relevant to the object of interest and a second category known to be unrelated to the object of interest; predicting each of the one or more tagged content items as one of: a) related to the object of interest or b) not related to the object of interest using a machine learning classifier using at least one attribute derived from the knowledge representation as a feature, wherein the machine learning classifier is trained based on a set of content items having the feature, and the feature comprises: a number of broader concepts that intersect between the knowledge representation and the one or more tagged content items, wherein a broader concept is a concept at a higher level in a concept hierarchy included in the knowledge representation, a number of narrower concepts that intersect between the knowledge representation and the one or more tagged content items, wherein narrower concepts are concepts at a lower level in a concept hierarchy included in the knowledge representation, and a sum of weights of concepts that intersect between the knowledge representation and the one or more tagged content items; as well as The knowledge representation is modified based on a comparison of the prediction made by the machine learning classifier for each content item in the first set with the label for each corresponding content item, wherein modifying the knowledge representation comprises adding one or more additional concepts to the knowledge representation based on the comparison.

101. The at least one non-transitory computer-readable storage medium of claim 100, wherein the method further comprises synthesizing the knowledge representation based on content of the object of interest.

102. At least one non-transitory computer-readable storage medium according to claim 101, wherein the synthesis further includes generating the at least one concept and / or the relationship between two or more concepts, wherein the concepts and / or the relationships are not listed in the object of interest.

103. At least one non-transitory computer-readable storage medium according to claim 100, wherein the knowledge representation includes a weight associated with the at least one concept.

104. The at least one non-transitory computer-readable storage medium of claim 100, wherein the prediction is based on an intersection of the one or more tagged content items with the feature.

105. The at least one non-transitory computer-readable storage medium of claim 100, wherein the object of interest comprises a topic, a tweet, a web page, a website, a document, a collection of documents, a document title, a message, an advertisement, and / or a search query.

106. The at least one non-transitory computer-readable storage medium of claim 100, wherein the method further comprises: After modifying the knowledge representation: re-predicting each tagged content item in the first set of the one or more tagged content items using the modified knowledge representation; as well as The knowledge representation is modified based on a comparison of the prediction made by the machine learning classifier for each content item in the first collection and the label for each respective content item.

107. At least one non-transitory computer-readable storage medium according to claim 106, wherein the re-prediction and the modification are repeated until the ratio of the number of the one or more tagged content items that are correctly predicted as being related to the object of interest to the total number of tagged content items in the first category equals or exceeds an accuracy threshold.

108. At least one non-transitory computer-readable storage medium according to claim 106, wherein the re-prediction and the modification are repeated until the ratio of the number of the one or more tagged content items that are correctly predicted as being related to the object of interest to the total number of the one or more tagged content items that are predicted as being related to the object of interest equals or exceeds a recall threshold.

109. At least one non-transitory computer-readable storage medium according to claim 100, wherein modifying the knowledge representation includes modifying a weight associated with at least one concept in the knowledge representation and / or adding additional concepts to the knowledge representation.

110. At least one non-transitory computer-readable storage medium according to claim 100, wherein modifying the knowledge representation based on the comparison includes modifying the knowledge representation when a ratio of the number of the one or more tagged content items that are correctly predicted to be related to the object of interest to the total number of the one or more tagged content items in the first category is less than a threshold accuracy value.

111. At least one non-transitory computer-readable storage medium according to claim 100, wherein modifying the knowledge representation based on the comparison includes modifying the knowledge representation when a ratio of the number of the one or more tagged content items that are correctly predicted to be related to the object of interest to the total number of the one or more tagged content items that are predicted to be related to the object of interest is less than a threshold recall value.