Relationship classification of context-sensitive relationships between content items

Abstract

The invention relates to relationship classification of context-sensitive relationships between content items. The techniques involve receiving, by a context-sensitive classifier, at least one reference item and at least one candidate item, and providing, by the context-sensitive classifier, a prediction of whether the candidate item is context-compatible or context-incompatible with the reference item. The techniques may determine whether to present the at least one candidate in the user interface based on the prediction of whether the candidate is contextually compatible or contextually incompatible with the reference item.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 504,959, filed on May 30, 2023, entitled “RELATIONSHIP CLASSIFICATION FOR CONTEXT-SENSITIVE RELATIONSHIPS BETWEEN CONTENT ITEMS,” the entire contents of which are expressly incorporated herein by reference. Background Art

[0003] Content recommendation engines are becoming increasingly important in today's digital environment, with a vast amount of content available on various platforms. Users often rely on content recommendation engines to discover relevant and interesting content tailored to their preferences. Content providers utilize recommendation engines as a mechanism to advertise relevant content items.

[0004] Recommendation engines typically rely on machine learning algorithms to identify items that are similar to a reference item, items that belong to a category related to the reference item, or items related to the reference item that a user is likely to view. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In order to describe the manner in which the above and other advantages and features of the present disclosure can be obtained, a more particular description of the principles briefly described above will be presented by reference to specific embodiments shown in the accompanying drawings. It should be understood that these drawings only illustrate exemplary embodiments of the present disclosure and are therefore not to be considered as limiting the scope of the present disclosure, so the principles of this document will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0006] Figure 1 An exemplary system for identifying content items to be recommended along with reference items in accordance with some embodiments of the present technology is illustrated.

[0007] Figure 2 An exemplary routine for determining whether a candidate item and a reference item are contextually compatible in accordance with some embodiments of the present technology is illustrated.

[0008] Figure 3 An example of a context-sensitive classifier in accordance with some embodiments of the present technology is illustrated.

[0009] Figure 4 An exemplary routine for training a context-sensitive classifier in accordance with some aspects of the present technology is illustrated.

[0010] Figure 5 Exemplary sampling strategies for active learning in accordance with aspects of the present technology are illustrated.

[0011] Figure 6 An example of a deep learning neural network that can be used to implement a context-sensitive classifier in accordance with some aspects of the present technology is illustrated.

[0012] Figure 7 An exemplary life cycle of an ML model in accordance with some aspects of the present technology is illustrated.

[0013] Figure 8 An example of a system for implementing some aspects of the present technology is illustrated. DETAILED DESCRIPTION

[0014] Various embodiments of the present disclosure are discussed in detail below. Although specific embodiments are discussed, it should be understood that this is only for illustrative purposes. Those skilled in the relevant art will recognize that other components and configurations can be used without departing from the spirit and scope of the present disclosure.

[0015] Additional features and advantages of the present disclosure will be set forth in the description that follows, and in part will be apparent from the description, or may be learned by practicing the principles disclosed herein. The features and advantages of the present disclosure may be realized and obtained by means of the instruments and combinations specifically pointed out in the appended claims. These and other features of the present disclosure will become more fully apparent through the following description and the appended claims, or may be learned by practicing the principles set forth herein.

[0016] As used herein, the term “configured” should be considered interchangeably used to refer to configured and configurable, unless the term “configurable” is explicitly used to distinguish from “configured.” The correct understanding of the term in the context in which it is used will be apparent to one of ordinary skill in the art.

[0017] The disclosed technology solves the need in the art to identify pairs of content items that may be considered related. Although the content item pairs may be related, they may be associated with contexts that may affect whether the content items should be presented together. For example, content items associated with two incompatible contexts should be prevented from being presented together. In another example, content items that match the context should be encouraged to be presented together.

[0018] Recommendation systems are critical to helping users discover new content. However, systems for item-by-item recommendation tasks are often based on shallow contextual correlations, which can make the systems inadequate for tasks where item relationships are more subtle. Context-sensitive item pairs can sometimes have controversial or problematic relationships and can degrade user experience and brand perception when recommended to users. For example, a recommendation of a book about divorce and a book about co-parenting can create an unsettling experience for someone who is downloading or viewing a book about marital therapy.

[0019] Item-by-item recommendation systems can effectively deliver context-sensitive recommendations to users. Therefore, product features such as "Frequently Bought Together", "Because You Watched", and "You Might Also Like" appear in many online services such as books, music, apps, movies, etc. These features help users discover relevant and interesting content based on similar or complementary content they have already engaged with. However, item-by-item associations are sometimes inappropriate from the perspective of some users or developers. For example, a user downloading a book about miscarriage may not like recommendations for books about fetal development during pregnancy, while a user purchasing a sobriety tracking workbook may not like recommendations for books about cocktail recipes. Similar examples can be found in other online services such as podcasts, apps, music, or movies.

[0020] The present technology relates to a system for predicting whether a pair of content items exhibits a type of contextual relationship. For example, the present technology can be used to determine whether content items have a sensitive contextual relationship. Pairs of content items with sensitive contextual relationships may be considered offensive, insensitive, objectionable, or problematic when presented together. Recommending items that have sensitive contextual relationships with reference items is rooted in a lack of deep understanding of the relationship between the items, meaning that they may only consider shallow features such as item similarity and user co-engagement without understanding the context of the items. Such recommendations may be insensitive and offensive to users.

[0021] Examples of sensitive contextual relationships may include incompatible audience age ranges (e.g., content in teen and adult categories), conflicting audiences (e.g., content about hiking and content about living in a wheelchair), sensitive categories, dangerous categories, opposing purposes (e.g., content about relationship advice and divorce, etc.), unwanted temptations, insensitive combinations (e.g., content about relationship advice and divorce, etc.), insensitive event locations (e.g., an event occurring in one location while the user is in another location, etc.).

[0022] While this description primarily provides examples of content items associated with sensitive contextual relationships, this is only because such pairs of content items provide clear examples of the present technology, but these examples should not be considered limiting. For example, a recommendation about leg exercises should not be provided with a reference to a physical fit for sitting in a wheelchair, or a recommendation about a cocktail recipe should not be provided with a reference to quitting drinking. These are obvious examples of content items that may be related but not compatible. But other contextual relationships exist as well. Some content items may not appear to be related, but they are contextually supportive of each other. For example, a content item about interacting with an embassy may not be strongly related to a content item about international destinations, but their context may be supportive and therefore encouraged to be presented together.

[0023] While Applicants contemplate practicing the present technology in a privacy-preserving manner (indeed, Applicants prefer to have access to as little user data as possible), it should be recognized that some aspects of the present technology may be used in conjunction with the practice of collecting and using data available from a variety of sources to improve the delivery of invitation content or any other content to users that may be of interest to them. The present disclosure contemplates that, in some instances, such collected data may include personal information data that uniquely identifies or can be used to contact or locate a particular person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, twitter IDs, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0024] The present disclosure recognizes that the use of such personal information data in the present invention technology can be used to benefit users. For example, the personal information data can be used to deliver targeted content that users are more interested in. Therefore, the use of such personal information data enables users to have planned control over the content delivered. In addition, the present disclosure also anticipates other uses of personal information data that benefit users. For example, health and fitness data can be used to provide insights into the user's overall health status, or can be used as positive feedback for individuals who use technology to pursue health goals.

[0025] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be performed after receiving informed consent from the user. In addition, such entities should consider taking any necessary steps to safeguard and protect access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0026] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, with respect to an advertising delivery service, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" at any time during or after registration for the service to participate in the collection of personal information data. In another example, a user may choose not to provide emotion-related data for a target content delivery service. In another example, a user may choose to limit the length of time that emotion-related data is retained, or completely prohibit the development of basic emotional conditions. In addition to providing "opt-in" and "opt-out" options, the present disclosure also contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then the user may be reminded again just before the personal information data is accessed by the application.

[0027] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0028] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with the user, other non-personal information available to a content delivery service, or publicly available information.

[0029] Figure 1 An exemplary system 100 for identifying content items to be recommended along with reference items 102 is illustrated in accordance with some embodiments of the present technology.

[0030] The system 100 includes a recommendation item candidate retrieval stage 106 for identifying candidate items 110 that may be related to the reference item 102. Given a reference item that a user is viewing, the recommendation item candidate retrieval stage 106 reduces the possible candidates in the candidate database 108 from millions to hundreds of candidates 110 or fewer.

[0031] While the present disclosure relates to reference terms, it is generally expected that reference terms will be content items such as media items such as web pages, books, applications, videos, music, social media posts, and / or products, but reference terms may also be search strings or even user accounts. The desired properties of a reference term are that candidate items related to the reference term can be identified and that the reference term is associated with a context.

[0032] The system 100 includes a ranking stage 112 for ranking the candidate items 110 using a ranking model 114. The ranking model 114 can fine-tune the ranking of the candidate items 110 to be presented to the user. For example, the ranking model 114 can be a machine learning algorithm that identifies items that are similar to a reference item, items that belong to a relevant category of the reference item, or items related to the reference item that the user is likely to view. Such algorithms typically power item-by-item recommendation systems such as "Frequently Bought Together," "Because You Viewed," and "You Might Also Like." The output of the ranking stage 112 is one or more of the highest ranked or most relevant candidates based on the reference item.

[0033] Although the recommendation item candidate retrieval stage 106 and the ranking stage 112 are shown as separate stages, they may be combined into a single stage.

[0034] The ranked candidates 116 may be provided to a context-sensitive classifier 118. In some embodiments, the context-sensitive classifier 118 is trained to predict the likelihood that a user will consider viewing the reference item 102 and the candidate items given a context. For example, the context-sensitive classifier 118 may be trained to predict whether a user would consider the reference item 102 and the candidate items to be constrained by a sensitive contextual relationship. In another example, the context-sensitive classifier 118 may be trained to predict whether a user would consider the reference item 102 and the candidate items to be constrained by a support relationship. In some embodiments, the context-sensitive classifier 118 may be trained to classify the type of relationship (e.g., context-sensitive, supportive, etc.) and the confidence of the classification.

[0035] The context-sensitive classifier 118 can determine a score or predicted probability of the likelihood that a user will consider viewing the reference item 102 and the candidate items given the context, and pass the result to the context filtering stage 120. In some embodiments, candidate items that may be considered to have an incompatible, sensitive contextual relationship with the reference item can be filtered. In some embodiments, the predicted probability can be a factor in the function used by the selection model 122 to select one of the candidates as the recommended item 104. For example, the selection model 122 can use the predicted probability to calculate a weighting factor to multiply the relevance score to determine which candidate item should be the recommended item. In some embodiments, when the predicted probability is above a threshold, it indicates that a negative contextual relationship may exist and the candidate can be filtered without further consideration.

[0036] The various stages of the system 100 may work together to provide recommendations 104 that are both relevant and contextually compatible.

[0037] Figure 2 An exemplary routine for determining whether a candidate item is context-compatible with a reference item according to some embodiments of the present technology is illustrated. Although the exemplary routine depicts a specific sequence of operations, the sequence may be changed without departing from the scope of the present disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality of the routine. In other examples, different components of the exemplary device or system that implements the routine may perform functions substantially simultaneously or in a specific order.

[0038] According to some examples, the method includes identifying at least one candidate item as being related to at least one reference item at block 202. For example, Figure 1 The recommendation item candidate retrieval stage 106 and ranking stage 112 illustrated in FIG. 1 may identify at least one candidate item as being related to at least one reference item.

[0039] As described above, at least one reference item can be a search criterion, a user profile, a media item such as a book, an application, a video, music, a website, a social media post, and / or a product. These are merely examples, and the required properties of the reference item are that a candidate item related to the reference item can be identified and that the reference item is associated with a context.

[0040] The at least one candidate item is a media item such as a book, application, video, music, website, social media post, and / or product. These are merely examples, and a desired property of a candidate item is that it is contextually relevant. In some embodiments, a candidate item may not necessarily be relevant. For example, if the candidate item is a promotional item, it may or may not be relevant. But even a promotional item should be evaluated for contextual relationship with the reference item.

[0041] According to some examples, the method includes receiving at least one reference item and at least one candidate item at block 204. For example, Figure 1 The context-sensitive classifier 118 illustrated in can receive at least one reference item and at least one candidate item.

[0042] According to some examples, the method includes creating a reference term embedding at block 206, and creating a candidate term embedding. For example, Figure 1 The context-sensitive classifier 118 illustrated in can create a reference item embedding and create a candidate item embedding.

[0043] like Figure 3 , the context-sensitive classifier 118 includes an embedding layer 314, a neural network 308, and a classification layer 310. In some embodiments, the neural network 308 is a multi-layer perceptron neural network.

[0044] The embedding layer is a two-tower neural network model that generates reference term embeddings and candidate term embeddings. The reference term embedding function 302 independently generates the reference term embeddings, and the candidate term embedding function 306 independently generates the candidate term embeddings.

[0045] In some embodiments, the embedding layer 314 creates the reference term embedding and the candidate term embedding using a language model as a reference term embedding function 302 and a candidate term embedding function 306. The language model may be specialized by tuning using an advertisement dataset related to the content item. In some embodiments, the language model may be a large language model or a transformer-based large language model.

[0046] According to some examples, the method includes determining at least one relationship feature between at least one reference item and at least one candidate item at block 208. For example, Figure 1 The context-sensitive classifier 118 illustrated in can determine at least one relationship feature between at least one reference item and at least one candidate item.

[0047] A "relationship feature" refers to a parameter or expression that describes the relationship between a reference item and a candidate item. A relationship feature can be a measure of the similarity or correlation between a candidate item and a reference item. For example, the measure can be a similarity score, a distance between embeddings in an embedding space, etc. A relationship feature can be an expression that expresses a relationship between properties of a candidate item and a reference item. For example, an expression where the candidate item and the reference item are of the same type is false, or an expression where the candidate item and the reference item have compatible themes is true. Some properties that may be useful for expressions in this context include properties that indicate whether the candidate item and the reference item are suitable for audiences of different ages or for audiences with different markings. For example, a children's reference item is not compatible with a mature audience candidate item. Other properties that may be useful for expressions in this context include properties that indicate whether the candidate item and the reference item belong to mixed or unmixed subject categories. For example, a reference item related to marriage counseling will not be mixed with a candidate item related to divorce. Other properties that may be useful for expressions in this context include properties that indicate whether the candidate item and the reference item are spatially related. For example, a reference item related to events that occurred in Miami will not be mixed with a reference item related to a candidate item related to Seattle. Other properties that may be useful for expressions may relate to the nature of support between two content items. For example, a first topic (eg, obtaining a visa) supports the presentation of a second topic (eg, traveling to an international destination).

[0048] According to some examples, the method includes inputting at least one relational feature into a classification layer along with a reference term embedding and a candidate term embedding at block 210. For example, Figure 1 The context-sensitive classifier 118 illustrated in can input at least one relational feature into the classification layer together with the reference item embedding and the candidate item embedding.

[0049] like Figure 3 As illustrated in , the relationship features 304 are provided to the neural network 308 along with the reference item embedding and the candidate item embedding.

[0050] According to some examples, the method includes providing a prediction of whether the candidate item is contextually compatible or contextually incompatible with the reference item at block 212. For example, Figure 1 The context-sensitive classifier 118 illustrated in can provide a prediction of whether a candidate item is contextually compatible or contextually incompatible with a reference item.

[0051] like Figure 3 As illustrated in , the classification layer 310 of the context-sensitive classifier 118 provides a prediction of whether a candidate item is contextually compatible or contextually incompatible with a reference item. The prediction may include a probability of whether a reference item and a candidate item pair are considered contextually compatible.

[0052] In some embodiments, the context-sensitive classifier 118 is trained to evaluate a single type of context, such as whether a pair of content items has a sensitive contextual relationship, or whether a pair of content items has a supportive contextual relationship. When evaluating a single type of context, the output may be a probability that the type of context is applicable.

[0053] In some embodiments, the context-sensitive classifier 118 is trained to evaluate multiple types of context simultaneously. When evaluating multiple types of context, the output may be an identification of the type of context and a probability that the type of context is applicable.

[0054] According to some examples, the method includes determining whether to present at least one candidate item in a user interface based on a prediction of whether the candidate item is contextually compatible or contextually incompatible with the reference item at block 214. For example, Figure 1 The selection model 122 illustrated in can determine whether to present at least one candidate item in the user interface based on a prediction of whether the candidate item is contextually compatible or contextually incompatible with a reference item.

[0055] According to some examples, the method includes causing at least one candidate item to be presented in a user interface at block 216. For example, Figure 1 The selection model 122 illustrated in FIG. 1 may cause at least one candidate item to be presented in the user interface. A reference item is also presented in the user interface.

[0056] Figure 3 An example of a context-sensitive classifier 118 in accordance with some embodiments of the present technology is illustrated.

[0057] like Figure 3 As illustrated in , the context-sensitive classifier 118 includes an embedding layer 314, a neural network 308, and a classification layer 310.

[0058] The embedding layer is a dual-tower neural network model (e.g., Deep Structured Semantic Model (C-DSSM) and Dual Bidirectional Encoder Representations from Transformers (BERT)) that generates reference term embeddings and candidate item embeddings. A "dual-tower neural network model" refers to a neural network with two sub-models that learn embeddings for reference terms and candidate items, respectively. The reference term embedding function 302 is the first tower that independently generates reference term embeddings, and the candidate item embedding function 306 is the second tower that independently generates candidate item embeddings.

[0059] In some embodiments, the embedding layer 314 creates the reference term embedding and the candidate term embedding using a language model as a reference term embedding function 302 and a candidate term embedding function 306. The language model may be specialized by tuning using an advertisement dataset related to the content item. In some embodiments, the language model may be a large language model or a transformer-based large language model.

[0060] The input features for each tower can include textual representations of items (e.g., embeddings of item overviews and descriptions) and item metadata (category, recommended age limit, etc.). The item embedding representations are generated using a custom encoder obtained by fine-tuning the bidirectional encoder representation from the Transformer (BERT) language model to perform multi-label category classification. The input to each tower can be represented by E = {e1, e2, ..., e i ,...e f} means that e i ∈R k represents the embedding or metadata feature of the i-th field of the feature, where f represents the total number of fields and k represents the embedding dimension. The output of each tower can be represented by p = {p1, p2, ..., p n} and q={q1,q2,...,q n}, where n represents the output dimension.

[0061] The outputs of the reference item embedding function 302 and the candidate item embedding function 306 are concatenated and combined with the relation features 304 to form the input to the matching layer.

[0062] A "relationship feature" refers to a parameter or expression that describes the relationship between a reference item and a candidate item. A relationship feature can be a measure of the similarity or correlation between a candidate item and a reference item. For example, the measure can be a similarity score, a distance between embeddings in an embedding space, etc. A relationship feature can be an expression that expresses a relationship between properties of a candidate item and a reference item. For example, an expression where the candidate item and the reference item are of the same type is false, or an expression where the candidate item and the reference item have compatible themes is true. Some properties that may be useful for expressions in this context include properties that indicate whether the candidate item and the reference item are suitable for audiences of different ages or for audiences with different markings. For example, a children's reference item is not compatible with a mature audience candidate item. Other properties that may be useful for expressions in this context include properties that indicate whether the candidate item and the reference item belong to mixed or unmixed subject categories. For example, a reference item related to marriage counseling will not be mixed with a candidate item related to divorce. Other properties that may be useful for expressions in this context include properties that indicate whether the candidate item and the reference item are spatially related. For example, a reference item related to events that occurred in Miami will not be mixed with a reference item related to a candidate item related to Seattle. Other properties that may be useful for expressions may relate to the nature of support between two content items. For example, a first topic (eg, obtaining a visa) supports the presentation of a second topic (eg, traveling to an international destination).

[0063] In some embodiments, the relationship feature 304 can be a set of expressions. For example, the relationship feature 304 can be <similarity score=0.8|=true, compatible age range|=false, compatible subject|candidate=promoted>. In this example, the relationship feature expresses a similarity feature that provides context for why the candidate was considered, provides conclusions about various aspects of the contextual relationship between the reference item and the candidate, and indicates that the candidate is an actively promoted item (e.g., the candidate received preferential treatment compared to potentially more similar candidates).

[0064] It should be understood that the types of expressions and data included in the relational features are presented herein only as examples. The specific type of data or representation of the data may be different. The only requirement for the relational feature is that it provides some context relating the reference item to the candidate item. In some embodiments, relational features are not required because the matching layer may also learn to identify some of these relational features.

[0065] In some embodiments, conclusions about relationships between contexts such as compatible age ranges and subject matter may be provided by heuristics, machine learning algorithms, or other functions.

[0066] like Figure 3 As illustrated in FIG. 3 , the relation features 304 are provided to a neural network 308 along with the reference term embedding and the candidate term embedding. In some embodiments, the neural network 308 is a multi-layer perceptron neural network.

[0067] The neural network 308 uses additional features that capture aspects of the relationships between items, including item similarities and other statistical features. These features r = {r1, r2, ..., r m} will be concatenated with p and q to form a 2n+m-dimensional dense real-valued feature vector x = {x1, x2, ..., x 2n+m} and is fed to a In the deep network of , L is the depth of the matching layer deep network, W, b, a are the model weights, biases and outputs in each layer. The learning process minimizes the cross entropy as the objective function:

[0068] The classification layer 310 utilizes the output of the neural network 308 to provide a prediction 312. The classification layer 310 can provide a prediction of whether the candidate item is contextually compatible or contextually incompatible with the reference item. The prediction can include a probability of whether the reference item and the candidate item pair are considered contextually compatible.

[0069] In some embodiments, the neural network 308 and the classification layer 310 are trained to evaluate a single type of context, such as whether a pair of content items has a sensitive contextual relationship, or whether a pair of content items has a supportive contextual relationship. When evaluating a single type of context, the output may be a probability that the type of context is applicable.

[0070] In some embodiments, the neural network 308 and the classification layer 310 are trained to evaluate multiple types of context simultaneously. When evaluating multiple types of context, the output may be an identification of the type of context and a probability that the type of context is applicable.

[0071] Figure 4 An exemplary routine for training a context-sensitive classifier 118 according to some aspects of the present technology is illustrated. Although the exemplary routine depicts a particular order of operations, the order may be changed without departing from the scope of the present disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality of the routine. In other examples, different components of the exemplary device or system implementing the routine may perform functions substantially simultaneously or in a particular order.

[0072] According to some examples, the method includes marking the pair of terms as contextually compatible or contextually incompatible at block 402. In some embodiments, marking the pair of terms further includes marking the pair of terms as contextually supporting or contextually complementary.

[0073] According to some examples, the method includes, at block 404 , providing the term pair to a context-sensitive classifier.

[0074] According to some examples, the method includes receiving a prediction from a context-sensitive classifier as to whether the pair of terms are contextually compatible at block 406. In some embodiments, the prediction is a probability that the candidate term is contextually incompatible with the reference term.

[0075] According to some examples, the method includes, at block 408 , providing positive feedback to the context-sensitive classifier when the prediction is correct, and providing negative feedback to the context-sensitive classifier when the prediction is incorrect.

[0076] According to some examples, the method includes continuing to train the context-sensitive classifier at block 410 until the context-sensitive classifier achieves a threshold performance metric for correct predictions.

[0077] According to some examples, the method includes, at block 412 , deploying the context-sensitive classifier in a production environment.

[0078] According to some examples, the method includes periodically continuing to train the context-sensitive classifier with new labeled pairs at box 414. In some embodiments, the new labeled pairs are labeled by a user who gives negative feedback / positive feedback about the candidate items when the candidate items are presented in the user interface. In some embodiments, the new labeled pairs are selected using active learning techniques that select pairs of items that are more difficult to classify. The item pairs are more difficult to classify when the context-sensitive classifier outputs ambiguous prediction probabilities, when the labeler is divided about the correct label for the pair, or when the items are both associated with sensitive categories or contexts.

[0079] More specifically, an implementation scheme of the context-sensitive classifier 118 is evaluated offline using manually labeled data for sensitive contextual relationships. The evaluation dataset is balanced and consists of 8k item pairs. The context-sensitive classifier 118 is compared with a rule-based baseline method. The baseline rule uses existing embedding signals of reference items and candidate items, and the sensitivity of the contextual relationship is determined by item distance and item category features. ROC-AUC, recall, and precision are used as metrics for evaluation. The goal is a recall close to 1, because it is critical to eliminate sensitive contextual relationship pairs from the display and remove any insensitive experience for users and advertisers. Therefore, a threshold is set to ensure a high recall when weighing against precision. Table 1 shows the experimental results.

[0080] The current technology outperforms rule-based methods on all evaluation metrics. The context-sensitive classifier 118 achieves a high recall of > 0.99 to almost completely eliminate sensitive contextual relationships from the display. In addition, our proposal has a precision of 0.6411, while the rule-based baseline has a much lower precision of 0.1335. Systems with low precision may be too conservative and issue false alarms, which prevents good item-by-item recommendations from being displayed and harms recommendation diversity. In this regard, compared to the baseline method, the present technology has improved the precision by nearly 5 times, greatly reducing false positives, and improving the discovery opportunities in the recommendation system.

[0081]

[0082] Table 1 - Experimental results The proposed method is compared with a rule-based baseline method.

[0083] Figure 5 Exemplary sampling strategies for active learning in accordance with aspects of the present technology are illustrated.

[0084] In some embodiments, an active sampling strategy may be deployed to improve the ability of the context-sensitive classifier 118 to perform well on difficult prediction tasks. In order to effectively eliminate controversial recommendations, the most influential items within sensitive categories are prioritized. For example, sensitive categories include health, medicine, and pharmaceutical industries, categories with romantic themes, items with monetary incentives or that may have an impact on the user's financial health, etc. These item samples are ranked based on predicted high click-through rates (CTR) and factors such as co-engagement statistics, item similarities, and related categories. The sampled <reference item, candidate item> pairs are unlabeled raw data, such as Figure 5 shown.

[0085] Active learning is crucial, especially when dealing with large unlabeled data, as it improves learning by selecting the most informative samples based on the prediction confidence of the unlabeled data. In this process, sampling is done adaptively as the model predicts the confidence of whether an item pair is problematic or not. Low confidence data samples are sent for human labeling. This human-machine interactive approach not only reduces the amount of labeled data required for effective training, but also improves model performance in less time.

[0086] exist Figure 6 In the present disclosure, the present disclosure now turns to further discussion of models that can be used through the environment and techniques described herein. Specifically, Figure 6 is an illustrative example of a deep learning neural network 600 that can be used to implement all or a portion of the context-sensitive classifier 118 as discussed above. The input layer 602 can be configured to receive sensor data and / or data related to the environment surrounding the AV. The neural network 600 includes a plurality of hidden layers 604a, 604b to 604c. The hidden layers 604a to 604c include "n" hidden layers, where "n" is an integer greater than or equal to one. The number of hidden layers can include as many layers as required for a given application. The neural network 600 also includes an output layer 606 that provides outputs generated by the processing performed by the hidden layers 604a to 604c. In an illustrative example, the output layer 606 can provide estimated treatment parameters that can be used / ingested by a differential simulator to estimate patient treatment outcomes.

[0087] Neural network 600 is a multi-layer neural network of interconnected nodes. Each node may represent a piece of information. Information associated with a node is shared between different layers, and each layer retains the information as it is processed. In some cases, neural network 600 may include a feedforward network, in which case there is no feedback connection that feeds the output of the network back to itself. In some cases, neural network 600 may include a recursive neural network, which may have loops that allow information to be carried across nodes as input is read in.

[0088] Information can be exchanged between nodes by node-to-node interconnection between each layer. The nodes of input layer 602 can activate a group of nodes in the first hidden layer 604a. For example, as shown in the figure, each input node in the input nodes of input layer 602 is connected to each node in the nodes of the first hidden layer 604a. The nodes of the first hidden layer 604a can transform the information of each input node by applying an activation function to the input node information. The information derived from the transformation can then be passed to the nodes of the next hidden layer 604b, and these nodes can be activated, and these nodes can perform the functions specified by themselves. Exemplary functions include convolution, upsampling, data conversion and / or any other suitable functions. Then, the output of hidden layer 604b can activate the nodes of the next hidden layer, etc. The output of the last hidden layer 604c can activate one or more nodes of output layer 606, providing output at the one or more nodes. In some cases, although the nodes in neural network 600 are shown as having multiple output lines, the nodes can have a single output, and all lines shown as outputs from the nodes represent the same output value.

[0089] In some cases, each node or interconnection between nodes may have a weight, which is a set of parameters derived from the training of neural network 600. Once neural network 600 is trained, it may be referred to as a trained neural network, which may be used to classify one or more activities. For example, an interconnection between nodes may represent a piece of information learned about the interconnected nodes. The interconnections may have tunable numerical weights that may be tuned (e.g., based on a training data set), thereby allowing neural network 600 to adapt to the input and be able to learn as more and more data is processed.

[0090] The neural network 600 is pre-trained to process features of data from the input layer 602 using different hidden layers 604 a - 604 c to provide an output via an output layer 606 .

[0091] In some cases, neural network 600 may use a training process called back propagation to adjust the weights of the nodes. The back propagation process may include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter / weight update are performed for one training iteration. For each set of training data, this process may be repeated for a certain number of iterations until neural network 600 is trained well enough so that the weights of the layers are accurately tuned.

[0092] To perform training, a loss function may be used to analyze the error in the output. Any suitable loss function definition may be used, such as a cross entropy loss. Another example of a loss function includes mean square error (MSE), which is defined as E_total = ∑(1 / 2(target - output)^2). The loss may be set equal to the value of E_total.

[0093] The loss (or error) for the initial training data will be high because the actual values ​​will be very different from the predicted outputs. The goal of training is to minimize the amount of loss so that the predicted outputs are the same as the training outputs. The neural network 600 can perform a backward pass by determining which inputs (weights) contribute the most to the network's loss, and the weights can be adjusted so that the loss is reduced and ultimately minimized.

[0094] The neural network 600 may include any suitable deep network. One example includes a convolutional neural network (CNN), which includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. The hidden layers of the CNN include a series of convolutional, nonlinear, pooling (for downsampling), and fully connected layers. The neural network 600 may include any other deep network other than a CNN, such as an autoencoder, a deep belief network (DBN), a recursive neural network (RNN), and the like.

[0095] As will be appreciated by those skilled in the art, machine learning-based classification techniques may vary depending on the desired implementation. For example, machine learning classification schemes may utilize one or more of the following alone or in combination: hidden Markov models; RNNs; CNNs; deep learning; Bayesian symbolic methods; generative adversarial networks (GANs); support vector machines; image registration methods; and applicable rule-based systems. In the case of using a regression algorithm, the regression algorithm may include, but is not limited to, a stochastic gradient descent regressor, a passive attack regressor, and the like.

[0096] The machine learning classification model may also be based on a clustering algorithm (e.g., a mini-batch K-means clustering algorithm), a recommendation algorithm (e.g., a Minwise hash function or a Euclidean locality sensitive hashing (LSH) algorithm), and / or an anomaly detection algorithm, such as a local anomaly factor. Additionally, the machine learning model may employ a dimensionality reduction method, such as one or more of the following: a mini-batch dictionary learning algorithm, an incremental principal component analysis (PCA) algorithm, a latent Dirichlet distribution algorithm, and / or a mini-batch K-means algorithm, etc.

[0097] Figure 7 An exemplary lifecycle 700 of an ML model according to some aspects of the present technology is illustrated. The first stage of the ML model lifecycle 700 is a data ingestion service 702 to generate the datasets described below. Figure 7The various processes described in require large amounts of data, and the data is preserved without any transformation to obtain an immutable record of the original data set. The data can be provided from third-party sources (such as publicly available private data sets). The data ingestion service 702 provides a service that allows efficient querying and end-to-end data lineage and traceability based on a dedicated pipeline for each data set, allows data partitioning to utilize multiple servers or cores, and allows data to be spread across multiple pipelines to reduce the overall time and thus reduce data retrieval functions.

[0098] In some cases, data can be retrieved offline, which decouples the producer of data from the consumer of data (e.g., an ML model training pipeline). For offline data production, when source data is available from the producer, the producer publishes a message and the data ingestion service 702 retrieves the data. In some examples, the data ingestion service 702 can be online, and data is streamed from the producer in real time for storage in the data ingestion service 702.

[0099] After the data ingestion service 702, the data preprocessing service preprocesses the data to prepare the data for use in the life cycle 700, and includes at least data cleaning, data transformation, and data selection operations. The data cleaning and annotation service 704 removes irrelevant data (data cleaning) and general preprocessing to transform the data into a usable form. The data cleaning and annotation service 704 includes labeling features relevant to the ML model. In some examples, the data cleaning and annotation service 704 can be a semi-supervised process performed by ML to clean and annotate data, which is supplemented with manual operations such as marking error scenarios, identifying untrained features, etc.

[0100] After the data cleaning and annotation service 704, the data isolation service 706 separates the data into at least a training set 708, a validation data set 710, and a test data set 712. Each of the training set 708, the validation data set 710, and the test data set 712 is distinct and does not include any common data to ensure that the evaluation of the ML model is isolated from the training of the ML model.

[0101] The training set 708 is provided to a model training service 714, which uses a supervisor to perform training of the ML model or initial fitting of parameters (e.g., weights of connections between neurons in an artificial neural network). The model training service 714 trains the ML model based on gradient descent or stochastic gradient descent to fit the ML model based on an input vector (or scalar) and a corresponding output vector (or scalar).

[0102] After training, the ML model is evaluated at the model evaluation service 716 using data from the validation dataset 710 and different evaluators to tune the hyperparameters of the ML model. The predictive performance of the ML model is evaluated based on predictions on the validation dataset 710, and the hyperparameters are iteratively tuned based on the different evaluators until the best fit for the ML model is identified. After the best fit is identified, the test dataset 712 or holdout dataset is used as a final check to perform an unbiased measurement of the performance of the final ML model by the model evaluation service 716. In some cases, the final dataset used for the final unbiased measurement may be referred to as a validation dataset, and the dataset used for hyperparameter tuning may be referred to as a test dataset.

[0103] After the model evaluation service 716 has evaluated the ML model, the ML model deployment service 718 can deploy the ML model to an application or a suitable device. It can be deployed to a further test environment (such as a simulation environment), or to another control environment to further test the ML model.

[0104] After deployment by ML model deployment service 718, performance monitoring service 720 monitors the performance of the ML model. In some cases, performance monitoring service 720 may also record additional transaction data that may be ingested via data ingestion service 702 to provide further data, additional context, and further enhance the training of the ML model.

[0105] Figure 8 An example of a computing system 800 is shown, which may be, for example, any computing device that constitutes the context-sensitive classifier 118 or any component thereof, wherein the components of the system communicate with each other using a connection 802. The connection 802 may be a physical connection via a bus, or a direct connection into the processor 804, such as in a chipset architecture. The connection 802 may also be a virtual connection, a networked connection, or a logical connection.

[0106] In some embodiments, computing system 800 is a distributed system, wherein the functionality described in the present disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components that each perform some or all of the functionality that the component is described for. In one embodiment, these components may be physical devices or virtual devices.

[0107] The exemplary computing system 800 includes at least one processing unit (CPU or processor) 804 and connections 802 that couple various system components, including system memory 808, such as read-only memory (ROM) 810 and random access memory (RAM) 812, to the processor 804. The computing system 800 may include a cache of high-speed memory 806 directly connected to the processor 804, in close proximity to the processor, or integrated as part of the processor.

[0108] Processor 804 may include any general purpose processor and hardware or software services (such as services 816, 818, and 820 stored in storage device 814) configured to control processor 804 as well as a dedicated processor in the case where software instructions are incorporated into the actual processor design. Processor 804 may essentially be a completely self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0109] In order to realize user interaction, computing system 800 includes input device 826, and this input device can represent any number of input mechanisms, such as microphone for voice, touch-sensitive screen for gesture or graphic input, keyboard, mouse, motion input, voice, etc. The computing system 800 may also include output device 822, and this output device may be one or more output mechanisms among a plurality of output mechanisms known to those skilled in the art. In some cases, multimodal system enables the user to provide multiple types of input / output to communicate with computing device 800. Computing system 800 may include communication interface 824, and this communication interface can control and manage user input and system output generally. Operation is not limited to any specific hardware arrangement, therefore, the basic features of this article can be easily replaced with these hardware or firmware arrangements after developing improved hardware or firmware arrangements.

[0110] The storage device 814 may be a non-volatile memory device and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic tape cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cartridge, a random access memory (RAM), a read-only memory (ROM), and / or some combination of these devices.

[0111] Storage device 814 may include software services, servers, services, etc., which cause the system to perform functions when code defining such software is executed by processor 804. In some embodiments, hardware services that perform a particular function may include software components stored in a computer-readable medium in combination with the necessary hardware components (such as processor 804, connection 802, output device 822, etc.) to perform the function.

[0112] For clarity of explanation, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks including devices, device components, steps or routines in a method embodied in software, or a combination of hardware and software.

[0113] Any of the steps, operations, functions or processes described herein may be performed or implemented by a combination of hardware and software services or services alone or in combination with other devices. In some embodiments, the service may be software that resides in the memory of the customer-end device and / or one or more servers of the content management system and performs one or more functions when the processor executes the software associated with the service. In some embodiments, the service is a program or a collection of programs that perform a specific function. In some embodiments, the service may be considered a server. The memory may be a non-transient computer-readable medium.

[0114] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0115] The method according to the above example can be implemented using computer executable instructions stored in a computer readable medium or otherwise obtainable from the computer readable medium. Such instructions may include, for example, instructions and data that make a general-purpose computer, a special-purpose computer, or a special processing device or otherwise configure the above-mentioned device to perform a certain function or a group of functions. The part of the computer resources used can be accessed through a network. Executable computer instructions can be, for example, binary, intermediate format instructions such as assembly language, firmware or source code. Examples of computer readable media that can be used to store instructions, information used and / or information generated during the method according to the example include disks or optical disks, solid-state memory devices, flash memory, USB devices provided with non-volatile memory, networked storage devices, etc.

[0116] Devices implementing methods according to these disclosures may include hardware, firmware, and / or software, and may take any of a variety of form factors. Typical examples of such form factors include servers, laptops, smart phones, small form factor personal computers, personal digital assistants, etc. The functionality described herein may also be embodied in peripheral devices or expansion cards. By way of another example, such functionality may also be implemented between different chips on a circuit board or in different processes executed in a single device.

[0117] Instructions, media for transmitting such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are means for providing the functionality described in these disclosures.

[0118] Some aspects of this technology include:

[0119] Aspect 1. A method, comprising: receiving at least one reference item and at least one candidate item by a context-sensitive classifier; providing a prediction by the context-sensitive classifier as to whether the candidate item is contextually compatible or contextually incompatible with the reference item; and determining whether to present the at least one candidate item in a user interface based on the prediction as to whether the candidate item is contextually compatible or contextually incompatible with the reference item.

[0120] Aspect 2. According to the method described in Aspect 1, the method also includes: presenting the at least one candidate item in the user interface.

[0121] Aspect 3. The method according to any one of Aspects 1 to 2, wherein the reference item is also presented in the user interface.

[0122] Aspect 4. According to the method described in any one of Aspects 1 to 3, the method further comprises: before the at least one candidate item is received by the context-sensitive classifier, identifying the at least one candidate item as being related to the at least one reference item.

[0123] Aspect 5. The method according to any one of Aspects 1 to 4, wherein the at least one candidate item is a promotion item.

[0124] Aspect 6. The method according to any one of Aspects 1 to 5, wherein the at least one candidate item is a promotional item and is related to the at least one reference item.

[0125] Aspect 7. The method according to any one of Aspects 1 to 6, wherein the at least one reference item is a search criterion, a media item such as a book, an application, a video, music, a social media post and / or a product.

[0126] Aspect 8. The method according to any one of Aspects 1 to 7, wherein the at least one candidate item is a media item such as a book, an application, a video, music, a social media post and / or a product.

[0127] Aspect 9. According to any one of aspects 1 to 8, the method further includes: creating a reference item embedding and creating a candidate item embedding after receiving the at least one reference item and the at least one candidate item and before providing the prediction.

[0128] Aspect 10. A method according to any one of Aspects 1 to 9, wherein the context-sensitive classifier comprises an embedding layer and a classification layer, wherein the classification layer is a multi-layer perceptron neural network.

[0129] Aspect 11. A method according to any one of Aspects 1 to 10, wherein the embedding layer is a dual-tower neural network model that generates the reference item embedding and the candidate item embedding.

[0130] Aspect 12. According to the method of any one of Aspects 1 to 11, the method further comprises: determining at least one relationship feature between the at least one reference item and the at least one candidate item.

[0131] Aspect 13. According to the method of any one of Aspects 1 to 12, the method further comprises: inputting the at least one relational feature together with the reference item embedding and the candidate item embedding into the classification layer.

[0132] Aspect 14. A method according to any one of Aspects 1 to 13, wherein the embedding layer utilizes a language model to create the reference item embedding and the candidate item embedding.

[0133] Aspect 15. A method according to any one of Aspects 1 to 14, wherein the language model is trained on an advertisement dataset related to the content item.

[0134] Aspect 16. A method according to any one of Aspects 1 to 15, wherein the context-sensitive classifier is a trained machine learning model trained according to the method, the method comprising: labeling an item pair as context-compatible or context-incompatible; providing the item pair to the context-sensitive classifier; receiving a prediction from the context-sensitive classifier as to whether the item pair is context-compatible; and providing positive feedback to the context-sensitive classifier when the prediction is correct, and providing negative feedback to the context-sensitive classifier when the prediction is incorrect.

[0135] Aspect 17. The method according to any one of Aspects 1 to 16, wherein marking the pair of terms further comprises marking the pair of terms as contextually supporting or contextually complementary.

[0136] Aspect 18. A method according to any one of Aspects 1 to 17, the method comprising: continuing to train the context-sensitive classifier until the context-sensitive classifier achieves a threshold performance indicator for correct prediction; after the context-sensitive classifier achieves the threshold performance indicator for correct prediction, deploying the context-sensitive classifier in a production environment; and periodically continuing to train the context-sensitive classifier with new labeled pairs.

[0137] Aspect 19. A method according to any one of Aspects 1 to 18, wherein the new marked pair is marked by a user, and when the candidate item is presented in the user interface, the user gives negative feedback / positive feedback about the candidate item.

[0138] Aspect 20. A method according to any one of aspects 1 to 19, wherein the new labeled pairs are selected using an active learning technique that selects pairs of terms that are more difficult to classify.

[0139] Aspect 21. According to the method of any one of Aspects 1 to 20, the item pair is more difficult to classify when the context-sensitive classifier outputs an ambiguous prediction probability, or when the labeler is divided as to the correct label for the pair.

[0140] Aspect 22. A method according to any one of aspects 1 to 21, wherein the prediction is a probability that the candidate item is contextually incompatible with the reference item.

Claims

1. A method comprising: receiving, by a context-sensitive classifier, at least one reference item and at least one candidate item; providing, by the context-sensitive classifier, a prediction of whether the candidate item is contextually compatible or contextually incompatible with the reference item; as well as A determination is made whether to present the at least one candidate item in a user interface based on the prediction of whether the candidate item is contextually compatible or contextually incompatible with the reference item.

2. The method according to claim 1, further comprising: When the at least one candidate item is context-compatible with the reference item, the at least one candidate item is presented in the user interface.

3. The method according to claim 1, further comprising: Prior to receipt of the at least one candidate item by the context-sensitive classifier, the at least one candidate item is identified as being related to the at least one reference item.

4. The method according to claim 1, further comprising: After receiving the at least one reference item and the at least one candidate item and before providing the prediction, a reference item embedding is created, and a candidate item embedding is created.

5. The method according to claim 1, wherein the context-sensitive classifier comprises an embedding layer and a classification layer, wherein the classification layer is a multi-layer perceptron neural network, wherein the embedding layer is a double-tower neural network model, and the double-tower neural network model generates the reference item embedding and the candidate item embedding.

6. The method according to claim 5, further comprising: At least one relationship characteristic between the at least one reference item and the at least one candidate item is determined.

7. The method according to claim 6, further comprising: The at least one relational feature is input into the classification layer along with the reference item embedding and the candidate item embedding.

8. The method of claim 5, wherein the embedding layer utilizes a language model to create the reference item embedding and the candidate item embedding.

9. The method of claim 8, wherein the language model is trained on an advertisement dataset associated with a content item.

10. A computing system comprising: processor; and a memory storing instructions that, when executed by the processor, configure the system to: receiving, by a context-sensitive classifier, at least one reference item and at least one candidate item; providing, by the context-sensitive classifier, a prediction of whether the candidate item is contextually compatible or contextually incompatible with the reference item; as well as A determination is made whether to present the at least one candidate item in a user interface based on the prediction of whether the candidate item is contextually compatible or contextually incompatible with the reference item.

11. The computing system of claim 10, wherein the instructions further configure the system to: Prior to receipt of the at least one candidate item by the context-sensitive classifier, the at least one candidate item is identified as being related to the at least one reference item.

12. The computing system of claim 10, wherein the instructions further configure the system to: After receiving the at least one reference term and the at least one candidate term and before providing the prediction, creating a reference term embedding, and creating a candidate term embedding.

13. The computing system of claim 11, wherein the instructions further configure the system to: At least one relationship characteristic between the at least one reference item and the at least one candidate item is determined.

14. The computing system of claim 13, wherein the instructions further configure the system to: The at least one relational feature is input into the classification layer along with the reference item embedding and the candidate item embedding.

15. A computing device comprising instructions that, when executed by the computing device, cause the computing device to: receiving, by a context-sensitive classifier, at least one reference item and at least one candidate item; providing, by the context-sensitive classifier, a prediction of whether the candidate item is contextually compatible or contextually incompatible with the reference item; as well as A determination is made whether to present the at least one candidate item in a user interface based on the prediction of whether the candidate item is contextually compatible or contextually incompatible with the reference item.

16. A computing device according to claim 15, wherein the context-sensitive classifier includes an embedding layer and a classification layer, wherein the classification layer is a multi-layer perceptron neural network, wherein the embedding layer is a double-tower neural network model, and the double-tower neural network model generates the reference item embedding and the candidate item embedding.

17. The computing device of claim 16, wherein the instructions further configure the computing device to: At least one relationship characteristic between the at least one reference item and the at least one candidate item is determined.

18. The computing device of claim 17, wherein the instructions further configure the computing device to: The at least one relational feature is input into the classification layer along with the reference item embedding and the candidate item embedding.

19. The computing device of claim 16, wherein the embedding layer utilizes a language model to create the reference term embedding and the candidate term embedding.

20. The computing device of claim 19, wherein the language model is trained on an advertisement dataset associated with content items.

Images