Item interaction prediction model combining id and text

Abstract

A method for operating a user interface includes initially training an item identifier prediction model to predict a next item given an input series of item identifiers and an item text prediction model to predict a next item given an input series of item text, generating, using the trained item identifier prediction model, a first set of hard negative examples, generating, using the trained item text prediction model, a second set of hard negative examples, retraining the item identifier prediction model using the second set of hard negative examples, and retraining the item text prediction model using the first set of hard negative examples. The method includes inputting a series of item interactions received from the user to the retrained item identifier prediction model and the retrained item text prediction model to generate a predicted next item.

Description

ITEM INTERACTION PREDICTION MODEL COMBINING ID AND TEXTCross-Reference to Related Application

[0001] This application claims the benefit of priority to US provisional application no. 63 / 551,041, filed on February 7, 2024, which is hereby incorporated by reference in its entirety.Field of the Disclosure

[0002] The present disclosure generally relates to predicting user interactions with items, including by combining session data containing user interactions with unique item identifiers, and semantic data encoded from item descriptions.Brief Description of the Drawings

[0003] FIG. 1 is a block diagram illustrating an example system for predicting and ranking user-item interactions.

[0004] FIG. 2 is a flow chart illustrating an example method of providing next item recommendations in response to user activity.

[0005] FIG. 3 is a flow chart illustrating an example method of training and applying a user-item interaction prediction model.

[0006] FIG. 4 is a diagrammatic view of an example user computing environment.Detailed Description

[0007] A robust system for predicting the next item a user may interact with in a particular browsing session should be able to predict future user interactions with items based on unique item identifiers as well as semantic similarities among item descriptions and titles.Traditional item identification-based models (“ID Models”), convert each item ID into a vector representation. Deep neural networks, such as recurrent neural networks, employ encoders to analyze user- item interaction sequences. These neural networks operate on the base principal of tracking unique identifiers (IDs) associated with items and users to build a predictive model to suggest what item a user is likely to interact with next based off of user history. However, ID Models alone fail to take into account valuable information about an item, as the models convert each item into a unique identifier or vector and do not utilize the valuable information about an item stored in the item’s title or description.

[0008] Text-Integrated Models (“Text Models”) for item interaction prediction combine text information in an item’s title and / or description to build semantic information about the item. Text Models can thus make associations among items based on their semantic similarities and predict items for user interactions based on those similarities. Recentadvancements in natural language processing and deep learning have led to the development of powerful pre-trained language models, such as BERT, GPT, and RoBERTa. These models have demonstrated impressive performance in various tasks, including e-commerce retrieval and ranking, by capturing the semantic relationships between queries and items.

[0009] In theory, combining an ID Model and a Text Model to form a hybrid Text / ID Model should lead to a system which more accurately predicts future item interactions. However, studies have shown that independent or “Uni-modal” Models (either ID or Text) perform as good as, and sometimes even better than, combined or hybrid models. Current hybrid models may only concatenate or sum an ID and Text model into a “naive fusion” model. In these scenarios one model, either ID or Text, predominates the prediction modeling and the other model either does nothing to support the predictive model or even hinders the predictive model with contradicting data. Thus, an alternative training approach is provided in this disclosure which separates the training of the ID Model from the training of the Text Model, thereby avoiding the issues in naive fusion. Additionally, the disclosed training process utilizes hard negative samples and / or hard positive samples generated by one model to train the other, enabling them to mutually enhance each other and thus improve overall performance.

[0010] Referring now to the drawings, wherein like numerals refer to the same or similar features in the various views, FIG. 1 is a diagrammatic view of an example system 100 for operating a user interface, including predicting future user-item interactions in the interface. The system 100 may include a source of training data 102, a user-item interaction prediction and ranking system 104, and a server 106 in electronic communication with one or more user computing devices 108a, 108b, . . . 108n.

[0011] The server 106 may, in conjunction with the user-item interaction prediction and ranking system 104, host or otherwise provide (e.g., support) a website, data for a mobile application, or other interface through which the users of the user computing devices 108 may navigate and otherwise interact with one or more documents respective of one or more items. As will be described below, the items with which the user may interact may have corresponding item unique identifiers and / or titles and descriptions in training data, and that training data may be used for next item prediction. In some embodiments, the server 106 may receive user interactions through the interface and provide responsive additional content through the interface. For example, the server may record items with which the user has interacted, and other user actions in the interface, and provide such item interaction and user action data to the user-item interaction prediction and ranking system 104. In response, theuser-item interaction prediction and ranking system 104 may predict a next item with which the user may select or otherwise interact, and the server may output the predicted next item to the user via the interface.

[0012] The training data 102 may include item unique identifiers 110, item title and text descriptions 112, and records of user activity 114. In some embodiments, the item identifiers 110 may be or may include product SKUs or other unique identifiers of the items. The unique identifiers may be listed on the interface and thus may be accessible to the user, or may be proprietary to the operator of the system 104. In some embodiments, the item title and text descriptions 112 may be or may include text from documents specific to or otherwise accessible through a particular electronic user interface, such as a website or mobile application, with each document respective of a particular item. The user activity 114 may be or may include user activity on the electronic user interface, such as user navigations to the items, user selections of the items, etc. For example, in some embodiments, the item titles and descriptions 112 may be respective of products and services offered through an e- commerce website (e.g., where each item title and description is respective of a given product or sendee), and the user activity 114 may include user search queries in an interface through which the item descriptions 112 are accessible, user interactions with the items themselves (e.g., user navigations to, clicks on, or examinations of the items), and / or user interactions with the products and services that are the subjects of those items (e.g., purchases, additions to cart, etc.). In some embodiments, the training data 102 may include associations between particular user activity, such as sequences of user selections of items, with a given sequence considered a “session,” as explained in greater detail below

[0013] The prediction and ranking system 104 may include a processor 116 and a non- transitory, computer-readable memory 118 storing instructions that, when executed by the processor 116, cause the prediction and ranking system 104 to perform one or more of the methods, operations, algorithms, etc. described herein.

[0014] Generally, the prediction and ranking system 104 may train and deploy one or more machine learning models (such as an ID prediction model and / or a text prediction model) to make next item predictions given an input series of item interactions from a user in a session. Consider a set of users and items, denoted as U and V, respectively. Let S denote the set of user-item interaction sequences (or sessions), and s = { si , si, , ..., sn} E S to represent one of the sequences, where si E V is the i-th item interacted with by the same userin session s, and n is the total number of interactions in the session. The number of users and items are represented by IUI and I VI, respectively.

[0015] Each item i in V is associated with text information, such as product descriptions, titles, or taxonomies, denoted by ti = { wi, W2, . . . , ,wc}. Here, each word Wj belongs to a shared vocabulary, and c represents the truncated length of the text. Given an item sequence, the objective of session-based recommendation is to predict the next item in the current sequence. Formally, this involves generating a ranking list ysfor all candidate items, where ys= [ys,i , • • • , ys,ivi] and each ys,z is a score indicating the likelihood of item i being the next interacted item given a session s. Processes for making such predictions are described in detail below.

[0016] The prediction and ranking system may further include one or more functional modules 120, 122, 124, 126, 128 embodied in hardware and / or software. In an embodiment, the functional modules 120, 122, 124, 126, 128 of the prediction and ranking system 104 may be embodied as instructions in the memory 118.

[0017] The prediction and ranking system 104 may include a text model training module 120 that may train a text prediction machine learning model, using the training data 102, to generate a representation (e.g., an embeddings vector) of an item based on a set of text related to the item and / or one or more predicted next items based on an input sequence of items. For example, in some embodiments, the text model training module 120 may train a machine learning model using the item title and description text 112 and user activity 114 training data.

[0018] The prediction and ranking system 104 may include an ID model training module 122 that may train an item identifier machine learning model, using the training data 102, to generate a representation (e.g., an embeddings vector) representation of an item based on a unique item identifier and / or one or more predicted next items based on an input sequence of items. For example, in some embodiments, the ID model training module 122 may train a machine learning model using the item identifiers 110 and user activity 114 training data.

[0019] The prediction and ranking system 104 may include a text prediction model 124 and an ID prediction model 126. The text prediction model 124 may be trained by the text model training module 120 and may generate, based on input text related to an item, a representation of the item, such as an embeddings vector, and / or one or more predicted next items based on an input sequence of items (i.e., the text related to those items). The text prediction model 124 may be or may include, for example, a sentence encoder, such as aBERT model, that receives a series of input item identifiers and generates an embeddings representation of that series of inputs. The text embeddings model 124 may also generate a respective embeddings representation of each individual input (that is, each item) in the series.

[0020] The text embeddings model 124 may be designed to extract textual information from items. For item i, represented by a text sequence ti = { wi,W2, ...,wc}, the text embeddings model 124 processes the input sentence to generate token embeddings and output a comprehensive embedding for the entire sentence. The text embeddings model 124 may further include a multi-layer perception neural network (MLP) to transform the embeddings from the sentence encoder (e.g., BERT model) into a d-dimensional matrix according to equation (1) below:Hi = MLP(SBERT(wl,w2, ...,wc )) (Eq. 1) where H E RIVIxd and each row Hi corresponds to the text embedding of item i.

[0021] The ID embeddings model 126 may be trained by the ID model training module 122 and may generate, based on an input identifier for an item, a representation of the item, such as an embeddings vector. The ID embeddings model 126 may be or may include, for example, an ID embedding matrix X E RIVIxd , where d is the size of the embedding. Each row, Xi, corresponds to the ID embedding of item i. Notably, this matrix is a learnable parameter — specifically, the assignment an item ID to a particular row in the matrix — and those assignments may be updated during training.

[0022] Either or both of the text prediction model 124 and the ID prediction model 126 may individually output an embeddings vector for a predicted next item, which embeddings vector may be compared outside the model to embeddings vectors respective of items to determine the next item predicted by the model. Alternatively, either or both of the text prediction model 124 and the ID prediction model 126 may directly generate a ranked list of one or more predicted next items.

[0023] The prediction and ranking system 104 may include a model application module 128 that is configured to use the output of the text embeddings model 124 and the ID embeddings model 126 to generate a final prediction for a next item. For example, in some embodiments, the model application module 128 may receive the outputs from the text embeddings model 124 and ID embeddings model 126, apply respective weighting parameters to those outputs, and assign a final ranking or prediction of user-item interactions based on the weighted combination.

[0024] FIG. 2 is a flow chart illustrating an example method 200 of providing next item recommendations in response to user activity. The method 200, or one or more portions of the method 200, may be performed by the system 100, and more particularly by the user-item interaction prediction and ranking system 104, in some embodiments, and thus the method 200 may be computer-implemented.

[0025] The method 200 may include, at block 202, training an item ID prediction model and an item text prediction model to predict a next item given an input series of items. Block 202 may include, for example, the training aspects of the method 300 of FIG. 3 described below.

[0026] The method 200 may further include, at block 204, receiving a series of item interactions from a user. The item interactions may be, for example, a user browsing items on a website, mobile application, or other user interface. A user may interact with an item by navigating to a dedicated page for that item, by hovering a cursor over an icon for the item, or otherwise perceptibly interacting with an image of the item, text listing of the item, etc.

[0027] The method 200 may further include, at block 206, inputting the received series of item interactions to the trained predictions models. Block 206 may include, for example, determining, for each item in the sequence, a unique identifier respective of the item and inputting the unique identifier into the trained ID prediction model. Block 206 may further include, for example, determining, for each item in the sequence, text respective of the item and inputting the text into the trained text prediction model. The text may be or may include, for example, an item description accessible to the user. The item identifier may be different from the item text, in embodiments.

[0028] The method 200 may further include, at block 208, applying the trained prediction models to generate one or more next item predictions. For example, block 208 may include receiving the output of the ID prediction model and the output of the text prediction model and combining the outputs to generate a final one or more next item predictions. Combining the outputs may include, for example, applying weighting parameters to those outputs to generate a combined recommendation. For example, combining the outputs may include determining a weighted average of the predictions output by the ID prediction model and the text prediction model.

[0029] The method 200 may further include, at block 210, outputting the one or more next item predictions to the user through the interface in response to receiving the item interaction series from the user at block 204. The output may be in a recommended item portion of the interface (e.g., “You May Also Be Interested In . . .”), in response to a search query by theuser, in a separate communication to the user (e.g., an automatically transmitted email, text message, in-app message, etc.), or in some other interface portion or communication.

[0030] In some embodiments, blocks 206, 208, 210 may be performed for each of a plurality of item interactions by a user in a browsing session.

[0031] FIG. 3 is a flow chart illustrating an example method 300 of training a text-based input machine learning model, training an identifier-based input machine learning model, and applying the two models to provide a refined hybrid machine learning model to predict useritem interactions. The method 300, or one or more portions of the method 300, may be performed by the system 100, and more particularly by the user-item interaction prediction and ranking system 104, in some embodiments, and thus the method 300 may be computer- implemented.

[0032] The method 300 may include, at block 302A, training an ID prediction model (e.g., model 126) to predict user- item interactions using an initial training data set including initial user activity. The training data may include a plurality of series of user-item interactions. Each series may be input to the ID prediction model to generate an embeddings vector representation of each item and one or more next item predictions. For example, the ID prediction model may include an embeddings matrix, and training may include altering, at each training round, the assignment of item IDs to embeddings vectors.

[0033] Training at block 302A may further include generating prediction scores respective of the generated item ID embeddings. In the context of session-based recommendation, the objective is to predict the next item in a sequence of items interacted with the same user, denoted as s = { si, si, Sn}. Training the model may include generating a prediction score, ysj, for each candidate item J. These scores are then used to rank all candidate items, with the top-ranked item predicted as the next item.

[0034] The scoring process may begin with obtaining the session embedding qsE Rd for session s, which encodes the user’s interaction behavior. The relevance between the session embedding qsand each candidate item’s embedding is calculated and used as the score for that item. Each item’s embedding may be generated at each training round and used as the candidate item embedding.

[0035] The session embedding may be determined according to a function g, shown in equation (2) below:where X is the row of the item embedding matrix (i.e., the item embedding).

[0036] The function g may be, for example, the mean function, which calculates the mean of the item embeddings.

[0037] Alternatively, the function g can be a transformer architecture employed to capture item-item transition patterns. When using a transformer, the embedding of the last item in the sequence may be used as the session embedding, as it encapsulates information from all items in the sequence.

[0038] In some embodiments, once a session embedding is determined, the relevance between the session and each candidate item may be determined using vector multiplication according to equation (3) below: y% = Xl <l, (Eq. 3)

[0039] ys‘Di may be used as the score for candidate item i for session s.

[0040] Candidate item embeddings vectors may have been generated by the current version of the ID prediction model, in some embodiments.

[0041] The method 300 may further include, at block 304A, generating a set of hard negative samples using the trained ID prediction model. Block 304A may include ranking the prediction scores generated at block 302A in descending order according to equation (4) below:where r / Ddenotes the sequence of ID indices corresponding to the sorted scores. Items ranked from k> to k represented as r / D[ki: k \, may be designated as the hard negative samples for training the text prediction model, fci and k-_ may be selected as two values at the end of the range of the sorted scores, in some embodiments. That is, the lowest-scoring items, i.e., the items least likely to be selected next by a user, according to the ID prediction model, may be selected as the hard negative samples. Any desired quantity of hard negatives may be selected.

[0042] In some embodiments, block 304A may further include generating a set of hard positive samples using the trained ID prediction model. The positive samples may be defined according to equation (4) above, with the items most likely to be selected next by a user, according to the ID prediction model, defined as the hard positive samples. The hard positive samples may be non-overlapping with the hard negative samples (i.e., p < k ), in some embodiments.

[0043] The method 300 may further include, at block 306A, receiving hard negative samples generated by the text prediction model. The received hard negative samples mayhave been generated by the text prediction model as described below with respect to block 304B. In some embodiments, block 306A may include receiving hard positive samples generated by the text prediction model as described below with respect to block 304B.

[0044] The method 300 may further include, at block 308A, retraining the ID prediction model using hard negative samples from the text prediction model. The hard negative samples may be used to define the loss function, as described below. Such retraining enables the ID prediction model to learn from the patterns identified by the text prediction model.

[0045] For a given session s with st as the target item, cross entropy may be used as the loss function LIDto train the ID prediction model according to equation (5) below:where f is the Softmax function applied over the target item st and the hard negative samples in rstext[ h k2].

[0046] In some embodiments, block 308A may further include retraining the ID prediction model using hard positive samples from the text prediction model. Where hard positive samples are used along with hard negative samples, the loss function L1may be defined according to equation (6) below:where / ? is a scaling parameter that can be adjusted to scale the relative significance of the hard positive samples and hard negative samples. The value of / ? can be determined empirically.

[0047] Accordingly, retraining at block 308A may include simultaneous training with respect to hard negatives and hard positives generated by the text prediction model.

[0048] Training at block 302 A may include the use of equation (4) or equation (5), with the hard negatives and / or hard positives randomly selected, rather than provided by the text prediction model, in some embodiments.

[0049] Separately from training the ID prediction model, the method 300 may include training the text prediction model.

[0050] The method 300 may include, at block 302B, training a text prediction model (e.g., model 124) to predict user- item interactions using an initial training data set including initial user activity. The training data may include a series of user-item interactions. Each series may be input to the text prediction model to generate an embeddings vector representation of each item. For example, the text prediction model may be a sentence encoder, such as a BERT model, that receives a series of input item identifiers and generates an embeddingsrepresentation of that series of inputs. Training at block 302B may include adjusting the weights of the model at each round of training.

[0051] Training at block 302B may further include generating prediction scores respective of the generated item text embeddings. Training the model may include generating a prediction score, ys, for each candidate item j. These scores are then used to rank all candidate items, with the top-ranked item predicted as the next item.

[0052] The scoring process may begin with obtaining the session embedding qsE Rd for session s, which encodes the user’s interaction behavior. The relevance between the session embedding qsand each candidate item’s embedding is calculated and used as the score for that item. Each item’s embedding may be generated at each training round and used as the candidate item embedding.

[0053] The session embedding may be determined according to a function g, shown in equation (7) below: qs= #(Hsi, HS2, . . ., Hsn) (Eq. 7) where H is the text embedding.

[0054] The function g may be, for example, the mean function, which calculates the mean of the item embeddings.

[0055] Alternatively, the function g can be a transformer architecture employed to capture item-item transition patterns. When using a transformer, the embedding of the last item in the sequence may be used as the session embedding, as it encapsulates information from all items in the sequence.

[0056] In some embodiments, once a session embedding is determined, the relevance between the session and each candidate item may be determined using vector multiplication according to equation (8) below:

[0057] may be used as the score for candidate item i for session s.

[0058] Candidate item embeddings vectors may have been generated by the current version of the text prediction model, in some embodiments.

[0059] The method 300 may further include, at block 304B, generating a set of hard negative samples using the trained text prediction model. Block 304B may include ranking the prediction scores generated at block 302B in descending order according to equation (9) below: rstext= argsort(xt, y^f , . . .. ys^f ) (Eq. 9)where r^extdenotes the sequence of text embeddings indices corresponding to the sorted scores. Items ranked from / ci to ki. represented as rgext[k ki], may be designated as the hard negative samples for training the ID prediction model, ki and ki may be selected as two values at the end of the range of the sorted scores. That is, the lowest-scoring items, i.e., the items least likely to be selected next by a user, according to the text prediction model, may be selected as the hard negative samples.

[0060] In some embodiments, block 304B may further include generating a set of hard positive samples using the trained text prediction model. The positive samples may be defined according to equation (9) above, with the items most likely to be selected next by a user, rstext[l:p] according to the text prediction model, defined as the hard positive samples.

[0061] The method 300 may further include, at block 306B, receiving hard negative samples generated by the ID prediction model. The hard negative samples may have been generated by the ID prediction model as described above with respect to block 304A. In some embodiments, block 306B may include receiving hard positive samples generated by the ID prediction model as described above with respect to block 304A.

[0062] The method 300 may further include, at block 308B, retraining the text prediction model using hard negative samples from the ID prediction model. The hard negative samples may be used to define the loss function, as described below. Such training enables the text prediction model to learn from the patterns identified by the ID prediction model.

[0063] For a given session s with st as the target item, cross entropy may be used as the loss function Ltextto train the text prediction model according to equation (10) below:where f is the Softmax function applied over the target item st and the negative samples in rDk -. k2].

[0064] In some embodiments, block 308B may further include retraining the text prediction model using hard positive samples from the ID prediction model. Where hard positive samples are used along with hard negative samples, the loss functionmay be defined according to equation (11) below:where / ? is a scaling parameter that can be adjusted to scale the relative significance of the hard positive samples and hard negative samples. The value of ? can be determined empirically.

[0065] Accordingly, retraining at block 308B may include simultaneous training with respect to hard negatives and hard positives generated by the ID prediction model.

[0066] Training at block 302B may include the use of equation (10) or equation (11), with the hard negatives and / or hard positives randomly selected, rather than provided by the ID prediction model, in some embodiments.

[0067] The training process described above may be considered training the ID prediction model and the text prediction model independent of each other because the loss functions of each model training only depend on the accuracy of one model, and not the other. That is, although hard positives and hard negatives generated by the opposite model are treated as ground truth for training purposes, the loss function solution does not depend on the accuracy of the other model, and thus the training of each model may be considered independent of the training of the other model. This is in contrast to some hybrid approaches in which such models are trained together, with the accuracy of both models to be optimized through a single loss function.

[0068] In some embodiments, the method 300 may include repeatedly (e.g., through multiple iterative rounds) retraining of the ID prediction model and the text prediction model, with one model retrained, then used to generate hard negative examples and / or hard positive examples, which are then used to retrain the other model, which is then used to generate further hard negative examples and / or hard positive examples, which are then used to further retrain the first model, and so on.

[0069] Once training of the models is completed, the models may be deployed at blocks 310A, 310B.

[0070] The method may further include, at block 312, receiving real-time user-item interactions and outputting, in response, a predicted one or more next items to the user. Block 312 may be similar to blocks 204-210 of method 200, for example.

[0071] FIG. 4 is a diagrammatic view of an illustrative computing system that includes a computing system environment 400, such as a desktop computer, laptop, smartphone, tablet, or any other such device having the ability to execute instructions, such as those stored within a non-transient, computer-readable medium. Furthermore, while described and illustrated in the context of a single computing system 400, those skilled in the art will also appreciate that the various tasks described hereinafter may be practiced in a distributed environment having multiple computing systems 400 linked via a local or wide-area network in which the executable instructions may be associated with and / or executed by one or more of multiple computing systems 400. The computing system environment 400, or one or more portions ofthe computing system environment 400, may comprise prediction and ranking system 104 of FIG. 1, in some embodiments.

[0072] Computing system environment 400 may include at least one processing unit 402 and at least one memory 404, which may be linked via a bus 406. Depending on the exact configuration and type of computing system environment, memory 404 may be volatile (such as RAM 410), non-volatile (such as ROM 408, flash memory, etc.) or some combination of the two. Computing system environment 400 may have additional features and / or functionality. For example, computing system environment 400 may also include additional storage (removable and / or non-removable) including, but not limited to, magnetic or optical disks, tape drives and / or flash drives. Such additional memory devices may be made accessible to the computing system environment 400 by means of, for example, a hard disk drive interface 412, a magnetic disk drive interface 414, and / or an optical disk drive interface 416. As will be understood, these devices, which would be linked to the system bus 406, respectively, allow for reading from and writing to a hard disk 418, reading from or writing to a removable magnetic disk 420, and / or for reading from or writing to a removable optical disk 422, such as a CD / DVD ROM or other optical media. The drive interfaces and their associated computer-readable media allow for the nonvolatile storage of computer readable instructions, data structures, program modules and other data for the computing system environment 400. Those skilled in the art will further appreciate that other types of computer readable media that can store data may be used for this same purpose. Examples of such media devices include, but are not limited to, magnetic cassettes, flash memory cards, digital videodisks, Bernoulli cartridges, random access memories, nano-drives, memory sticks, other read / write and / or read-only memories and / or any other method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Any such computer storage media may be part of computing system environment 400.

[0073] A number of program modules may be stored in one or more of the memory / media devices. For example, a basic input / output system (BIOS) 424, containing the basic routines that help to transfer information between elements within the computing system environment 400, such as during start-up, may be stored in ROM 408. Similarly, RAM 410, hard drive 418, and / or peripheral memory devices may be used to store computer executable instructions comprising an operating system 426, one or more applications programs 428 (such as one or more applications that execute the methods and processes of this disclosure), other program modules 430, and / or program data 432. Still further, computer-executableinstructions may be downloaded to the computing environment 400 as needed, for example, via a network connection.

[0074] An end-user may enter commands and information into the computing system environment 400 through input devices such as a keyboard 434 and / or a pointing device 436. While not illustrated, other input devices may include a microphone, a joystick, a game pad, a scanner, etc. These and other input devices would typically be connected to the processing unit 402 by means of a peripheral interface 438 which, in turn, would be coupled to bus 406. Input devices may be directly or indirectly connected to processor 402 via interfaces such as, for example, a parallel port, game port, firewire, or a universal serial bus (USB). To view information from the computing system environment 400, a monitor 440 or other type of display device may also be connected to bus 406 via an interface, such as via video adapter 442. In addition to the monitor 440, the computing system environment 400 may also include other peripheral output devices, not shown, such as speakers and printers.

[0075] The computing system environment 400 may also utilize logical connections to one or more computing system environments. Communications between the computing system environment 400 and the remote computing system environment may be exchanged via a further processing device, such a network router 448, that is responsible for network routing. Communications with the network router 448 may be performed via a network interface component 444. Thus, within such a networked environment, e.g., the Internet, World Wide Web, LAN, or other like type of wired or wireless network, it will be appreciated that program modules depicted relative to the computing system environment 400, or portions thereof, may be stored in the memory storage device(s) of the computing system environment 400.

[0076] The computing system environment 400 may also include localization hardware 446 for determining a location of the computing system environment 400. In embodiments, the localization hardware 446 may include, for example only, a GPS antenna, an RFID chip or reader, a WiFi antenna, or other computing hardware that may be used to capture or transmit signals that may be used to determine the location of the computing system environment 400.

[0077] In a first aspect of the present disclosure, a computer-implemented method for operating a user interface is provided. The method includes training an item prediction model including an item identifier prediction model and an item text prediction model by training an item identifier prediction model according to an initial training data set, the initial training data set including a set of user-item interaction sequences, to predict a next itemgiven an input series of item identifiers, training an item text prediction model according to the initial training data set to predict a next item given an input series of item text, generating, using one of the trained item identifier prediction model or the trained item text prediction model, a set of hard negative examples, and retraining the other of the trained item identifier prediction model or the trained item text prediction model using the hard negative examples. The method includes deploying the item prediction model to provide next item predictions to users in response to item interactions by the users.

[0078] In an embodiment of the first aspect, the method includes receiving, from a user, a series of item interactions through the user interface, inputting the series of item interactions to the deployed item prediction model to generate a predicted next item, and outputting, in response to receiving the series of item interactions, the predicted next item to the user through the user interface.

[0079] In an embodiment of the first aspect, the generating and retraining include generating the set of hard negative examples includes generating the hard negative examples with the trained item text prediction model and retraining the item identifier prediction model using the hard negative examples, and the training further includes generating a second set of hard negative examples using the retrained item identifier prediction model, and retraining the trained item text prediction model using the second set of hard negative examples. In a further embodiment of the first aspect, the training further includes repeatedly generating first hard negative examples with one of the retrained trained item identifier prediction model or the retrained item text prediction model, further retraining the other of the retrained trained item identifier prediction model or the retrained item text prediction model using the first hard negative examples, generating second hard negative examples with the other of the retrained trained item identifier prediction model or the retrained item text prediction model, and further retraining the one of the retrained trained item identifier prediction model or the retrained item text prediction model using the second hard negative examples.

[0080] In an embodiment of the first aspect, the generating and retraining include generating the set of hard negative examples includes generating the hard negative examples with the trained item identifier prediction model and retraining the item text prediction model using the hard negative examples, and the training further includes generating a second set of hard negative examples using the retrained item text prediction model, and retraining the trained item identifier prediction model using the second set of hard negative examples. In a further embodiment of the first aspect, the training further includes repeatedly generating first hard negative examples with one of the retrained trained item identifier prediction model orthe retrained item text prediction model, further retraining the other of the retrained trained item identifier prediction model or the retrained item text prediction model using the first hard negative examples, generating second hard negative examples with the other of the retrained trained item identifier prediction model or the retrained item text prediction model, and further retraining the one of the retrained item identifier prediction model or the retrained item text prediction model using the second hard negative examples.

[0081] In an embodiment of the first aspect, training the item identifier prediction model includes application of a first loss function, training the item text prediction model includes application of a second loss function, and the first loss function is separate from the second loss function.

[0082] In an embodiment of the first aspect, the method further includes generating, using the one of the trained item identifier prediction model or the trained item text prediction model, a set of hard positive examples, wherein the retraining further uses the hard positive examples.

[0083] In an embodiment of the first aspect, deploying the item prediction model to provide next item predictions to users in response to item interactions by the users includes determining a weighted combination of the respective outputs of the item identifier prediction model and the item text prediction model to determine final next item predictions.

[0084] In a second aspect of the present disclosure, a system for operating a user interface is provided. The system includes a processor and a non-transitory, computer-readable memory storing instructions that, when executed by the processor, cause the system to perform operations including training an item prediction model including an item identifier prediction model and an item text prediction model by training an item identifier prediction model according to an initial training data set, the initial training data set including a set of user-item interaction sequences, to predict a next item given an input series of item identifiers, training an item text prediction model according to the initial training data set to predict a next item given an input series of item text, generating, using one of the trained item identifier prediction model or the trained item text prediction model, a set of hard negative examples, and retraining the other of the trained item identifier prediction model or the trained item text prediction model using the hard negative examples. The operations further include deploying the item prediction model to provide next item predictions to users in response to item interactions by the users.

[0085] In an embodiment of the second aspect, the operations further include receiving, from a user, a series of item interactions through the user interface, inputting the series ofitem interactions to the deployed item prediction model to generate a predicted next item, and outputting, in response to receiving the series of item interactions, the predicted next item to the user through the user interface.

[0086] In an embodiment of the second aspect, the generating and retraining include generating the set of hard negative examples includes generating the hard negative examples with the trained item text prediction model and retraining the item identifier prediction model using the hard negative examples, and the training further includes generating a second set of hard negative examples using the retrained item identifier prediction model, and retraining the trained item text prediction model using the second set of hard negative examples. In a further embodiment of the second aspect, the training further includes repeatedly generating first hard negative examples with one of the retrained trained item identifier prediction model or the retrained item text prediction model, further retraining the other of the retrained trained item identifier prediction model or the retrained item text prediction model using the first hard negative examples, generating second hard negative examples with the other of the retrained trained item identifier prediction model or the retrained item text prediction model, and further retraining the one of the retrained trained item identifier prediction model or the retrained item text prediction model using the second hard negative examples.

[0087] In an embodiment of the second aspect, the generating and retraining include generating the set of hard negative examples includes generating the hard negative examples with the trained item identifier prediction model and retraining the item text prediction model using the hard negative examples, and the training further includes generating a second set of hard negative examples using the retrained item text prediction model, and retraining the trained item identifier prediction model using the second set of hard negative examples. In a further embodiment of the second aspect, the training further includes repeatedly generating first hard negative examples with one of the retrained trained item identifier prediction model or the retrained item text prediction model, further retraining the other of the retrained trained item identifier prediction model or the retrained item text prediction model using the first hard negative examples, generating second hard negative examples with the other of the retrained trained item identifier prediction model or the retrained item text prediction model, and further retraining the one of the retrained item identifier prediction model or the retrained item text prediction model using the second hard negative examples.

[0088] In an embodiment of the second aspect, training the item identifier prediction model includes application of a first loss function, training the item text prediction modelincludes application of a second loss function, and the first loss function is separate from the second loss function.

[0089] In an embodiment of the second aspect, the operations further include generating, using the one of the trained item identifier prediction model or the trained item text prediction model, a set of hard positive examples, wherein the retraining further uses the hard positive examples.

[0090] In an embodiment of the second aspect, deploying the item prediction model to provide next item predictions to users in response to item interactions by the users includes determining a weighted combination of the respective outputs of the item identifier prediction model and the item text prediction model to determine final next item predictions.

[0091] In a third aspect of the present disclosure, a computer-implemented method for operating a user interface is provided. The method includes training an item prediction model including an item identifier prediction model and an item text prediction model by initially training an item identifier prediction model to predict a next item given an input series of item identifiers, initially training an item text prediction model to predict a next item given an input series of item text, generating, using the trained item identifier prediction model, a first set of hard negative examples, generating, using the trained item text prediction model, a second set of hard negative examples, retraining the item identifier prediction model using the second set of hard negative examples, and retraining the item text prediction model using the first set of hard negative examples, receiving, from a user, a series of item interactions through the user interface, inputting the series of item interactions to the retrained item identifier prediction model and the retrained item text prediction model to generate a predicted next item, and outputting, in response to receiving the series of item interactions, the predicted next item to the user through the user interface.

[0092] In an embodiment of the third aspect, inputting the series of item interactions includes inputting a respective unique identifier for each item in the series to the item identifier prediction model, and inputting a respective text description of each item in the series to the item text prediction model.

[0093] While this disclosure has described certain embodiments, it will be understood that the claims are not intended to be limited to these embodiments except as explicitly recited in the claims. On the contrary, the instant disclosure is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the disclosure. Furthermore, in the detailed description of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the disclosedembodiments. However, it will be obvious to one of ordinary skill in the art that systems and methods consistent with this disclosure may be practiced without these specific details. In other instances, well known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure various aspects of the present disclosure.

[0094] Some portions of the detailed descriptions of this disclosure have been presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer or digital system memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. A procedure, logic block, process, etc., is herein, and generally, conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these physical manipulations take the form of electrical or magnetic data capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system or similar electronic computing device. For reasons of convenience, and with reference to common usage, such data is referred to as bits, values, elements, symbols, characters, terms, numbers, or the like, with reference to various embodiments of the present invention.

[0095] It should be borne in mind, however, that these terms are to be interpreted as referencing physical manipulations and quantities and are merely convenient labels that should be interpreted further in view of terms commonly used in the art. Unless specifically stated otherwise, as apparent from the discussion herein, it is understood that throughout discussions of the present embodiment, discussions utilizing terms such as “determining” or “outputting” or “transmitting” or “recording” or “locating” or “storing” or “displaying” or “receiving” or “recognizing” or “utilizing” or “generating” or “providing” or “accessing” or “checking” or “notifying” or “delivering” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data. The data is represented as physical (electronic) quantities within the computer system’s registers and memories and is transformed into other data similarly represented as physical quantities within the computer system memories or registers, or other such information storage, transmission, or display devices as described herein or otherwise understood to one of ordinary skill in the art.

Claims

ClaimsWhat is claimed is:

1. A computer-implemented method for operating a user interface, the method comprising: training an item prediction model comprising an item identifier prediction model and an item text prediction model by: training an item identifier prediction model according to an initial training data set, the initial training data set comprising a set of user-item interaction sequences, to predict a next item given an input series of item identifiers; training an item text prediction model according to the initial training data set to predict a next item given an input series of item text; generating, using one of the trained item identifier prediction model or the trained item text prediction model, a set of hard negative examples; and retraining the other of the trained item identifier prediction model or the trained item text prediction model using the hard negative examples; deploying the item prediction model to provide next item predictions to users in response to item interactions by the users.

2. The computer-implemented method of claim 1, further comprising: receiving, from a user, a series of item interactions through the user interface; inputting the series of item interactions to the deployed item prediction model to generate a predicted next item; and outputting, in response to receiving the series of item interactions, the predicted next item to the user through the user interface.

3. The method of claim 1 , wherein: the generating and retraining comprise generating the set of hard negative examples comprises generating the hard negative examples with the trained item text prediction model and retraining the item identifier prediction model using the hard negative examples; and the training further comprises:generating a second set of hard negative examples using the retrained item identifier prediction model; and retraining the trained item text prediction model using the second set of hard negative examples.

4. The method of claim 3, wherein the training further comprises repeatedly: generating first hard negative examples with one of the retrained trained item identifier prediction model or the retrained item text prediction model; further retraining the other of the retrained trained item identifier prediction model or the retrained item text prediction model using the first hard negative examples; generating second hard negative examples with the other of the retrained trained item identifier prediction model or the retrained item text prediction model; and further retraining the one of the retrained trained item identifier prediction model or the retrained item text prediction model using the second hard negative examples.

5. The method of claim 1 , wherein: the generating and retraining comprise generating the set of hard negative examples comprises generating the hard negative examples with the trained item identifier prediction model and retraining the item text prediction model using the hard negative examples; and the training further comprises: generating a second set of hard negative examples using the retrained item text prediction model; and retraining the trained item identifier prediction model using the second set of hard negative examples.

6. The method of claim 5, wherein the training further comprises repeatedly: generating first hard negative examples with one of the retrained trained item identifier prediction model or the retrained item text prediction model; further retraining the other of the retrained trained item identifier prediction model or the retrained item text prediction model using the first hard negative examples;generating second hard negative examples with the other of the retrained trained item identifier prediction model or the retrained item text prediction model; and further retraining the one of the retrained item identifier prediction model or the retrained item text prediction model using the second hard negative examples.

7. The method of claim 1, wherein training the item identifier prediction model comprises application of a first loss function, training the item text prediction model comprises application of a second loss function, and the first loss function is separate from the second loss function.

8. The method of claim 1, further comprising generating, using the one of the trained item identifier prediction model or the trained item text prediction model, a set of hard positive examples; wherein the retraining further uses the hard positive examples.

9. The method of claim 1, wherein deploying the item prediction model to provide next item predictions to users in response to item interactions by the users comprises: determining a weighted combination of the respective outputs of the item identifier prediction model and the item text prediction model to determine final next item predictions.

10. A system for operating a user interface, the system comprising: a processor; and a non-transitory, computer-readable memory storing instructions that, when executed by the processor, cause the system to perform operations comprising: training an item prediction model comprising an item identifier prediction model and an item text prediction model by: training an item identifier prediction model according to an initial training data set, the initial training data set comprising a set of user-item interaction sequences, to predict a next item given an input series of item identifiers; training an item text prediction model according to the initial training data set to predict a next item given an input series of item text;generating, using one of the trained item identifier prediction model or the trained item text prediction model, a set of hard negative examples; and retraining the other of the trained item identifier prediction model or the trained item text prediction model using the hard negative examples; and deploying the item prediction model to provide next item predictions to users in response to item interactions by the users.

11. The system of claim 10, the operations further comprising: receiving, from a user, a series of item interactions through the user interface; inputting the series of item interactions to the deployed item prediction model to generate a predicted next item; and outputting, in response to receiving the series of item interactions, the predicted next item to the user through the user interface.

12. The system of claim 10, wherein: the generating and retraining comprise generating the set of hard negative examples comprises generating the hard negative examples with the trained item text prediction model and retraining the item identifier prediction model using the hard negative examples; and the training further comprises: generating a second set of hard negative examples using the retrained item identifier prediction model; and retraining the trained item text prediction model using the second set of hard negative examples.

13. The system of claim 12, wherein the training further comprises repeatedly: generating first hard negative examples with one of the retrained trained item identifier prediction model or the retrained item text prediction model; further retraining the other of the retrained trained item identifier prediction model or the retrained item text prediction model using the first hard negative examples;generating second hard negative examples with the other of the retrained trained item identifier prediction model or the retrained item text prediction model; and further retraining the one of the retrained trained item identifier prediction model or the retrained item text prediction model using the second hard negative examples.

14. The system of claim 10, wherein: the generating and retraining comprise generating the set of hard negative examples comprises generating the hard negative examples with the trained item identifier prediction model and retraining the item text prediction model using the hard negative examples; and the training further comprises: generating a second set of hard negative examples using the retrained item text prediction model; and retraining the trained item identifier prediction model using the second set of hard negative examples.

15. The system of claim 14, wherein the training further comprises repeatedly: generating first hard negative examples with one of the retrained trained item identifier prediction model or the retrained item text prediction model; further retraining the other of the retrained trained item identifier prediction model or the retrained item text prediction model using the first hard negative examples; generating second hard negative examples with the other of the retrained trained item identifier prediction model or the retrained item text prediction model; and further retraining the one of the retrained item identifier prediction model or the retrained item text prediction model using the second hard negative examples.

16. The system of claim 10, wherein training the item identifier prediction model comprises application of a first loss function, training the item text prediction model comprises application of a second loss function, and the first loss function is separate from the second loss function.

17. The system of claim 10, wherein the operations further comprise generating, using the one of the trained item identifier prediction model or the trained item text prediction model, a set of hard positive examples;wherein the retraining further uses the hard positive examples.

18. The system of claim 10, wherein deploying the item prediction model to provide next item predictions to users in response to item interactions by the users comprises: determining a weighted combination of the respective outputs of the item identifier prediction model and the item text prediction model to determine final next item predictions.

19. A computer-implemented method for operating a user interface, the method comprising: training an item prediction model comprising an item identifier prediction model and an item text prediction model by: initially training an item identifier prediction model to predict a next item given an input series of item identifiers; initially training an item text prediction model to predict a next item given an input series of item text; generating, using the trained item identifier prediction model, a first set of hard negative examples; generating, using the trained item text prediction model, a second set of hard negative examples; retraining the item identifier prediction model using the second set of hard negative examples; and retraining the item text prediction model using the first set of hard negative examples; receiving, from a user, a series of item interactions through the user interface; inputting the series of item interactions to the retrained item identifier prediction model and the retrained item text prediction model to generate a predicted next item; and outputting, in response to receiving the series of item interactions, the predicted next item to the user through the user interface.

20. The method of claim 19, wherein inputting the series of item interactions comprises:inputting a respective unique identifier for each item in the series to the item identifier prediction model; and inputting a respective text description of each item in the series to the item text prediction model.

Images