Content item recommendation

Abstract

Described herein are techniques, devices, and systems, among others, for generating one or more trained machine learning models for generating content item recommendations. Described herein are also techniques, devices, and systems for applying a particular user's consumption history to the trained model to generate score data indicating a relevance between each content item work and the consumption history, and using one or more bias factors to modify the score data for generating result data. Further, the techniques, devices, and systems can use the result data, along with received user input, to determine an order in which to present one or more content items to a user. For example, this can include determining which content items to recommend to a user and in what order to do so.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Patent Application 16 / 505,112, filed July 8, 2019, entitled “Content-Item Recommendations,” the entirety of which is incorporated by reference herein. BACKGROUND

[0003] As consumption of content items on electronic devices continues to grow, the number of available content items is also increasing. For example, the number of songs, movies, television shows, and games available for streaming or download has increased dramatically in recent years. While this increase in available content items is generally beneficial to users, it remains difficult to identify content items of interest to individual users. BRIEF DESCRIPTION OF DRAWINGS

[0004] The detailed description is described with reference to the accompanying figures. In these figures, the left-most digit(s) of the reference numbers identify the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical components or features.

[0005] Figure 1 is a diagram illustrating an example environment including a remote computing system configured to train and use machine learning models for recommending content items to different users. In some cases, the remote computing system trains multiple models, and thereafter inputs a user’s consumption history into each model for generating score data indicating a relevance between each of a plurality of content item works and the consumption history. Further, the computing system can generate result data using one or more bias factors, which the computing system can send to a client device of the user. The client device can use the result data, along with different user inputs, to determine an order in which to present content item recommendations to the user.

[0006] Figure 2 is an example process for generating machine learning models, applying them to historical data of a particular user, sending generated result data to a client device of the user, and using the result data and received user inputs to determine an order in which to present content item recommendations to the user.

[0007] Figure 3 is an example process for generating machine learning models, applying them to historical data of a particular user, sending generated result data to a client device of the user, and using the result data and received user inputs to determine an order in which to present content item recommendations to the user.

[0008] Figure 4Exemplary result data that a remote computing system can generate and send to a user’s client device is shown. As shown, the result data can include an M x N matrix of scores that the client device can interpolate in as needed in response to receiving varying user input.

[0009] Figure 5 An exemplary user interface (UI) that a client device can present in response to receiving a request from a user for content item recommendations is shown.

[0010] Figure 6 An exemplary UI that a client device can present in response to a user changing the value of a first parameter (in this example, the popularity of content items) and the value of a second parameter (in this example, the recency of content items) is shown.

[0011] Figure 7 A flowchart of an exemplary process that a remote computing system can use to generate one or more trained machine learning models and to generate result data for use by a client device in providing content item recommendations to a user is shown.

[0012] Figure 8 A flowchart of an exemplary process that a user’s client device can use to determine an order in which to present one or more content items to the user is shown. DETAILED DESCRIPTION

[0013] Described herein are techniques, devices, and systems, among other things, for generating one or more trained machine learning models for generating content item recommendations. Also described herein are techniques, devices, and systems for applying a particular user’s consumption history to that trained model to generate score data indicating a relevance between each content item work and that consumption history, and using one or more bias factors to modify that score data for use in generating result data. Further, the techniques, devices, and systems can use that result data, along with received user input, to determine an order in which to present one or more content items to a user. For example, this can include determining which content items to recommend to a user and in what order to do so.

[0014] The disclosed technology can be implemented, at least in part, by a remote computing system that distributes content items (e.g., video games, movies, television shows, songs, etc.) to client devices of a community of users (e.g., as part of a video game service, a movie service, a song service, etc.). These client devices can individually install a client application that is configured to execute content items received (e.g., downloaded, streamed, etc.) from the remote computing system. In the example of a video game service, a video game platform enables registered users of the community to play video games in the capacity of a “player.” For example, a user can load the client application, log in using a registered user account, select a desired video game, and execute the video game on his / her client machine via the client application. Of course, while some examples are described with reference to video game works, it will be appreciated that the described technology is applicable to any other type of content item, as described above.

[0015] In one example, a computing system, typically remote from the client devices of individual users, generates a plurality of trained models, each model associated with a particular value of a parameter. For example, the computing system can generate a first trained model for recommending content items that have been released within a first time window (e.g., the last six months), a second trained model for recommending content items that have been released in a second time window (e.g., the last year), and so on. As will be described below, these models can be applied to individual user data for generating customized content item recommendations.

[0016] While the computing system can train these models in any suitable manner, in one example, the computing system begins training by accessing respective consumption histories (e.g., game histories, mobile histories, etc.) of a plurality of users associated with a community of the computing system. For example, the computing system can access a game history of a first user of a community of a video game service and can use this history to train a first model. For example, if the first model is used to recommend game titles that have been released in the past six months, the computing system can select one game title from the user’s game history that has been released in the past six months and can indicate to the first model that this selected game title will be the output of the classifier. Thereafter, the computing system can provide the entire game history of the user as input to the first model and the selected game title as output of the first model for training one or more internal layers of the first model. For example, if the first model comprises an artificial neural network, selecting a game title as output and providing information about the user’s game history as input can effectively train one or more internal layers of the neural network. It should be appreciated that the inputs and outputs of the neural network, as well as other types of models that can be used, can include features of corresponding game titles or other content items, such as release data, genre, game type, etc. Further, the inputs to the model can include information associated with the user, such as geographic locations associated with the user, demographic information associated with the user, etc. Additionally, while a few example inputs to the model have been described, it should be appreciated that any other type of input can be provided to the model.

[0017] In some cases, the computing system can consider the amount of time a user has played each game in their history when selecting a game title to use as output for a machine learning model. For example, the computing system can initially determine the amount of time a user has played each individual game title referenced in the user’s game history. This can be determined absolutely (e.g., in terms of time spent playing the game) or relatively (e.g., in terms of time played relative to other users who also played the game title). This information can be applied as a weight to individual game titles or can be passed into the model as input. After applying weights to each game title associated with the game history or inputting this information into the model (possibly filtering out some game titles due to the user playing them very little), the computing system can then select one (or more) game title to use as output for the first model. By doing so, the computing system increases the chance that a game the user has played the most is selected as output for the first model, thus training the internal layers of the model in a way that makes it more likely to select games of interest to the user. However, it should be appreciated that while the above description illustrates one way of training a model, any other number of ways can be employed to train other models for outputting content item recommendations.

[0018] In addition to training the first model, in some cases the computing device can also train a plurality of other models, such as a model for recommending games released within the last year, a model for recommending games released within the last three years, and so forth. In each case, the computing system can access the user’s game history and select game titles that are both indicated in the game history and released within the amount of time associated with the particular model (e.g., within the last year, within the last three years, etc.) as output for that particular model. Moreover, while the above example describes accessing a single game history for training each of a plurality of machine learning models, it should be understood that the computing system can analyze thousands or more game histories for training these models.

[0019] After training the machine learning models, the computing system can then use the models to generate data that can be used by a client device for displaying content item (e.g., video game) recommendations. For example, suppose a particular user sends a request for video game recommendations via a client device. Upon receiving the request, the computing system can identify the user (e.g., via a device or user identifier) and can access the user’s game history. The computing system can then input the game history into each of the plurality of machine learning models along with information about available games associated with each of the models. For example, for the first trained model (which has been trained to weight games released within the last six months more highly), the computing system can input the user’s game history and information about games available for recommendation, for the second trained model (which has been trained to weight games released within the last year more highly), the computing system can input the user’s game history and information about those games available for recommendation, and so forth. That is, while the entire catalog of available games can be input to the first model, the second model, and so forth, the first model can be more likely to recommend games that have been released within the last six months, the second model can be more likely to recommend games released within the last year, and so forth.

[0020] Each trained machine learning model can output score data that indicates, for each of a plurality of game titles, a correlation between the user’s game history and the particular game title. As the reader will appreciate, those games that are highly correlated with the game history will receive a relatively high score, and vice versa. Moreover, if the computing system implements “N” models, the computing system can generate “N” score data. It should also be understood that the “correlation” can not be a statistical correlation, but rather simply indicate a likelihood that a user with the same game history as the user of interest would select to play the game.

[0021] Additionally, the computing system can be configured to bias the score data based on different values of one or more bias factors. The bias factors can include any number of factors, such as the popularity of each game, the cost of each game, the genre of each game, etc. In one example, the bias factors can include the popularity of each game, as determined by the sales volume of the game, the play volume of the game, etc. In some cases, each score data (determined by the respective trained models) can be biased at different levels of popularity, such that the resulting biased data will more closely match that particular level of popularity if desired by the user. That is, the computing system can bias the scores of each game title (and each model) according to “M” target popularity levels ranging from very popular to very unpopular (or very “niche”). As the reader will appreciate, the computed scores of those very popular game titles will be boosted for a “very high” target popularity level, and penalized for a “very low” target popularity level. Conversely, the computed scores of more niche game titles will be penalized for a “very high” target popularity level, and boosted for a “very low” target popularity level.

[0022] Finally, given that the computing system has now computed score data for each of the “N” trained models for a particular user, and has biased each of those scores using the “M” popularity values, the computing system can now generate “M x N” scores for the user. For example, if the computing system utilizes six models and five popularity levels, the computing system will generate a total of thirty scores for any particular user (assuming that each of the six score data generated by the models will be biased five times). Of course, it should be appreciated that the computed score data can have any other number of values, and can be sampled in any other (e.g., non-linear) manner. Additionally, while this example describes inputting the value of a first parameter (in this case, based on release date) prior to training the model, and inputting the value of a second parameter (in this case, based on popularity) after training, any other number of parameters can be used as input to the model and / or as part of the post-processing described with reference to the popularity data. It should also be noted that while the post-processing parameters are described as bias factors, these bias factors represent parameters that can be applied as input to the model or as part of the post-processing.

[0023] In computing this score data in a matrix or other form, the computing system can then send this “result data” to the user’s client device. The user’s client device can then use this result data along with input from the user to determine an order in which to present one or more game titles available to the user. That is, the client device can use the result data and input data from the user to determine which video games to recommend to the user, and how to change that recommendation when the user provides different input data.

[0024] To continue the example above, the client device can enable the user to provide as input a selection of a desired recency of release and a selection of a desired level of popularity. That is, the user can indicate to the client device whether the user prefers to see recently released games or older games as game recommendations. Further, the user can specify whether she wishes to receive recommendations for mainstream (i.e., popular) games or more niche (i.e., less popular) games. While the example below illustrates and describes the user providing this input via one or more sliders, it should be understood that this input can be provided in any other manner.

[0025] Upon receiving input from the user, such as input indicating that the user wishes to see recommendations for games that are generally newer and more niche, the client device can use the result data to determine which games to recommend to the user and in what order. In some cases, the client device can identify from this input which one or more of the computed score data that the client device has received for this user is closest to the user’s input data. For example, in the example where the user specifies relatively new games that are less popular, the client device can identify from the computed score data which scores correspond to: (1) a model that has been trained over this time range, and (2) are biased to increase scores for less popular titles. Upon identifying these scores, the client device can use interpolation to determine scores associated with individual games, and can then present game recommendations to the user based on these scores. For example, the client device can utilize bilinear interpolation or any other form of interpolation to compute scores for game titles, and can order these game titles according to the computed scores, with those having the highest scores being placed at the top of the list. Further, if the user provides different input data, such as by moving one or both of the sliders described above, the client device can again identify the most relevant score data from the matrix or other representation of scores, and can interpolate this score data for reordering of game titles.

[0026] By generating the score data at the remote computing system and performing interpolation of the score data locally at the client device, the techniques greatly improve the smoothness of the process of ordering and reordering content item recommendations at the client device. That is, because the techniques involve sending multiple score data to the client device instead of a single result, and because the client device is configured to perform real-time interpolation of the score data in response to receiving different user inputs, the techniques are able to order and reorder recommendations without requiring additional round-trip requests to the remote computing system. Additionally, as opposed to requiring a call to the system and waiting for a response from the system each time the user provides a different input, the techniques enable the content item recommendations to be reordered without limit at the cost of a single request and response to the remote computing system.

[0027] Furthermore, the techniques described herein provide very granular content recommendations. By enabling the user to specify desired values for a first parameter (e.g., cost, release date, etc.) and a second parameter (e.g., popularity, cost, etc.) and by performing client-side interpolation of these inputs, the techniques provide highly customized content item recommendations.

[0028] It should be appreciated that while some of the techniques are described herein with reference to two parameters (release data and popularity), the techniques can be used with any other number of parameters (e.g., one, five, etc.) and any other parameter values (e.g., cost, genre, etc.). It should also be appreciated that while the recommendations are described as being presented on one or more UIs, the recommendations can be output in any other manner (e.g., auditory, visual, etc.).

[0029] Figure 1 is a diagram illustrating an example environment that includes a client device 102 associated with a display 104, and a remote computing system 106 configured to train and use machine learning models for recommending content items to different users. In some cases, and as described in further detail below, the remote computing system 106 trains multiple models, and thereafter inputs a user’s consumption history into each model for generating score data indicative of a relevance between each of a plurality of content item works and the consumption history. Furthermore, the computing system 106 can generate result data using one or more bias factors, which the computing system can send to the user’s client device 102. The client device 102 can use the result data, along with different user inputs, to determine an order in which to present content item recommendations to the user on the display 104 or otherwise.

[0030] The client devices 102 can be implemented as any suitable type of computing device configured to execute content items, such as video games, movies, songs, and the like. The client devices 102 can include, but are not limited to, personal computers (PCs), desktop computers, laptop computers, mobile phones (e.g., smartphones), tablet computers, portable digital assistants (PDAs), wearable computers (e.g., virtual reality (VR) headsets, augmented reality (AR) headsets, smart glasses, and the like), in-vehicle (e.g., in-car) computers, televisions (smart televisions), set-top boxes (STBs), game consoles, music players, voice-controlled assistants, and / or any similar computing device. The client devices 102 can communicate with a remote computing system 106 (sometimes simply referred to herein as “computing system 106”) over a computer network 108. The computer network 108 can represent and / or include, but is not limited to, the Internet, other types of data and / or voice networks, wired infrastructures (e.g., coaxial cables, fiber optic cables, and the like), wireless infrastructures (e.g., radio frequency (RF), cellular, satellite, and the like), and / or other connection technologies. In some cases, the computing system 106 can be part of a network-accessible computing platform that is maintained and accessible via the computer network 108. Network-accessible computing platforms such as these can be referred to by terms such as “on-demand computing,” “software as a service” (SaaS), “platform computing,” “network-accessible platform,” “cloud services,” “data center,” and the like.

[0031] In some embodiments, the computing system 106 functions as or has access to a video game platform that implements a video game service to distribute (e.g., download, stream, and the like) video games (or any other type of content item) to the client devices 102. In one example, the client devices 102 can each have installed a client application. The installed client application can be a video game client (e.g., game software for playing a video game). The client devices 102 with the installed client application can be configured to download, stream, or otherwise receive programs (e.g., video games and their content) from the computing system 106 over the computer network 108. Any type of content distribution model can be used for this purpose, such as a direct purchase model in which programs (e.g., video games) can be purchased individually for download and execution on the client devices 102, a subscription-based model, a content distribution model in which programs are rented or leased for a period of time, streamed, or otherwise provided to the client devices 102. Thus, the individual client devices 102 can include one or more installed video games that can be executed by loading the client application.

[0032] The client devices 102 can be used to register with and subsequently log in to a video game service. A user can create a user account for this purpose, and specify / set credentials (e.g., a password, a PIN, a biometric ID, etc.) that are bound to the registered user account. When multiple users interact with the video game platform (e.g., by accessing their user / player profiles using the registered user accounts, playing video games on their respective client devices 102, etc.), the client devices 102 send data to the remote computing system 106. For a given client machine 104, the data sent to the remote computing system 106 can include, but is not limited to, user input data, video game data (e.g., game performance statistics uploaded to the remote system), social network messages and related activity, identifiers (IDs) of video games played on the client device 102, etc. This data can be streamed in real-time (or substantially real-time), sent to the remote system 106 at defined intervals, and / or uploaded in response to events (e.g., exiting a video game). As described in further detail below, this data can be used to determine a game history of the user of the client device 102, which can be used to determine a score for recommending game titles to the user.

[0033] For example, Figure 1 The client devices 102 can be used to register with and subsequently log in to a video game service. A user can create a user account for this purpose, and specify / set credentials (e.g., a password, a PIN, a biometric ID, etc.) that are bound to the registered user account. When multiple users interact with the video game platform (e.g., by accessing their user / player profiles using the registered user accounts, playing video games on their respective client devices 102, etc.), the client devices 102 send data to the remote computing system 106. For a given client machine 104, the data sent to the remote computing system 106 can include, but is not limited to, user input data, video game data (e.g., game performance statistics uploaded to the remote system), social network messages and related activity, identifiers (IDs) of video games played on the client device 102, etc. This data can be streamed in real-time (or substantially real-time), sent to the remote system 106 at defined intervals, and / or uploaded in response to events (e.g., exiting a video game). As described in further detail below, this data can be used to determine a game history of the user of the client device 102, which can be used to determine a score for recommending game titles to the user.

[0034] Figure 1The computing system 106 is also shown to include one or more processors 116 (e.g., central processing units (CPUs)) and computer-readable media 118. The computer-readable media 118 can include volatile memory and non-volatile memory implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium which can be used to store the desired information and which can be accessed by the computing devices. The computer-readable media 118 can be implemented as computer-readable storage media (“CRSM”), which can be any available physical media accessible by the processor 116 to execute instructions stored on the computer-readable media 118. In one basic implementation, the CRSM can include random access memory (“RAM”) and flash memory. In other implementations, the CRSM can include, but is not limited to, read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), or any other tangible medium which can be used to store the desired information and which can be accessed by the processor 116.

[0035] As shown, the computer-readable media 118 can store or otherwise have access to a training component 120, a correlation component 122, and a bias component 124. In addition, the media 118 can store popularity data 126 indicative of the popularity of respective content items (e.g., sales volume, play time, etc.), content items 128 available for acquisition at a client device, and history data 130 indicative of respective consumption histories of different users.

[0036] The training component 120 can use the history data 130 to train one or more machine learning models 132(1), 132(2), …, 132(N). For example, the training component 120 can train each of a plurality of machine learning models using a portion of the history data 130 associated with a set of sampled user accounts as training data to obtain trained machine learning models 132(1)-(N).

[0037] The trained machine learning models 132 can represent a single model or a collection of base-level machine learning models, and can be implemented as any type of machine learning model 132. For example, suitable machine learning models 132 for use with the techniques and systems described herein include, but are not limited to, neural networks, tree-based models, support vector machines (SVMs), kernel methods, random forests, splines (e.g., multivariate adaptive regression splines), hidden Markov models (HMMs), Kalman filters (or augmented Kalman filters), Bayesian networks (or Bayesian belief networks), expectation maximization, genetic algorithms, linear regression algorithms, nonlinear regression algorithms, logistic-regression-based classification models, or collections thereof. A “collection” can include a collection of machine learning models 132 whose outputs (predictions) are combined such as by using a weighted average or voting. Individual machine learning models in the collection can differ in their specialized skills, and the collection can operate as a committee of individual machine learning models that is collectively “smarter” than any individual machine learning model in the collection.

[0038] The training data used to train each machine learning model 132 can include various types of data. Generally, training data for machine learning can include two components: features and labels. However, in some embodiments, the training data used to train the machine learning models 132 can not have labels. Accordingly, the machine learning models 216 can be trained using any suitable learning technique such as supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc. The features included in the training data can be represented by a set of features such as in the form of an n-dimensional feature vector that takes quantifiable information about the attributes of the training data. The following is a list of exemplary features that can be included in the training data used to train the machine learning models 132 described herein. However, it should be understood that the following list of features is non-exhaustive, and that the features used in training can include additional features not described herein, and in some cases, include some (but not all) of the features listed herein. Exemplary features included in the training data can include, but are not limited to, a release date of a content item, a genre of a content item, a work of a content item, a length of a content item, a rating of a content item, a user that has consumed the content item, etc. Further, as part of the training process, the training component 120 can set weights for the machine learning. These weights can be applied to the set of features included in the training data as derived from the historical data 130. In some embodiments, the weights set during the training process can be applied to parameters internal to the machine learning model (e.g., weights of neurons in a hidden layer for a neural network). These internal parameters of the machine learning model can or can not be mapped one-to-one with individual input features of the set of features. The weights can indicate the influence of any given feature or parameter on a score output by the trained machine learning model 132.

[0039] In some cases, the training component 120 can train the individual models by accessing a user's personal consumption history, selecting a particular content item indicated by the respective history as an output of the model, inputting the consumption history as an input of the model, and training one or more internal layers of the model. For example, and as described above, the training component can access a first user's consumption history, and can use that history to train the first model 132(1). For example, if the first model 132(1) is used to recommend content item works that have been released within the past six months, the training component 120 can select one game work from the user's history that has been released within the past six months, and can indicate to the first model 132(1) that this selected work will be the output of the model. Thereafter, the training component 120 can provide the user's entire consumption history as an input to the first model 132(1), and can provide the selected content item work as an output of the first model 132(1) for training one or more internal layers of the first model 132(1). For example, if the first model 132(1) comprises an artificial neural network, selecting a content item work as an output and providing information about the user's consumption history as an input can effectively train one or more internal layers of the neural network. It will be appreciated that the inputs and outputs of a neural network, and other types of models that can be used, can include features of corresponding game works or other content items, such as release data, genre, game type, etc.

[0040] In some cases, the training component 120 can consider the amount of time each content item that the user has consumed in their history when selecting a work to use as an output of the machine learning model 132(1). For example, the training component 120 can initially determine the amount of time that the user has consumed each individual work referenced in the user's consumption history. This can be determined absolutely (e.g., in terms of time) or relatively (e.g., in terms of time relative to other users who have also consumed the work). This information can be applied as a weight to the individual content item works. After applying the weight to each work associated with the history or inputting this information to the model (possibly filtering out some works that were consumed very little by the user), the training component 120 can then select one (or more) content item works to use as an output of the first model 132(1). By doing so, the training component 120 increases the chances that the content items that the user consumed the most are selected as an output of the first model 132(1), and thus, trains the internal layers of the model in a manner that makes it more likely that the model will select content items that the user is interested in.

[0041] In addition to training the first model 132(1), in some cases, the training component 120 can also train a number of other models, such as a model 132(2) for recommending games released within the last year, a model 132(3) for recommending games released within the last three years, and so on. In each case, the training component 120 can access the user's consumption history and select works that are both indicated in the history and released within the amount of time associated with the particular model (e.g., within the last year, within the last three years, and so on) as the output of that particular model. Moreover, while the above examples describe accessing a single history for training each of a number of machine learning models, it should be appreciated that the training component 120 can analyze thousands or more histories for training the models 132.

[0042] Meanwhile, the correlation component 122 can be used to generate a correlation score for individual users in response to receiving a request for recommendations from those individual users. For example, as shown, the client device 102 can include one or more processors 134 and a computer-readable medium 136 that can store or otherwise access one or more client applications 138 and an interpolation component 140. In some cases, a user of the client device 102 can interact with the computing system 106 using one of the client applications 138 (e.g., a browser, a dedicated application, and so on), such as to request data for presenting the UI 110. In response to receiving the request, the correlation component can determine an identifier associated with the user of the client device 102 for generating score data that can be used to recommend one or more content items to the user.

[0043] In one example, the user correlation component 122 accesses a consumption history of the user of the client device 102 from the history data 130 after identifying the user. The correlation component can then input the consumption history into each of the "N" trained models 132. The models 132 can each output score data for each of a number of content items associated with a time range corresponding to the individual models 132, the score data indicating a correlation between the content item and the consumption history of the user. As described above, a higher score can indicate a higher correlation. In some cases, each time range associated with a particular model is exclusive, such as in the example of a first model being associated with content items released within the last six months, a second model being associated with content items released between the previous six months and twelve months, and so on. In other cases, the time range is not exclusive, as in the example of a first model being associated with content released within the last six months, a second model being associated with content items released within the last year (including the previous six months), and so on.

[0044] At the same time, the bias component 124 can be used to bias each of the “N” score data generated by the correlation component 122 according to “M” values of a particular parameter. The parameter can include a level of popularity, a cost, a date of publication, and the like. In some cases, the parameter includes a level of popularity indicated by the popularity data 126.

[0045] For example, for each content item associated with each score data output by each model, the bias component can determine whether to boost or penalize the individual score according to “M” different values of popularity. In one example, the bias component 124 boosts or penalizes the score of each content item according to five different values of popularity, from very popular to very unpopular (or, more precisely, very “niche”). After doing so, the bias component can have generated a matrix or other representation of scores, with one particular implementation of reference Figure 3 shown and described.

[0046] After having generated the matrix or other representation of scores, the computing system 106 can send this “result data” back to the client device 102 that initially requested the recommendations. The interpolation component 140 can then determine an order in which to present the recommended content items based at least in part on the received result data and the values of the UI controls 112(1) and 112(2). For example, for any given set of values of these two controls, the interpolation component 140 can identify from the result data one or more score data that are closest and can interpolate the scores of the content items based on the identified score data. The interpolation component 140 can employ bilinear interpolation and / or any other interpolation method. Moreover, as the user provides input via the UI controls 112(1) and / or 112(2), the interpolation component 140 can perform new interpolations and update the UI 110 to indicate new recommendations.

[0047] Figure 2 An exemplary process 200 is shown for generating machine learning models, applying them to historical data of a particular user, sending the generated result data to the user’s client device, and using the result data and received user input to determine an order in which to present content item recommendations to the user. While this process 200 is described with reference to the environment 100 of Figure 1 This process can be applicable to other environments.

[0048] At operation 202, the computing system 106 can train one or more machine learning models using any of the techniques described above. In some cases, operation 202 includes training “N” models, each associated with a particular value of a parameter, such as a particular time window in which a content item was published.

[0049] In one example, the computing system 106 can begin the training operation by accessing respective consumption histories (e.g., game histories, mobile histories, etc.) of a plurality of users associated with a computing system community. For example, the computing system can access a game history of a first user of a video game service community, and can use the history to train a first model. For example, if the first model is used to recommend game titles that have been released in the past six months, the computing system can select one game title from the user’s game history that has been released in the past six months, and can indicate to the first model that the selected game title will be the output of the classifier. That is, the computing system 106 can identify a game title or other content item that has a value for a parameter of the model being trained.

[0050] Thereafter, the computing system 106 can provide the entire game history of the user as input to the first model, and provide the selected game title as output of the first model for training one or more internal layers of the first model. For example, if the first model comprises an artificial neural network, selecting a game title as output and providing information about the user’s game history as input can effectively train one or more internal layers of the neural network. It will be appreciated that the inputs and outputs of a neural network, and other types of models that can be used, can include features of corresponding game titles or other content items, such as release data, genre, game type, etc.

[0051] In some cases, the computing system 106 can consider the amount of time a user has played each game in their history when selecting a game title to use as output for a machine learning model. For example, the computing system can initially determine the amount of time a user has played each individual game title referenced in the user’s game history. This can be determined absolutely (e.g., in terms of time spent playing the game) or relatively (e.g., in terms of time played relative to other users who also played the game title). This information can be applied as a weight to individual game titles or input as information about the model. After applying weights to each game title associated with a game history or inputting this information to the model (possibly filtering out game titles that the user has played very little of), the computing system can then select one (or more) game title(s) to use as output for the first model. By doing so, the computing system increases the chances that a game that the user has played the most is selected as output for the first model, thus training the internal layers of the model in a way that makes it more likely that the model will select games that the user is interested in.

[0052] In addition to training the first model, in some cases the computing device can also train a plurality of other models, each associated with different values of the parameter, such as a model for recommending games released within the last year, a model for recommending games released within the last three years, and so on. In each case, the computing system can access the user's game history or other consumption history, and select game titles that are both indicated in the game history and released within the amount of time associated with the particular model (e.g., within the last year, within the last three years, and so on) as the output of that particular model. Moreover, while the above example describes accessing a single game history for training each of a plurality of machine learning models, it will be appreciated that the computing system can analyze thousands or more game histories for training these models.

[0053] At operation 204, the client device 102 receives a request from a user for one or more content item recommendations. This can include the user navigating to a site provided by the computing system 106 using the client device 102, interacting with a dedicated application generated by the computing system 106, and so on.

[0054] At operation 206, the computing system 106 receives the request, and at operation 208, the computing system generates a set of "N" relevance scores using the "N" trained models. As described above, this can include identifying the consumption history of the requesting user, and inputting the consumption history into each of the "N" trained models to receive the "N" sets of relevance scores.

[0055] At operation 210, the computing system 106 modifies each score within the "N" sets of relevance scores using one or more bias factors. For example, for each bias factor, the computing system 106 can use the values of the "M" bias factors to generate M x N sets of now-biased relevance scores. Of course, if multiple bias factors are used, a greater number of sets can be generated. For example, if two bias factors are used, the computing system 106 can generate M x M x N biased scores. As described above, in some cases, the bias factors can correspond to a level of popularity of each content item. Moreover, it will also be appreciated that the computing system 106 can generate the relevance scores and biased relevance scores non-linearly. Moreover, any number of values can be computed.

[0056] At operation 212, the computing system sends the "result data" to the client device 102, which receives the result data at operation 214. At operation 216, the client device 102 presents one or more content item recommendations to the user based on the received result data. In some cases, the initial recommendations can be based on the current settings of the values of any parameters that can have values that can be set by the user.

[0057] At operation 218, the client device 102 receives input from the user. This can include changing the value of one or more parameters, such as the desired publication data, the desired level of popularity, etc. At operation 220, the client device 102 uses the input data and the result data to interpolate the scores of the content items. That is, the client device 102 uses the result data and the received input to generate an order of recommendations. Finally, at operation 222, the client device 102 can re-present the recommendations, albeit in the newly computed order.

[0058] Figure 3 Exemplary components and their interactions for generating models, biasing resulting scores, applying the models to particular user's historical data, and using that result data and received user input to determine an order in which to present content item recommendations to a user are shown. It should be appreciated that while the following discussion describes the components as being implemented in a particular manner, other implementations are possible. Figure 3 An exemplary set of components for performing the described techniques are shown, although other components can be used in other cases.

[0059] As shown, in this example, some components reside on the server side 302, while other components reside on the client side 104. For example, the components of the server side 302 can reside on the computing system 106, while the components of the client side can reside on the client device 102.

[0060] First, the historical data 130 and one or more parameters 306(1), 306(2),..., 306(N) can be input into the training component 120 for generating respective models 132(1)-(N). That is, the training component 120 can receive an indication of particular values of parameters that the training component 120 is to use to train the models. For example, the parameter 306(1) can indicate a value of "publication date within the last six months," the parameter 306(2) can indicate a value of "publication date within the last year," etc. Of course, in other cases, the parameters can correspond to a price of the content item, a popularity of the content item, etc. Further, the historical data 130 can include a consumption history of a user population as described above.

[0061] As also described above, the training component 120 can input the individual consumption histories into the respective machine learning models to train the models. For example, for the model 132(1) having a parameter value of "release date within the past six months," the training component can input the individual consumption histories as inputs to the model and can specify, as an output of the model 132(1), content item works that have been released within the past six months according to each respective history. The training component 120 can then use this input and output to train one or more internal layers of the model 132(1). The training component 120 can perform this operation for an array of different consumption histories from the historical data 130 and for each of the models 132(1)-(N). Furthermore, the inputs to the models 32 can include information associated with the user, such as a geographic location associated with the user, demographic information associated with the user, and the like. Additionally, while a few exemplary inputs to the models have been described, it should be understood that any other type of input can be provided to the models.

[0062] After the models 132(1)-(N) have been trained, these models can be used to generate content item recommendations for users. Upon receiving a recommendation request from a particular user, the consumption history associated with that user (stored in the historical data 130) can be input into the correlation component 122 along with a library of available content items 128. The correlation component can input this data into each of the models 132(1)-(N), which can output respective correlation scores 308(1), 308(2),..., 308(N). For example, the model 132(1) can output content item recommendations that have been released within the past six months and that are tailored to the user based on the user's consumption history, and so on.

[0063] In some cases, these scores 308(1)-(N) can be input into the bias component 124, which can generate a set of bias scores 310(1), 310(2),..., 310(N) for each of the correlation scores 308(1)-(N). That is, the bias component can select one or more values of a particular bias factor and can either score up or score down the score of each content item within each of the scores 308(1)-(N) based on a corresponding degree to which the content item corresponds to the respective value of the bias factor. For example, the bias factor can include a popularity of the content item, as indicated by the popularity data 126 that can be input into the bias component. As described above, this popularity data 126 can include sales data, consumption times, and the like.

[0064] In one example, the bias component selects "M" values for the bias factor, such as M popularity values from very popular (e.g., having a very high number of sales) to very unpopular or niche (e.g., having a very low number of sales). Thus, the bias component 124 can boost the scores of very popular works and penalize very niche works according to earlier values of the bias factor, and can take the opposite actions according to later values of the bias. In this example, the bias component 124 can output "M x N" or any other number of bias scores, assuming that sampling can occur non-linearly. In some examples, the computing system 106 sends these bias scores 310(1)-(N) to the client device 102 associated with the user that initiated the recommendation request.

[0065] As shown, this set of bias scores 310(1)-(N) can be provided as result data 312 to the client device 102. This result data 312 can be input into the interpolation component 140 along with input data 314 generated in response to user input received at the client device 102. For example, the input data 314 can indicate particular values for the parameters upon which the models 132(1)-(N) are based and / or particular values for the bias factor used to bias the scores. In response to receiving this input data 314, the interpolation component 140 can determine an order 316 in which to present recommendations to the user. For example, upon receiving the input data 314, the interpolation component 140 can identify one or more closest bias scores from the result data and can interpolate between these values if necessary. For example, the interpolation component 140 can perform bilinear interpolation or any other type of interpolation for determining the order 316 in which to present recommendations to the user. Further, upon receiving different input data 314 (e.g., different values for the parameters or the bias factor), the interpolation component 140 can re-interpolate the order 316 using the input data 314 and the result data 312, and can present the recommendations in the new order 316 to the user. Further, and as described above, the interpolation component 140 can use the result data 312 and the input data 314 to re-compute a different order without interacting with the computing system 106, enabling real-time reordering of recommendations on the display of the client device 102.

[0066] Figure 4Exemplary result data 312 that the remote computing system 102 can generate and send to a client device 102 of a user is shown. As shown, in this example, the result data 312 can include an M x N matrix of scores that the client device can interpolate in as needed in response to receiving changed user input. However, it should be understood that the result data can be sampled in any pattern in a parameter space of any number of dimensions. For example, a matrix or other representation of scores can be sampled at different points than the M x N points shown. For example, the sampling can occur non-linearly. Further, the number of sampling points can vary (e.g., there can be more or less than M x N points). Further, more than two parameters and / or bias factors can be used, resulting in a higher dimensional parameter space.

[0067] In this example, the result data 312 corresponds to a parameter of a time range in which a content item was published and a bias factor corresponding to the popularity of the content item. Further, the training component 120 has trained "N" values of this parameter, as indicated by the result data having "N" values of this parameter (time range 1-N). Further, the bias component 124 can bias each score output by the correlation component 122 using "M" values of the bias factor (popularity 1-M). Thus, in this example, the result data 312 includes M x N scores.

[0068] In response to a user providing input that changes the value of a parameter (time range) or a bias factor (popularity), the interpolation component can identify the closest scores and can perform interpolation to calculate an interpolated score that can be used to determine an order in which to recommend content items to the user. For example, assume that a user operating the client device 102 requests to view recommendations for content items that have been published in a time between "time range 1" and "time range 2" and that have a popularity that falls between "popularity 1" and "popularity 2". In response to receiving this input data, the interpolation component 140 can identify the four scores closest to this input, such as "biased score 11", "biased score 12", "biased score 21", and "biased score 22". The interpolation component can then use these scores to perform a bilinear interpolation or any other type of suitable interpolation algorithm to generate an interpolated score. This score can then be used to determine an order in which to present content item recommendations to the user. For example, the content item with the highest score can be presented first, the content item with the second highest score can be presented second, and so on.

[0069] Figure 5An example user interface (UI) 500 that the client device 102 can present in response to receiving a request from a user for content item recommendations is shown. As shown, the UI 500 can present a customized list 114 of recommended content items 114(1), 114(2), 114(3), and 114(4), where the list 114 is determined based on the consumption history of the user with which the UI 500 was generated and the respective values of each of the UI controls 112(1) and 112(2). For example, the UI control 112(1) can include a slider or other type of UI control that enables the user to select a desired level of popularity, while the UI control 112(2) can include a slider or other type of UI control that enables the user to select a desired recency of the recommended content items. In some cases, the UI 500 presented to the user can include default values for each of these UI controls 112(1), such as in the middle of each range, etc. Thus, the scores for the content items and, consequently, the list 114 can be generated based on these default values, which can be changed as discussed below with reference to FIGS. 6-8. Figure 6

[0070] In addition, the UI 500 can include a UI control 502 that the user can request to filter out games having a user-specified tag, and a UI control 504 that the user can request to filter out games not having a user-specified tag. In this example, for instance, the user can enter or select a tag in either of the controls 502 or 504 via a drop-down menu. In response to receiving a tag in the UI control 502, the client device 102 can change the list 114 by removing any content items associated with the specified tag. Meanwhile, in response to receiving a tag in the UI control 504, the client device 102 can change the list 114 by removing any content items not associated with the specified tag.

[0071] Figure 6 An example UI 600 that the client device 102 can present in response to a user changing the value of a first parameter (in this example, the popularity of the content items) and the value of a second parameter (in this example, the recency of the content items) is shown. For instance, in this example, the user of the client device has moved the slider to the left of the UI further to the right (e.g., requesting less popular content items), while moving the slider to the right of the UI to the left (e.g., requesting relatively older content items). In response, the interpolation component 114 can thus compute a new list 602 of content items that correspond to these user inputs. For example, the list 602 can include content items that were published over a relatively large time range and that are not overly popular.

[0072] Figure 7 ​A flowchart showing an example process 700 by which a remote computing system can generate one or more trained machine learning models and generate result data for use by a client device in providing content item recommendations to a user. This process, as well as each process described herein, is shown as a collection of blocks in a logical flow graph, representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and / or in parallel to implement the processes.

[0073] In some cases, the computing system 106 can be configured to perform some or all of these operations, but in other cases other devices can additionally or alternatively perform some or all of these operations.

[0074] At operation 702, the computing system 106 can train a machine learning model. As shown, this operation can include a series of sub-operations. For example, training the model can include determining, at sub-operation 702(1), a consumption history of one or more users. As described above, this can include a game history, a consumption history of movies or music, etc. Meanwhile, sub-operation 702(2) can include selecting, from each history, a work to be the output of the model being trained. In some cases, this selection can be based on: (1) those content items in the history that satisfy parameter values associated with the model currently being trained, and (2) how much each content item was consumed by the user. For example, the computing system 106 can apply a larger weight to those content items that were consumed more by the respective user, thereby increasing the likelihood that these content items will be selected as the output. Finally, at sub-operation 702(3), the computing system 106 can train one or more internal layers of the model using each consumption history as input and each selected work as output. These sub-operations can be performed consecutively for each of a plurality of consumption histories.

[0075] At operation 704, the process 700 determines whether there are additional models to train. For example, if the process 700 is training“N” models, operation 704 represents determining whether any of the N models are still to be trained. If so, the process 700 returns to operation 702. If not, the process 700 proceeds to operation 706. At operation 706, the computing system 106 can receive a request for content item recommendations from a user’s client device. At operation 708, the computing system 106 determines the user’s consumption history, and at operation 710 inputs that history into the trained model.

[0076] At operation 712, the computing system 106 generates score data indicating a correlation between the user’s consumption history and each respective content item as output from the trained model. In some cases, a relatively higher score corresponds to a relatively higher level of correlation. At operation 714, the computing system 106 modifies each score in the score data based on a particular value of the biasing factor, as described above. Operation 716 represents determining whether additional values of the biasing factor are to be used to bias the scores. For example, if the computing system 106 is to use“M” values of the biasing factor to bias the score data, operation 716 represents determining whether any of the“M” values are still to be used to bias the scores. If so, the process 700 returns to operation 714 for a new value of the biasing factor.

[0077] If not, the process 700 proceeds to operation 718, which represents determining whether additional models are to be used to generate the user’s score data. For example, if the computing system 106 is configured to use“N” models to generate score data tailored to the user, operation 718 represents determining whether any of the N models are still to be used to generate the user’s score data. If so, the process 700 returns, inputting the user’s history data into the next model at operation 710. If not, the process 700 proceeds, sending the result data (e.g., a matrix or other representation of the biased score data) to the client device to enable presentation of recommendations to the user.

[0078] Figure 8 A flowchart illustrating an example process 800 by which a user’s client device can determine an order in which to present one or more content items to the user is shown. While the process 800 is described as being performed on the client device 102, it should be understood that the process 800 can additionally or alternatively be performed on other devices.

[0079] At operation 802, the client device 102 receives score data indicating a correlation between a consumption history of a user of the client device and each of a plurality of content items. In some cases, the score data can include the result data 312 discussed above. At operation 804, the client device 102 receives input from the user specifying at least one of a value of a parameter used to train a model or a value of a bias factor used to bias the score data. For example, the input can specify a desired recency of content items to be recommended (e.g., a time range associated with their respective publication dates), a desired cost range of content items, a desired level of popularity of content items, and the like. At operation 806, the client device 102 determines an order in which to present the recommended content items based at least in part on the score data and the received input. For example, the client device 102 can perform interpolation on the score data based on the values of the parameters and / or bias factors specified by the user.

[0080] At operation 808, the client device 102 presents the content item recommendations in the determined order, and at operation 810, the client device determines whether additional input from the user is received. If so, the process 800 proceeds to determine the order in which to present the content item recommendations to the user again based on the input. If not, the process 800 ends at operation 812.

[0081] Although the subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as illustrative forms of implementing the claims.

Claims

1. A method comprising: determining, by a computing system, a history of one or more game titles played by a user; inputting, by the computing system, data indicative of the history into a first trained machine learning model; generating, as output of the first trained machine learning model, first score data indicative, for each of a plurality of first game titles released within a first time window, of a score representing a correlation between the history and the respective first game title; inputting, by the computing system, the data indicative of the history into a second trained machine learning model; generating, as output of the second trained machine learning model, second score data indicative, for each of a plurality of second game titles released within a second time window, of a score representing a correlation between the history and the respective second game title, wherein the second time window is different from the first time window; and in response to a request from a client device associated with the user, sending, by the computing system to the client device, result data comprising a set of bias scores generated using the first score data and the second score data for interpolation of the result data at the client device.

2. The method of claim 1, further comprising: for each of the plurality of first game titles, determining a level of popularity of the respective first game title; modifying the first score data based at least in part on the respective levels of popularity of the plurality of first game titles and a first specified level of popularity to generate third score data; modifying the first score data based at least in part on the respective levels of popularity of the plurality of first game titles and a second specified level of popularity to generate fourth score data; for each of the plurality of second game titles, determining a level of popularity of the respective second game title; modifying the second score data based at least in part on the respective levels of popularity of the plurality of second game titles and the first specified level of popularity to generate fifth score data; and modifying the second score data based at least in part on the respective levels of popularity of the plurality of second game titles and the second specified level of popularity to generate sixth score data; and wherein the result data comprises the third score data, the fourth score data, the fifth score data, and the sixth score data.

3. The method of claim 1, wherein the first time window comprises a first time period before a current time, and the second time window comprises a second time period before the current time, the second time period including and being greater than the first time period.

4. The method of claim 1, wherein the user comprises a first user, and the method further comprises training a first machine learning model to generate the first trained machine learning model, the training comprising: ​ determining, by the computing system, a history of one or more game titles played by a second user; selecting, by the computing system, a game title from among the one or more game titles played by the second user, the selected game title having been released within the first time window; and training one or more internal layers of the first machine learning model using the one or more game titles played by the second user as input to the first machine learning model and using the selected game title as output from the first machine learning model.

5. The method of claim 4, further comprising training a second machine learning model to generate the second trained machine learning model, the training comprising: determining, by the computing system, a history of one or more game titles played by a third user; selecting, by the computing system, a game title from among the one or more game titles played by the third user, the selected game title from among the one or more game titles played by the third user having been released within the second time window; and training one or more internal layers of the second machine learning model using the one or more game titles played by the third user as input to the second machine learning model and using the selected game title from among the one or more game titles played by the third user as output from the second machine learning model.

6. A method comprising: sending, from a client device associated with a user, a request for content item recommendations to a remote computing system; in response to the request, receiving, at the client device, result data from the remote computing system, the result data including a set of bias scores generated using first score data and second score data, wherein, for each first content item from among a plurality of first content items released within a first time window, the first score data indicates a correlation between a consumption history of the user and the respective first content item; and for each second content item from among a plurality of second content items released within a second time window, the second score data indicates a correlation between the consumption history of the user and the respective second content item, wherein the second time window is different from the first time window; receiving, at the client device, first input data specifying a first time; generating third score data using an interpolation of at least the result data, wherein the interpolation is performed at the client device using the first time; determining, at the client device, a first order of one or more content items to be presented on a display associated with the client device based at least in part on the third score data; presenting the one or more content items in the determined first order on the display; receiving, at the client device, second input data specifying a second time different from the first time; generating fourth score data using an interpolation of at least the result data, wherein the interpolation is performed at the client device using the second time; determining, at the client device, a second order of one or more additional content items to be presented on the display based at least in part on the fourth score data; and presenting the one or more additional content items on the display in the determined second order.

7. The method of claim 6, wherein: the first score data is further based at least in part on a respective popularity level of each of the plurality of first content items; and the second score data is further based at least in part on a respective popularity level of each of the plurality of second content items.

8. The method of claim 6, wherein the first score data has been biased based at least in part on (1) a first target popularity level and (2) a popularity level of each respective first content item, and the method further comprises: receiving, at the client device, fifth score data indicating, for each of the plurality of first content items published within the first time window, a correlation between the consumption history of the user and the respective first content item, wherein the fifth score data has been biased based at least in part on (1) a second target popularity level and (2) the popularity level of each respective first content item.

9. The method of claim 8, wherein: the first input data further specifies a desired popularity level; and the determination of the first order is further based at least in part on the fifth score data and the specified desired popularity level.

10. The method of claim 9, further comprising: generating sixth score data using an interpolation of at least the first score data, the second score data, and the fifth score data; and wherein the determined first order is based at least in part on the sixth score data.

11. The method of claim 6, wherein the first time window comprises a first time period before a current time, and the second time window comprises a second time period before the current time, the second time period including and being greater than the first time period.

12. One or more computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause a client device associated with a user to perform acts comprising: sending, from the client device, a request for content item recommendations to a remote computing system; in response to the request, receiving, at the client device, result data from the remote computing system, the result data including a set of biased scores generated using first score data and second score data, wherein, for each of a plurality of first content items published within a first time window, the first score data indicates a correlation between a consumption history of the user and a respective first content item; and the second score data indicates a correlation between the consumption history of the user and a respective second content item. for each of a plurality of second content items published within a second time window, the second score data indicating a correlation between the consumption history of the user and the respective second content item, wherein the second time window is different from the first time window; receiving, at the client device, first input data specifying a first time; generating third score data using interpolation of at least the result data, wherein the interpolation is performed at the client device using the first time; determining, at the client device, a first order of one or more content items to be presented on a display associated with the client device based at least in part on the third score data; presenting the one or more content items on the display in the determined first order; receiving, at the client device, second input data specifying a second time different from the first time; generating fourth score data using interpolation of at least the result data, wherein the interpolation is performed at the client device using the second time; determining, at the client device, a second order of one or more further content items to be presented on the display based at least in part on the fourth score data; and presenting the one or more further content items on the display in the determined second order.

13. The one or more computer-readable media of claim 12, wherein: the first score data is further based at least in part on a respective popularity level of each of the plurality of first content items; and the second score data is further based at least in part on a respective popularity level of each of the plurality of second content items.

14. The one or more computer-readable media of claim 12, wherein the first score data has been biased based at least in part on (1) a first target popularity level and (2) a popularity level of each respective first content item, and the method further comprises: receiving, at the client device, fifth score data indicating, for each of the plurality of first content items, a correlation between the consumption history of the user and the respective first content item, wherein the fifth score data has been biased based at least in part on (1) a second target popularity level and (2) the popularity level of each respective first content item.

15. The one or more computer-readable media of claim 14, wherein: the first input data further specifies a desired popularity level; and the determination of the first order is further based at least in part on the fifth score data and the specified desired popularity level.

16. The one or more computer-readable media of claim 15, the acts further comprising: generating sixth score data using interpolation of at least the first score data, the second score data, and the fifth score data; and wherein the determined first order is based at least in part on the sixth score data.

Images