Smart agent for assets and layouts in a cold start scenario

Abstract

Systems and methods for generating content prediction data for accurate content retrieval. Such a method includes (i) determining whether to generate historical performance training data or predicted performance training data; (ii) when determining to generate the historical performance training data for a first content item of the plurality of content items: (a) generating the historical performance training data for the first content item based on historical serving data associated with the first content item, and (b) training a smart agent model using the historical performance training data; and (iii) when determining to generate the predicted performance training data for a second content item of the plurality of content items: (a) generating, using the smart agent model, the predicted performance training data for the second content item; and (b) training a content retrieval model using the historical performance training data and the predicted performance training data.

Description

PATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00SMART AGENT FOR ASSETS AND LAYOUTS IN A COLD START SCENARIOFIELD OF TECHNOLOGY

[0001] The present disclosure relates to providing new content to users and, more specifically, to techniques for analyzing and generating predictions regarding new content without available historical information.BACKGROUND

[0002] The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventor(s), to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

[0003] When serving content to a user, conventional systems use historical performance training data for the content to determine how best and / or to which users to serve the content. However, when new content assets are first uploaded to a server to be served to users, there is no historical performance training data associated with the new content. As such, conventional systems have no data with which to determine how and / or when to serve new content, which can in turn lead the new content not being served even in situations where that content is otherwise a good candidate for serving (e.g., highly relevant, etc.). This can be referred to as the “cold start problem.”

[0004] Some systems attempt to address the cold start problem by reserving some number of content serves for new content, in an attempt to “kick start” the generation of historical performance training data that will in turn facilitate later serving of the content. However, such an approach may lead to a significant quantity of mismatched serves until sufficient data is gathered. Other systems simply wait to organically serve the new content. However, such an approach may lead to an inordinately long start-up time for new content. Because newer content may in effect be unavailable during that start-up time, the system may fail to select / serve the best candidate for a given situation (e.g., the most relevant digital advertisement).SUMMARY

[0005] In some aspects, the techniques described herein relate to a computer-implemented method for generating content prediction data for accurate content retrieval, the computer-PATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 implemented method including: determining, by one or more processors and based on an amount of historical serving data for at least some of a plurality of content items, whether to generate historical performance training data or predicted performance training data; when determining to generate the historical performance training data for a first content item of the plurality of content items: generating, by the one or more processors, the historical performance training data for the first content item based on historical serving data associated with the first content item, and training, by the one or more processors, a smart agent model using the historical performance training data; when determining to generate the predicted performance training data for a second content item of the plurality of content items: generating, by the one or more processors and using the smart agent model, the predicted performance training data for the second content item; and training, by the one or more processors, a content retrieval model using the historical performance training data and the predicted performance training data.

[0006] In some aspects, the techniques described herein relate to a computer-implemented method, further including: determining, by the one or more processors and using the content retrieval model, to auction a group of content items including the second content item of the plurality of content items; and determining, by the one or more processors and using an auction model, a particular content item of the group of content items that is to be sent to a client device.

[0007] In some aspects, the techniques described herein relate to a computer-implemented method, further including: training, by the one or more processors, the content retrieval model based on an output of the auction model.

[0008] In some aspects, the techniques described herein relate to a computer-implemented method, wherein the training of the content retrieval model includes: replacing, by the one or more processors, the predicted performance training data with the historical performance training data for the second content item.

[0009] In some aspects, the techniques described herein relate to a computer-implemented method, wherein: a first subset of the at least some of the plurality of content items includes a first layout to be applied to the first subset, the first subset including the first content item; a second subset of the at least some of the plurality of content items includes a second layout to be applied to the second subset, the second subset including the second content item; and thePATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 content retrieval model selects content item assets based at least on a layout associated with the content item assets.

[0010] In some aspects, the techniques described herein relate to a computer-implemented method, wherein the first content item includes a same content asset as the second content item.

[0011] In some aspects, the techniques described herein relate to a computer-implemented method, wherein determining whether to generate the historical performance training data or the predicted performance training data includes: determining, by the one or more processors, whether the at least some of the plurality of content items meet a historical serving data threshold.

[0012] In some aspects, the techniques described herein relate to a computer-implemented method, wherein the determining whether the content items meet the historical serving data threshold includes: retrieving, by the one or more processors and from at least one database, one or more first tables of content items and one or more second tables of impression counters; and determining, by the one or more processors, whether each impression counter of the impression counters meets the historical serving data threshold.

[0013] In some aspects, the techniques described herein relate to a computer-implemented method, wherein the smart agent model is a large language model (LLM).

[0014] In some aspects, the techniques described herein relate to a computer-implemented method, wherein the historical performance training data includes at least (i) one or more input impression features for a content item and (ii) an indication of an auction winner for an auction involving the content item.

[0015] In some aspects, the techniques described herein relate to a computer-implemented method, wherein the one or more input impression features includes at least one of: (i) a language associated with the content item; (ii) a location from which a query for the content item originated; or (iii) a category of one or more categories associated with the location.

[0016] In some aspects, the techniques described herein relate to a computer-implemented method, wherein the historical performance training data further includes a ranking for the content item in each category of the one or more categories.PATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00

[0017] In some aspects, the techniques described herein relate to a computer-implemented method, wherein the predicted performance training data includes at least a predicted ranking for the second content item in one or more potential categories.

[0018] In some aspects, the techniques described herein relate to a computer-implemented method, wherein the content retrieval model is configured to determine whether to output the second content item based on the predicted ranking.

[0019] In some aspects, the techniques described herein relate to an apparatus, functioning as a server device, including: a transceiver; and one or more processors configured to perform a method according to any one of the preceding aspects.BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG. 1 is a block diagram of an example system in which techniques of the present disclosure can be implemented.

[0021] FIG. 2A depicts an example artificial intelligence model that may be implemented in the system of FIG. 1.

[0022] FIG. 2B depicts an example large language model that may be implemented in the system of FIG. 1.

[0023] FIG. 3 depicts a block diagram of an example system of modules and databases, implemented in the system of FIG. 1, for training and utilizing a smart agent.

[0024] FIG. 4 is a flow diagram of an example method for training and utilizing a smart agent and content retrieval model for content items with and / or without historical data, implemented in the system of FIG. 1.DETAILED DESCRIPTION OF THE DRAWINGS

[0025] By using a hybrid approach that combines historical data analysis along with smart agent data predictions, the instant systems and methods address the cold start problem while avoiding the problems inherent in conventional approaches. In particular, by using smart agent functioning as a large language model (LLM), a computing system may predict performance / user behavior for new content (e.g., predict view impression rates, click-through rates, user ratings, etc.). The content retrieval model that matches content with potential user display opportunities (e.g., digital ad slots) can then use the output prediction data of the smart agent for training and determining whether to serve the new content. As such, thePATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 server device may overcome the cold start problem while matching the new content for serving to an appropriate user.

[0026] In particular, the smart agent may be trained using historical data for other content. For example, the smart agent may be trained using impression data and / or rankings associated with other content. The trained smart agent may then generate prediction data for the new content. The system may then train the retrieval model using both the prediction data for the new content and historical data for other content. As the retrieval model and / or an auction model serve the new content and the system begins generating historical data for the new content, the system may replace the prediction data with the new historical data.

[0027] In some implementations, the new content includes a new layout for content in addition to or in place of the actual new content assets. As such, the structure of the content, which may be complicated or difficult to analyze, is broken down into a simpler construction for the model, and the distribution is analyzed accordingly. Furthermore, by ranking for content using a layout, larger swathes of content items may be more easily analyzed, particularly where many images are able to be generated automatically (e.g., with generative Al technologies) and individual analysis may be difficult and / or resource intensive.Therefore, ranking for content using the layout may be analyzed in concert, improving the overall performance of the model and of the ability of the model to distribute content accordingly.

[0028] FIG. 1 illustrates an example system 100 in which the techniques disclosed herein may be implemented. The example system 100 includes a client device 102, a computing system 104, an impressions database 106, a content database 108, and a network 110. The computing system 104 in some implementations is remote from the client device 102 and / or impressions database 106, and communicatively coupled to the client device 102 and / or impressions database 106 via the network 110. It will be understood that system 100 is exemplary, and that other systems may include additional, fewer, or alternative components (e.g., auction model 166 may be omitted). Similarly, arrangements of the components of system 100 may be modified. For example, some elements of system 100 may be combined, split apart, swapped, etc.

[0029] The network 110 may be a single communication network (e.g., the Internet), and in some implementations also includes one or more additional networks. As an example, the network 110 may include a cellular network, the Internet, and a server-side local areaPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 network (LAN). While FIG. 1 shows only a single client device 102, impressions database 106, and content database 108, it will be understood that the system 100 may include any suitable number of similar client devices, computing devices, and / or databases operating according to the principles disclosed herein.

[0030] Generally, the client device 102 is configured to access information resources (e.g., web pages, application user interfaces, etc.) that may be supplied or published by the computing system 104, content providers (e.g., storing content in content database 108), and / or other entities, and the computing system 104 is generally configured to analyze and select content to be served to the client device 102 along with links to information resources (e.g., landing pages) associated with the sponsored content options. As used herein, “sponsored content” may refer to advertisement / marketing content, or any other type of content that is provided by and / or otherwise associated with a third party (e.g., a party other than an entity associated with the computing system 104 of FIG. 1). The information resources, and / or content items (e.g., digital advertisements) associated with the information resources, may be stored in content databases such as content database 108. In other implementations, the content database 108 is instead a part of the computing system 104 (e.g., if third parties supply such content for local storage at computing system 104, or if computing system generates content on behalf of third parties, etc.).

[0031] The client device 102 may be or include any stationary, mobile, or portable computing device with wired and / or wireless communication capability (e.g., a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart wearable device such as smart glasses or a smart watch, a vehicle head unit computer, etc.). In the example implementation of FIG. 1, the client device 102 includes a network interface 120, a processor 122, memory 124, and a display 126. The processor 122 may be a single processor (e.g., a central processing unit (CPU)), or may include a set of processors (e.g., multiple CPUs, or one or more CPUs and one or more graphics processing units (GPUs)).

[0032] The memory 124 includes one or more computer-readable, non-transitory storage units or devices, which may include persistent (e.g., hard disk) and / or non-persistent memory components. The memory 124 stores instructions that are executable by the processor 122 to perform various operations, including the instructions of various software applications and the data generated and / or used by such applications. In the example implementation of FIG. 1, the memory 124 stores at least an application 130, which may be, for example, a webPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 browser application, a mobile application downloaded from an application store, or a video player application.

[0033] Generally, application 130 is executed by processor 122 to present information resources and / or image data to the user of the client device 102 via the display 126 (and / or one or more speakers of the client device 102, not shown in FIG. 1). In an implementation where the application 130 is a web browser application, for instance, an information resource may be a web page hosted by a publisher or a content provider, with the web browser causing the client device 102 to download HyperText Markup Language (HTML), scripts, and / or other code of the web page for presentation to a user via the display 126.

[0034] The display 126 includes hardware, firmware, and / or software configured to enable a user to view visual outputs of the client device 102, and may use any suitable display technology (e.g., LED, OLED, LCD, etc.). In some implementations, the display 126 is incorporated in a touchscreen having both display and manual input capabilities. Moreover, in some implementations where the client device 102 is a wearable device, the display 126 is a transparent viewing component (e.g., lenses of smart glasses) with integrated electronic components. For example, the display 126 may include micro-LED or OLED electronics embedded in lenses of smart glasses.

[0035] The network interface 120 includes hardware, firmware, and / or software configured to enable the client device 102 to exchange electronic data with the computing system 104 via the network 110. For example, the network interface 120 may include a cellular communication transceiver, a Wi-Fi transceiver, and / or transceivers for one or more other wired and / or wireless communication technologies.

[0036] While FIG. 1 shows client device 102 as a single component communicating directly (i.e., via network 110) with the computing system 104, in some implementations the subcomponents of client device 102 shown in FIG. 1 are instead divided among two or more user-side devices. As just one example, a pair of smart glasses may include the processor 122, the memory 124, and the display 126, while a smartphone may include another processing unit, another memory, another display, and the network interface 120. The smart glasses (or smart helmet, etc.) may then communicate as needed with the smartphone (e.g., via Bluetooth) to enable the operations described herein.

[0037] The computing system 104 includes a network interface 140, a processor 142, and memory 144. The network interface 140 includes hardware, firmware, and / or softwarePATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 configured to enable the computing system 104 to exchange electronic data with the client device 102 and other, similar client devices via the network 110. For example, the network interface 140 may include a wired or wireless router and a modem. The processor 142 may be a single processor, may include two or more processors, etc. The computing system 104 may include one or more servers, for example, which may reside at a single location or multiple locations.

[0038] The memory 144 is a computer-readable, non-transitory storage unit or device, or collection of units / devices that may include persistent and / or non-persistent memory components. The memory 144 stores the instructions of a smart agent module 150, a retrieval and auction module 152, and a training module 154, each of which may be executed by the processor 142. In the example system 100, the smart agent module 150 includes (or remotely accesses) a prediction module 160 and a ranking module 162. The retrieval and auction module 152 includes (or remotely accesses) a content retrieval model 164 and an auction model 166. The training module 154 uses historical data 168 and / or predicted data 169 to train one or more machine learning models (e.g., the smart agent module 150 and / or a content retrieval model 164). In some implementations, some of the software modules / units shown in FIG. 1 are omitted. For example, the retrieval and auction module 152 may omit the auction model 166, or the computing system 104 may omit training module 154 (e.g., if the training is done by a different computing system). The smart agent module 150 and / or the content retrieval model may be or include an LLM or other suitable generative Al model. As another example, the smart agent module 150 includes an LLM while the content retrieval model 164 is a non-LLM model.

[0039] The smart agent module 150, retrieval and auction module 152, and training module 154 may be software modules comprising instructions executed by the processor 142 to perform the various operations described herein. It is understood, however, that other architectures are also possible (e.g., with functionality of modules 150 and 152 being provided by a single software module, or with functionality of module 150 being split among a plurality of software modules, and so on).

[0040] Generally, the smart agent module 150 uses a prediction module 160 in conjunction with a ranking module 162 to generate predictions and / or predicted user rankings regarding input content (e.g., content that has a cold start as described above) and criteria or features (e.g., a language, where a query is coming from, a vertical category, etc.) for a scenario. ForPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 example, the prediction module 160 may receive an input of a content item and, in conjunction with a ranking module 162, may generate a predicted user response to the content item. In some implementations, the prediction module 160 is, includes, or functions similarly to a large language model (LLM) as described below with regard to FIGs. 2A and 2B. In particular, the prediction module 160 may be trained to determine an output sequence of events based on similar input concepts and / or features (e.g., as described in PCT Pat. App. US2023 / 085,881, titled “USER BEHAVIOR PREDICTION VIA GENERATIVE MODELING OF EVENT SEQUENCES” and filed on December 26, 2023, which is hereby incorporated herein by reference). In further implementations, the prediction module 160 may be trained to receive a prompt to generate an output analysis of an input content item. In some implementations, the prediction module 160 additionally or alternatively generates and / or receives a scenario for the content item (e.g., a website on which the content item would be served, a demographic of a simulated user, account preference information for a simulated user, etc.).

[0041] In some implementations, the ranking module 162 receives and utilizes the responses generated by the prediction module 160 to rank the performance of the content item in the generated scenario. For example, the ranking module 162 may receive indications of user responses to the content item in a scenario and generate a click-through rate, a general interaction rate, a vertical category ranking, and / or any other such metric as described herein. In some implementations, the ranking may be a binary 1 or 0 to indicate whether the content would be a match or not, respectively. In further implementations, the ranking is on a scale (e.g., 1 to 5, 1 to 10, -5 to 5, etc.) based on how much of a match for the scenario features the content item is (e.g., how likely a user is to interact). In still further implementations, the ranking is for each feature (e.g., a ranking for a particular query location, a ranking for a particular vertical category, etc.) and / or each element of the content item (e.g., a ranking for an asset, a ranking for a layout, etc.). The smart agent module 150 may then provide the output of the ranking module 162 to the training module 154 (e.g., as predicted data 169) and / or the retrieval and auction module 152 to be used in training the retrieval and auction module 152.

[0042] In some implementations, the retrieval and auction module 152 uses the content retrieval model 164 in conjunction with the auction model 166 to match a content item (e.g., from a content database 108) with a user. In some implementations, the content retrieval model 164 is trained by the training module 154 using historical data 168 and predicted dataPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00169. In particular, the content retrieval model 164 is trained to determine one or more content items suitable for a content item to be displayed to a user (e.g., via a client device 102). For example, the content retrieval model 164 may receive one or more content items as well as scenario information (e.g., a vertical category (e.g., fashion, music, entertainment, sports, cooking, etc.), a user demographic, user account information, geographical information, etc.) as inputs. The content retrieval model 164 may then determine which of the one or more content items are suitable to be potentially displayed to the user. In some implementations, the content retrieval model 164 determines such using historical data (e.g., from impressions database 106) and / or predicted data (e.g., generated by the smart agent module 150). In some such implementations, the content retrieval model 164 analyzes the historical data and / or the predicted data to generate a prediction and / or likelihood of matching a user preference and / or other such content item ranking criterion (e.g., publisher criterion, account preference, click-through likelihood, etc.), as described herein.

[0043] In further implementations, the content item includes a layout for content item assets, and the output of the ranking module 162 includes rankings for one or more layouts for the content item asset. In some such implementations, the content retrieval model 164 determines with which layout the content item asset has a highest score and retrieves the version of the content item with the corresponding layout. As such, the content retrieval model 164 provides the content item with the highest scoring / ranking layout to the auction model 166 to be auctioned to a client device 102, as described below.

[0044] In further implementations, the content retrieval model 164 determines one or more potential matching content items and provides the content items to the auction model 166. The auction model 166 then auctions the slot for providing content item to a user (e.g., via a client device 102) to the one or more content items and / or to publisher devices (not shown) associated with the respective content items. The auction model 166 then provides the auction winner to the client device 102. Depending on the implementation, the computing system 104 may update the impressions database 106 and / or content database 108 with the results of the auction. In some implementations, the computing system 104 further updates historical data 168 and / or predicted data 169 based on the results of the auction. In some implementations, if the auction results cause a content item to no longer require and / or be considered for a cold start, the computing system 104 may update the smart agent module 150 and / or retrieval and auction module 152 accordingly.PATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00

[0045] In some implementations and / or scenarios, the computing system 104 (or another computing system not shown in FIG. 1) trains the models of the smart agent module 150 and / or the retrieval and auction module 152 (e.g., content retrieval model 164 and / or the auction model 166). In particular, the training module 154 may train the modules using historical data 168 (e.g., from the impressions database 106) and / or predicted data 169 (e.g., as generated by the prediction module 160 and / or ranking module 162) as described herein. In some implementations, the historical data 168 includes data associated with past criteria, third party information resources, content items, layouts, etc. Similarly, the predicted data 169 may include predicted information in categories matching those included in the historical data 168. In some implementations, the historical data 168 includes data provided by content providers (e.g., via content database 108). In some implementations, training module 154 is included in a computing system other than computing system 104, and computing system 104 only includes or accesses the models and / or modules in question after the model(s) / module(s) is / are trained. In some implementations, training machine learning models may produce byproduct weights, or parameters which may be initialized to random values. The weights may be modified as the network is iteratively trained, by using one of several gradient descent algorithms, to reduce loss and to cause the values output by the network to converge to expected (or “learned”) values.

[0046] In some implementations, as noted above, the modules and / or models (e.g., the smart agent module 150, the content retrieval model 164, and / or the auction model 166) may be or include a generative Al model, and may have been trained by computing system 104 or another computing system using supervised or semi- supervised learning techniques, using training data of the appropriate modality (e.g., text data). Such generative Al models may be general -purpose models (e.g., trained on a wide array of publicly available datasets such as web pages, documents, etc., available via the Internet) or may be a domain- specific model (e.g., trained or finetuned on custom and / or proprietary datasets, such as documents / data available via one or more intranets). In some implementations, the generative Al models have parameters tuned, via the training process, specifically for high performance in the context of generating text having one or more particular qualities and / or characteristics.

[0047] In some implementations, the computing system 104 accesses a remote server / system that provides generative Al as a service (i.e., with at least a portion of the smart agent module 150, the content retrieval model 164, and / or the auction model 166 residing at a location remote from the computing system 104). In other implementations, the smart agentPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 module 150, the content retrieval model 164, and / or the auction model 166 are local to the computing system 104. Thus, the smart agent module 150, the content retrieval model 164, and / or the auction model 166 may reside at the computing system 104 as shown in FIG. 1, or the computing system 104 may access the smart agent module 150, the content retrieval model 164, and / or the auction model 166 by communicating with another computing system via the network 110. For example, the smart agent module 150, the content retrieval model 164, and / or the auction model 166 may be or include Al models that a remote server makes available to computing systems (including computing system 104) via an application programming interface (API).

[0048] The historical data 168 and / or predicted data 169 may generally include any text data used for training purposes. The historical data 168 and / or predicted data 169 may include, for example, labeled or unlabeled category data, auction / serving data, content layout data, and / or historical data for past categories, auctions, layouts, and / or other metrics as described herein.

[0049] The operation of the smart agent module 150, the retrieval and auction module 152, the training module 154, and their constituent parts, will be discussed in further detail below in connection with various example implementations.

[0050] In some implementations, content providers hold accounts related to the services provided by the computing system 104. In these implementations, information associated with the accounts may be stored in an account database (not shown in FIG. 1). The account database may be stored in the memory 144 or may be stored in one or more memories that are remote from the computing system 104, for example. The account information may include information such as entity name, subscription level, entity preferences (e.g., brand control preferences), and so on. In some implementations, the account information includes selection parameters (e.g., bid amounts or maximum bid amounts), for use by the retrieval and auction module 152 in selecting sponsored content for inclusion in search result information resources. Depending on the implementation, the computing system 104 may utilize account information (e.g., one or more constraints as noted above) at different times depending on the implementation.

[0051] FIGs. 2A and 2B depict exemplary models that may be used (or parts of which may be used) as part of smart agent module 150, content retrieval model 164, and / or auctionPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 model 166, for example. It is understood, however, that these are just some of a number of suitable Al model types that may be used by computing system 104.

[0052] Turning first to FIG. 2A, an exemplary model 200A uses generative Al techniques. The model 200A may be used as part of smart agent module 150, for example. In particular, a generator model 210 and a discriminator model 220 receive inputs to generate a binary classification 235 and output a sequence of words and / or other metrics as described herein. In particular, the generator model 210 receives an input vector 205A to generate a generated example 215. In some implementations, the input vector 205A is a fixed-length random vector. In some implementations, the input vector 205A may be drawn randomly from a Gaussian distribution. Depending on the implementation, the vector space corresponding to the input vector 205 A may include one or more hidden variables (e.g., variables that are not directly observable). In some implementations, the input vector 205 A is used to seed the generative process. Using the input vector 205A, the generator model 210 then generates a generated example 215.

[0053] In some implementations, the discriminator model 220 then receives the generated example 215 or a real example 225. The discriminator model 220 may generate a binary classification 235 inferring / indicating whether the received input is model-generated or real. The exemplary model 200A may additionally output an output product (e.g., a predicted user action, an output answer to a prompt regarding a user action, etc.) and / or use the binary classification 235 in training the generator model 210 and / or discriminator model 220.

[0054] In still further implementations, the exemplary model 200A uses both the generator model 210 and the discriminator model 220 for training and subsequently uses only the generator model 210 for generative modeling as described herein.

[0055] In some implementations, the generator model 210 and the discriminator model 220 are trained according to adversarial techniques (e.g., when the discriminator model 220 correctly generates the binary classification 235, the generator model 210 is updated and, when the discriminator model 220 incorrectly generates the binary classification 235, the discriminator model 220 is updated).

[0056] Depending on the implementation, the generator model 210 and / or the discriminator model 220 may be or include neural networks, such as artificial neural networks (ANN), convolution neural networks (CNN), or recurrent neural networks (RNN). In further implementations, the model 200A, the generator model 210, and / or thePATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 discriminator model 220 may incorporate, include, be, and / or otherwise use techniques including and / or in a manner reminiscent to language model techniques (e.g., an LLM, a bag- of-words model, etc.). Similarly, the model 200A, the generator model 210, and / or the discriminator model 220 may incorporate, include, be, and / or otherwise use a transformer architecture to utilize the appropriate language model techniques, as described with regard to FIG. 2B below.

[0057] FIG. 2B illustrates an exemplary LLM 200B, which receives an input vector 205B similar to input vector 205A and provides an output 260. The LLM 200B may be used as and / or in the smart agent module 150, for example. In some implementations, the LLM 200B is initially trained to predict a word and / or event in a sequence of words and / or events. For example, the LLM 200B may be given a word sequence that leads up to “Today is a,” and predict a next word, such as “sunny day”, “Saturday”, “holiday”, etc. Similarly, the LLM 200B may be trained to generate an event in a series of events. For example, the LLM 200B may be given a series of events that leads up to a content item being displayed to a user and predicting a user response, such as clicking through the content item to a webpage, ignoring the content item, etc.

[0058] In some implementations, transformers are used to train the LLM 200B (e.g., a generative pre-trained transformer (GPT) model). More specifically, some implementations use a GPT model that includes (i) an encoder that processes the input sequence, and (ii) a decoder that generates the output sequence. The encoder and decoder may both include a multi-head self-attention mechanism that allows the GPT model to differentially weight parts of the input sequence to infer meaning and context (e.g., using metadata in the historical and / or training data), for example.

[0059] The input vector 205B may be a vector representative of relationships between words, sequences, etc. in the input. The LLM 200B may include a self-attention block 252 component to attend to different parts of the input simultaneously or near- simultaneously to capture relationships and / or dependencies between the different parts of the input (e.g., referred to as a multi self-attention block, multi-head attention block, multi-head selfattention block, masked multi self-attention block, masked multi-head attention block, masked multi-head self-attention block, etc.). In particular, the self-attention block 252 relates different positions of a sequence to compute a representation of the sequence. As such, the self-attention block 252 may weigh an impact of different words in a sequencePATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 when sequencing. As such, the LLM 200B learns to give emphasis to different portions of an input vector 205B.

[0060] The self-attention block 252 may then compute an attention score representing the impact of each word and / or event in the sentence with respect to the other words and / or events in the sentence (e.g., by taking a dot product between different vector sets). The output then proceeds to the normalization layer 254. The normalization layer 254 may normalize the output of the self-attention block 252 (e.g., by applying a softmax function to normalize the scores).

[0061] Similarly, the self-attention block 252 may provide output to a feed-forward network block 256, which performs a non-linear transformation to generate a new representation of the input and / or relationships between words, sequences, etc. In particular, the feed-forward network block 256 may compute a weighted sum of the vectors, using the calculated and normalized attention scores to capture the contextual relationships between words. In some implementations, the normalization layer 254 and / or the self-attention block 252 performs the computation to generate a representation of the relationship between words, etc. After the feed-forward network block 256, an additional normalization layer 258 may normalize the respective output and / or add residual connection(s) to allow the output to move directly to another input. The LLM 200B may therefore learn which parts of an input are important (e.g., remain prevalent through the normalization process). Depending on the implementation, the training of LLM 200B may repeat the process any suitable number of times.

[0062] As such, the LLM 200B may receive a prompt from a user and / or from another module that may request a predicted user response. The LLM 200B may then generate a predicted user response as trained. For example, the LLM 200B may receive a prompt such as “predict a user response for Content Item A on a site associated with a sports vertical category,” and the LLM 200B may predict the outcome (and / or interact with another module to predict the outcome) and generate an output of a predicted user response (e.g., “the user will click through the content item to the associated resource”).

[0063] Depending on the implementation, an encoder and / or a decoder may be trained as described above. In further implementations, the encoder is trained in accordance with the above, and a decoder includes an additional self-attention block (not shown) receiving the output of the encoder.PATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00

[0064] FIG. 3 depicts an example block diagram 300 for training and using a smart agent to address a cold start. In some implementations, a computing system (e.g., computing system 104 of FIG. 1) may perform the techniques detailed in block diagram 300 to train and / or use a smart agent (e.g., via smart agent module 150) before serving content to a client device (e.g., client device 102).

[0065] In the implementation and scenario of FIG. 3, the computing system 104 receives data from an impressions database 390 (e.g., impressions database 106) and a content database 395 (e.g., content database 108). In some implementations, the computing system 104 and / or computing devices associated with the impressions database 390 and content database 395 perform a join operation on data from each database before the computing system 104 analyzes the data as described below. Depending on the implementation, the content database 395 may include one or more content items (e.g., images, videos, text, etc.) to be served to a user 350. Similarly, the impressions database 390 may include historical data such as impression logs, records of serving outcomes, click-through rates, view impression rates, user ratings, etc.

[0066] The computing system 104 determines, based on the historical data in the impressions database 390 and / or the content item from the content database 395, whether one or more content items require a cold start. In particular, the computing system 104 determines whether a given content item has a sufficient quantity of historical data (e.g., meets a historical data threshold) to proceed without a cold start procedure. In some implementations, the computing system 104 may determine that the content item has a sufficient quantity of historical data to proceed without a cold start procedure when at least a threshold number of samples meet a predetermined metric. For example, the metric may be or include a number of successful serves, a number of determinations to auction, a number of click-through events by a client device, etc. In further implementations, the computing system 104 may determine that the content item has a sufficient quantity of historical data to proceed without a cold start procedure using an algorithm based on multiple performance metrics and / or threshold values.

[0067] It will be understood that, in some implementations, another computing device other than the computing system 104 determines whether the content item requires a cold start. For example, the impressions database 390 may include and / or be stored on aPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 computing device that determines whether a content item has a sufficient quantity of associated historical data to avoid a cold start.

[0068] If the computing system 104 determines that a content item has a sufficient quantity of historical data, then the computing system 104 determines that a cold start is not required and generates training data 315 to use in training the smart agent 320 (e.g., the smart agent module 150) and / or the content retrieval model 325 (e.g., content retrieval model 164). In particular, the computing system 104 trains the smart agent 320 by feeding the generated training data in and training the system as described with regard to FIGs. 2A and 2B (e.g., via the training module 154). Depending on the implementation, the computing system 104 my generate the training data 315 by labeling historical data from the impressions database 390, by analyzing the historical data from the impressions database 390 (e.g., to determine an outcome, to determine input values, etc.), by isolating historical data associated with the content item in particular, etc. Depending on the implementation, the computing system 104 may evaluate multiple content items simultaneously, in succession, and / or in batches. As such, the computing system 104 may train the smart agent 320 and / or the content retrieval model 325 using multiple content items and historical data associated with each of the respective content items. As such, the computing system 104 can train the smart agent 320 on some content items stored in a content database 395, all content items stored in a content database 395, selected item(s) from the content database 395, etc.

[0069] If the computing system 104 determines that the content does require a cold start, the computing system 104 uses the smart agent 320 to analyze the content and generate prediction data 310. In some implementations, the computing system 104 generates the rendered content 305 and feeds the content 305 into the smart agent 320, which in turn generates the prediction data 310. In some such implementations, the rendered content 305 includes a layout and the asset. For example, the asset may be an image or video, and the layout may be a particular method of displaying the asset and / or other elements for the content. The rendered content 305 may additionally include text, a logo, etc. For example, the text, logo, etc. may be overlayed on the asset, to the side of the asset, under the asset, and / or otherwise displayed according to the layout.

[0070] In some implementations, the computing system 104 initially determines whether each content item requires a cold start prior to training the smart agent 320 and / or using the smart agent 320 to analyze data for any of the content items. The computing system 104 thenPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 sorts the content items and performs one operation before the other (e.g., trains the smart agent 320 using historical data for content items that do not require a cold start before analyzing content that does require a cold start). In further implementations, the computing system 104 makes the determination and performs the corresponding operation for each content item sequentially. In some such implementations, the computing system 104 can determine to update the analysis of a content item that requires a cold start after updating the smart agent 320, after training the smart agent 320 a predetermined number of times, after a predetermined period of time, etc.

[0071] In further implementations, because the smart agent 320 is trained on content items with a sufficient quantity of historical data, the smart agent 320 can predict, with a relatively high degree of accuracy, what a user response would be to the new content item and therefore generate a prediction regarding how the user would respond and / or how the user would rate the rendered content 305. In various implementations, the prediction data 310 includes the content item, a rating associated with the content item (e.g., for one or more vertical categories such as sports, fashion, entertainment, etc.), characteristics associated with the content and / or a user query (e.g., a user location, a website from which the query originates, one or more account details associated with the user, etc.). In some such implementations, the prediction data 310 is predicated on one or more hypothetical scenarios (e.g., a user in a predetermined age demographic, a webpage classified according to a vertical category, a query originating from a particular geographic location, etc.) and / or predetermined criteria.

[0072] The computing system 104 trains the content retrieval model 325 using the generated training data 315 and the prediction data 310. In some implementations, the computing system 104 collates the generated training data 315 and the prediction data 310 and trains the content retrieval model 325 using the collated pool of training data. In further implementations, the computing system 104 keeps the generated training data 315 and the prediction data 310 separated (e.g., by tagging the data respectively, by including one or more markers in the data, by annotating the data appropriately, etc.). In still further implementations, the computing system 104 separates the training data by the content item with which the training data is associated.

[0073] The content retrieval model 325 then matches content from the content database 395 with a user query for content. For example, the content retrieval model 325 may use analyzed scenarios and / or criteria associated with the content item (e.g., either historical fromPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 the generated training data 315 or predicted from the prediction data 310) to determine a rating associated with the content item and for the scenario. In further implementations, the prediction data 310 and generated training data 315 includes ratings for the content item (e.g., in various vertical categories), and the content retrieval model 325 uses and / or generates ratings based on such.

[0074] After the content retrieval model 325 selects content to potentially provide to the user 350 based on the category, the content retrieval model 325 may provide the selected content items to an auction model 330 for auctioning to serve to the user 350. In further implementations, the content retrieval model 325 provides the selected content directly to the user 350. In still further implementations, the content retrieval model 325 and the auction model 330 are a single model and / or module, and therefore work in concert to determine ratings for the content item and provide to a user 350.

[0075] It will be understood that, although FIG. 3 depicts example scenarios with particular components, a system may include additional, alternate, or fewer components and / or functionalities as those described above.

[0076] FIG. 4 is a flow diagram of an example method 400 for providing information responsive to a user search. The method 400 may be implemented as instructions stored on one or more non-transitory, computer-readable media (e.g., memory 144) and executed by one or more processors in one or more computing devices. For example, the method 400 may be implemented by the processor 142 of the computing system 104 in FIG. 1, when executing instructions of the smart agent module 150 and / or retrieval and auction module 152. It will be understood that additional, fewer, and / or alternate components may be used to implement the example method 400.

[0077] At block 402, the computing system 104 determines, based on an amount of historical serving data associated with at least some of a plurality of content items, whether to generate historical performance training data or predicted performance training data. In some implementations, the computing system 104 determines whether to generate the historical performance training data or the predicted performance training data based on whether the at least some of the plurality of content items meet a historical serving data threshold. In some such implementations, the computing system 104 determines whether the content items meet the historical serving threshold by retrieving, from at least one database, one or more first tables of content items and one or more second tables of impression counters. The computingPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 system 104 then determines whether each impression counter of the impression counters meets the historical serving data threshold.

[0078] In at least some scenarios, the plurality of content items includes at least (i) a first content item with a sufficient quantity of historical serving data to generate historical performance training data and (ii) a second content item without a sufficient quantity of historical serving data (e.g., the computing system 104 determines to generate predicted performance training data). It will be understood that, as used herein, the first content item may refer to a first set of content items and / or a content item of a first set of content items (e.g., the computing system 104 may train a smart agent and / or content retrieval model(s) with historical data associated with (each of) the first set of content items). Similarly, the second content item may refer to a second set of content items and / or a content item of a second set of content items. Depending on the implementation, the first content item and / or the second content item may include a content item layout, a content item asset, etc. For example, a first subset of the plurality of content items (e.g., including the first content item) may include a first layout to be applied to assets associated with the first subset of content items, and a second subset of the plurality of content items (e.g., including the second content item) may include a second layout to be applied to assets associated with the second subset of content items. The content retrieval model may select content item assets based at least on the layout associated with the respective assets. In some implementations and / or scenarios, the first content item and second content item may include a same content item asset but different layouts. In further implementation, the first content item and second content item may include a same layout, but different content item assets.

[0079] At block 404, the method 400 splits based on the determination at block 402 (e.g., based on whether the computing system 104 determines that the content item should have historical performance training data or predicted performance training data (e.g., requires a cold start or not)). In particular, when the computing system 104 determines to generate the historical performance training data for a first content item of the plurality of content items, the flow proceeds to block 406. Otherwise, when the computing system 104 determines to generate the predicted performance training data for a second content item of the plurality of content items, the flow proceeds to block 410.

[0080] At block 406, the computing system 104 generates the historical performance training data for the first content item based on historical serving data associated with the firstPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 content item. In some implementations, the historical performance training data includes at least (i) one or more input impression features for a content item; (ii) an indication of an auction winner for an auction involving the content item; (iii) a ranking for the content item in each category (e.g., of one or more vertical categories such as sports, entertainment, fashion, food, etc.). In further implementations, the one or more input impression features include at least one of (i) a language associated with the content item; (ii) a location from which a query for the content item originated (e.g., a geographical location of a user, device, account, etc. requesting the content item); (iii) a category of one or more categories associated with the location; and / or (iv) any other such input impression feature.

[0081] At block 408, the computing system 104 trains a smart agent model using the historical performance training data. In some implementations, the smart agent is a large language model (LLM) and is trained as described above with regard to FIGs. 2A and 2B. In some implementations, the computing system 104 trains the smart agent to mimic user behavior (e.g., by predicting view impressions, click-through rates, content item asset ratings, content item layout ratings, etc.). The smart agent can then feed the prediction performance training data (e.g., as described below), to a content retrieval model and / or an auction model.

[0082] At block 410, the computing system 104 generates, using the smart agent model, the predicted performance training data for the second content item. In some implementations, the predicted performance training data includes similar data as the historical performance training data. For example, the predicted performance training data may include input impression features, a predicted auction winner, a ranking for the second content item in each of one or more vertical categories (e.g., as described above), etc.

[0083] At block 412, the computing system 104 trains a content retrieval model using the historical performance training data and the predicted performance training data. In some implementations, the content retrieval model is configured to determine whether to output the second content item based on a predicted ranking for the content item and / or a similar content item provided to the content retrieval model in training.

[0084] In some implementations, the computing system 104 determines, using the content retrieval model, to auction a group of content items including the second content item of the plurality of content items. In further such implementations, the computing system 104 determines (e.g., selects), using an auction model, a particular content item of the group of content items is to be sent to a client device. In some implementations, the computing systemPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00104 further trains the content retrieval model based on output of the auction model (e.g., using a winner of the auction scenario to further train the model).

[0085] In further implementations, the computing system 104 further trains the content retrieval model by replacing the predicted performance training data with the historical performance training data for the second content item. As such, the computing system 104 can replace prediction data with actual performance data as the computing system 104 gathers such. In some such implementations, the computing system 104 augments the training for the model by adding the historical performance training data rather than replacing the predicted performance data. In still further implementations, the computing system 104 replaces the data by using previous annotations, marking, and / or other such indicators (e.g., as described above with regard to FIG. 3) to determine which data to replace.

[0086] Artificial intelligence (Al) is a segment of computer science that focuses on the creation of models that can perform tasks with little to no human intervention. Artificial intelligence systems can utilize, for example, machine learning and computer vision.Machine learning, and its subsets, such as deep learning, focus on developing models that can infer outputs from data. The outputs can include, for example, predictions and / or classifications. Computer vision focuses on analyzing and interpreting images and videos. Artificial intelligence systems can include generative models that generate new content in response to input prompts and / or based on other information.

[0087] Example machine-learned models include neural networks or other multi-layer nonlinear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some machine-learned models can include multi-headed self-attention models (e.g., transformer models).

[0088] The model(s) can be trained using various training or learning techniques. The training can implement supervised learning, unsupervised learning, reinforcement learning, etc. The training can use techniques such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used toPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 iteratively update the parameters over a number of training iterations. A number of generalization techniques (e.g., weight decays, dropouts) can be used to improve the generalization capability of the models being trained.

[0089] The model(s) can be pre-trained before domain- specific alignment. For instance, a model can be pretrained over a general corpus of training data and fine-tuned on a more targeted corpus of training data. A model can be aligned using prompts that are designed to elicit domain- specific outputs. Prompts can be designed to include learned prompt values (e.g., soft prompts). The trained model(s) may be validated prior to their use using input data other than the training data and may be further updated or refined during their use based on additional feedback / inputs.

[0090] In some implementations, the computing system 104 may use one or more the machine learning models noted above to perform any one or more of the operations discussed herein in connection with machine learning.

[0091] Although the foregoing text sets forth a detailed description of numerous different aspects and implementations of the invention, it should be understood that the scope of the patent is defined by the words of the claims set forth at the end of this patent. The detailed description is to be construed as exemplary only.

[0092] The following additional considerations apply to the foregoing discussion. Throughout this specification, plural instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and nothing requires that the operations be performed in the order illustrated. Structures and functionality presented as separate components in example configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter of the present disclosure.

[0093] Unless specifically stated otherwise, discussions in the present disclosure using words such as “processing,” “computing,” “calculating,” “determining,” “presenting,” “displaying,” or the like may refer to actions or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or aPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0094] As used in the present disclosure any reference to “one implementation” or “an implementation” means that a particular element, feature, structure, or characteristic described in connection with the implementation is included in at least one implementation or implementation. The appearances of the phrase “in one implementation” in various places in the specification are not necessarily all referring to the same implementation.

[0095] As used in the present disclosure, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a nonexclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present), and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

[0096] Unless otherwise apparent from the context of use, reference in the present disclosure to a same set of “one or more processors” (or a same “plurality of processors,” etc.) performing multiple operations can encompass implementations in which performance of the operations is divided among the processor(s) in any suitable way. For example, “generating, by one or more processors, X; and generating, by the one or more processors, Y” can encompass: (1) implementations in which a first subset of the processors (e.g., in a first computing device) generates X and an entirely distinct, second subset of the processors (e.g., in a different, second computing device) independently generates Y ; (2) implementations in which one or more or all of the processor(s) (e.g., one or multiple processors in the same device, or multiple processors distributed among multiple devices) contribute to the generation of X and / or Y ; and (3) other variations.

[0097] Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs through the principles described herein. Thus, while particular implementations and applications have been illustrated and described, it is to be understood that the disclosed implementations are not limited to the precise construction and components disclosed in the present disclosure. Various modifications, changes, andPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and apparatus disclosed in the present disclosure without departing from the spirit and scope defined in the appended claims.

Claims

PATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00What is claimed is:

1. A computer-implemented method for generating content prediction data for accurate content retrieval, the computer-implemented method comprising: determining, by one or more processors and based on an amount of historical serving data for at least some of a plurality of content items, whether to generate historical performance training data or predicted performance training data; when determining to generate the historical performance training data for a first content item of the plurality of content items: generating, by the one or more processors, the historical performance training data for the first content item based on historical serving data associated with the first content item, and training, by the one or more processors, a smart agent model using the historical performance training data; when determining to generate the predicted performance training data for a second content item of the plurality of content items: generating, by the one or more processors and using the smart agent model, the predicted performance training data for the second content item; and training, by the one or more processors, a content retrieval model using the historical performance training data and the predicted performance training data.

2. The computer- implemented method of claim 1, further comprising: determining, by the one or more processors and using the content retrieval model, to auction a group of content items including the second content item of the plurality of content items; and determining, by the one or more processors and using an auction model, a particular content item of the group of content items that is to be sent to a client device.

3. The computer- implemented method of claim 2, further comprising: training, by the one or more processors, the content retrieval model based on an output of the auction model.

4. The computer-implemented method of claim 3, wherein the training of the content retrieval model includes:PATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 replacing, by the one or more processors, the predicted performance training data with the historical performance training data for the second content item.

5. The computer- implemented method of any one of claims 1-4, wherein: a first subset of the at least some of the plurality of content items includes a first layout to be applied to the first subset, the first subset including the first content item; a second subset of the at least some of the plurality of content items includes a second layout to be applied to the second subset, the second subset including the second content item; and the content retrieval model selects content item assets based at least on a layout associated with the content item assets.

6. The computer- implemented method of claim 5, wherein the first content item includes a same content asset as the second content item.

7. The computer- implemented method of any one of claims 1-6, wherein determining whether to generate the historical performance training data or the predicted performance training data includes: determining, by the one or more processors, whether the at least some of the plurality of content items meet a historical serving data threshold.

8. The computer- implemented method of claim 7, wherein the determining whether the content items meet the historical serving data threshold includes: retrieving, by the one or more processors and from at least one database, one or more first tables of content items and one or more second tables of impression counters; and determining, by the one or more processors, whether each impression counter of the impression counters meets the historical serving data threshold.

9. The computer- implemented method of any one of claims 1-8, wherein the smart agent model is a large language model (LLM).

10. The computer-implemented method of any one of claims 1-9, wherein the historical performance training data includes at least (i) one or more input impression featuresPATENT APPLICATIONAttorney Docket No.: 31730 / 308147-00 for a content item and (ii) an indication of an auction winner for an auction involving the content item.

11. The computer- implemented method of claim 10, wherein the one or more input impression features includes at least one of: (i) a language associated with the content item; (ii) a location from which a query for the content item originated; or (iii) a category of one or more categories associated with the location.

12. The computer- implemented method of claim 11, wherein the historical performance training data further includes a ranking for the content item in each category of the one or more categories.

13. The computer- implemented method of any one of claims 1-12, wherein the predicted performance training data includes at least a predicted ranking for the second content item in one or more potential categories.

14. The computer-implemented method of claim 13, wherein the content retrieval model is configured to determine whether to output the second content item based on the predicted ranking.

15. An apparatus, functioning as a server device, including: a transceiver; and one or more processors configured to perform a method according to any one of the preceding claims.

Images