Conditional Content Selection Procedure

Abstract

A system, method, and / or algorithm facilitates efficiently delivering content to client devices to fill content slots of an information resource having multiple content slots. A server receives a request from a client device for content to be included in the information resource and, in response, performs one or more content selection procedures to select one or more content items for one or more content slots. Based on the results of the content selection procedure(s), the server may conditionally perform or not perform one or more additional content selection procedures for one or more other content slots of the information resource. Alternatively, or in addition, the server may apply one or more constraints based on the format or content of one or more content items selected during the initial content selection procedure(s) to limit which content items may be candidates in one or more additional content selection procedures.

Description

[Technical Field]

[0001] TECHNICAL FIELD This disclosure relates to content selection procedures, and more particularly to procedures for selecting content to be provided to a client device for presentation to a user within a content slot of an information resource. [Background technology]

[0002] The background art provided herein is described for purposes of generally presenting the context of the present disclosure. The work of the inventor(s) named herein, to the extent described in this background art section, as well as aspects of the present disclosure that may not otherwise qualify as prior art at the time of filing, are not admitted expressly or impliedly as prior art to the present disclosure.

[0003] In some scenarios, such as when a user is visiting a web page or viewing a screen provided by a dedicated application, the user is presented with additional content that is dynamically added within a specific reserved area of ​​the page or screen. For example, a web page may reserve one or more "slots" for third-party content, and a client device may request that a server provide content items, such as digital advertisements or other third-party content items, to fill these slots. Typically, a client device sends a request for a given content slot to a server, causing the server to perform a procedure to select a specific content item. The server then provides the selected content item to the client device, which then fills the slot with the content item. For example, in the case of digital advertisements, the content selection procedure typically includes an auction. After some preliminary filtering of the advertisements (e.g., based on various targeting parameters), bid scores and ranks are determined for the remaining candidates. A winner of the auction is determined based on the rank, and the winning advertisement is used to fill the content slot.

[0004] When multiple content slots are available to be filled in a web page, user interface, or another information resource, various complexities may arise. For example, different slots and different content item formats may be associated with different priorities and / or may have dependencies (e.g., conflicts) on one another. As an example in the context of digital advertising, to increase or maintain the value of a prominent (e.g., top-of-page) advertisement slot, when a sponsored content item (ad) is selected to fill that slot, it may be desirable to avoid placing another advertisement in a nearby slot that may distract and / or annoy the user. In other examples, it may be preferable to avoid filling a single page with too many advertisements and / or to avoid placing competing advertisements (e.g., for similar products) in adjacent slots on a page. Thus, various decisions must be made regarding whether to select a particular advertisement, which advertisement slot to fill, and / or whether to apply certain constraints on which advertisements may be candidates in the advertisement selection process.

[0005] In some existing systems, this decision-making process is performed by conducting an auction for all available ad slots and then determining whether to drop a particular ad from among the auction winners. For example, in response to a request from a client device, a server may conduct a first auction for a first slot on a page and a second auction for a second slot on the same page. If the first auction has a specific outcome, such as the first slot being filled (or the first slot being filled with an advertisement for a particular product or brand), the winning advertisement for the second slot may be discarded. Otherwise, the winning advertisement for the second slot is used to fill the second slot. However, this approach has certain technical drawbacks. First, although an auction is performed for all requested slots (e.g., all slots on the page the user is visiting), only a subset of the winning advertisements may be served to the user's device, which typically results in a waste of processing resources. Specifically, processing resources (e.g., processing time, processing power) at the server are wasted by conducting auctions even for slots in which no advertisements are ultimately presented. Second, the "fill rate" of ad slots may decrease as ads in some slots may be dropped based on the results / winners of other slots, resulting in inefficient provision of ad content by the server (and therefore fewer impressions and less ad revenue).

[0006] In other systems, the decision-making process is implemented by the client device sequentially making multiple remote procedure calls (RPCs) to the server to fetch advertisements for different slots of the page. Under this approach, the client device makes a first RPC to the server for a first slot, then waits and considers the results / responses obtained from the server before making a second RPC to the server for a second slot. This approach also has significant drawbacks. First, because the client device must sequentially wait for results from the server for each RPC / slot, the delay between each request and the server response increases latency (e.g., web page load time), potentially to the point of being annoying to users. Second, because the client device must perform a significant amount of front-end processing (e.g., to determine whether to make a subsequent RPC based on the results of a previous RPC), the processing load can cause the client device to crash (e.g., a browser or other application crash), resulting in a poor user experience. This can be particularly problematic when the client device has relatively limited memory, as may be the case with smart TV devices. Third, because decisions are made on the client device, there is generally less information available to the server, which may hinder the server's ability to make future decisions (e.g., for future content selection) or provide useful analytical or problem-solving feedback (e.g., information indicating the performance of an advertisement or information indicating why an advertisement was not served). Summary of the Invention

[0007] In some implementations described herein, the server determines whether to perform one or more content selection procedures to fill a content slot(s) of an information resource and / or the manner in which to perform the content selection procedure(s) based on results from one or more previous content selection procedures used to fill other content slot(s) of the information resource.

[0008] In one exemplary implementation, a method for efficiently delivering content to a client device includes receiving, by a server and from the client device, a request for content to be included in an information resource having a plurality of content slots, and selecting, by the server, one or more content items in response to the request. Selecting the one or more content items includes (1) performing a first content selection procedure using selection parameters associated with at least the candidate content items to determine a first content selection result for a first subset of the plurality of content slots, and (2) conditionally performing or not performing a second content selection procedure for a second subset of the plurality of content slots based on the first content selection result. The method also includes causing, by the server, the client device to fill one or more content slots of the plurality of content slots with the selected one or more content items.

[0009] In another exemplary implementation, a method for efficiently delivering content to a client device includes receiving, by a server and from the client device, a request for content to be included in an information resource having a plurality of content slots, and selecting, by the server, a plurality of content items in response to the request. Selecting the plurality of content items includes (1) performing a first one or more content selection procedures using selection parameters associated with at least the candidate content items to select a first set of one or more content items for the plurality of content slots, and (2) applying, based on the format or content of the first set of content items, one or more constraints that limit which content items may be candidates in a second one or more content selection procedures for a second subset of the plurality of content slots. The method also includes causing, by the server, the client device to fill at least some of the plurality of content slots with the selected plurality of content items.

[0010] In some embodiments, the server includes one or more processors and one or more memories, the one or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the above embodiments. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram of an example system in which techniques for efficiently serving content to client devices may be implemented. [Figure 2] 1 illustrates a first exemplary information resource having content slots configured to display content items selected by a content selection procedure. [Figure 3]10 illustrates a second exemplary information resource having content slots configured to display content items selected by a content selection procedure. [Figure 4] 2 is a message sequence diagram illustrating an exemplary implementation in which the content server of FIG. 1 selects content for the client device of FIG. 1 and provides the selected content to the client device. [Figure 5] 1 is a flow diagram of an example method for efficiently serving content to client devices. [Figure 6] 1 is a flow diagram of another exemplary method for efficiently serving content to client devices. DETAILED DESCRIPTION OF THE INVENTION

[0012] Generally, in aspects of the disclosed invention, a server executes conditional content selection procedures. Specifically, in response to receiving a request for content from a client device, the server executes one or more content selection procedures to fill one or more content slots of an information resource (e.g., a web page or application screen / user interface displayed on the client device), and for one or more additional content slots of the information resource, the server conditions the execution of the content selection procedure(s) on the results of previous content selection procedure(s).

[0013] In certain example implementations and scenarios described herein, the content items used to fill the content slots are advertisements and / or the corresponding content selection procedure is an auction. However, the content items may instead be any other type of content that can be dynamically selected for presentation within an information resource, and the content selection procedure may instead be any other type of procedure suitable for selecting content (e.g., a procedure that selects content based on the relevance scores of the content items).

[0014] In one aspect of the invention, a server conditionally performs or does not perform one or more content selection procedures based on the results of one or more previous content selection procedures for the same information resource slot. In some embodiments, for example, the server performs a second content selection procedure only if a first content selection procedure does not result in the selection of any content items for a particular content slot. In other embodiments, the server conditionally performs or does not perform a second content selection procedure based on one or more characteristics of the content items selected during the first content selection procedure. For example, the server may use a machine learning model to determine the relevance score of the content items selected during the first content selection procedure, and then perform the second content selection procedure only if the relevance score exceeds some predetermined threshold.

[0015] In another aspect of the invention, the server may alternatively or additionally condition how one or more content selection procedures are performed on the results of one or more previous content selection procedures for the same information resource slot. In particular, the server may apply one or more constraints based on the format and / or content of the content items selected during the first content selection procedure, the constraint(s) serving to limit which content items may be candidates for the second content selection procedure. In various implementations, for example, the server may exclude content items from the second content selection procedure based on the category or format of those content items (e.g., based on whether the category or format of the content item is the same as the content item resulting from the first content selection procedure), based on user cost, based on the level of similarity between the content item and the content item selected during the first content selection procedure, and / or based on other factors.

[0016] For either of the above two aspects, in some embodiments, the server may sort the content slots of the information resource according to their priority levels to determine the order in which the server should attempt to fill the content slots before the server performs the first content selection procedure. In some embodiments, at least some of the content slots of the information resource may be grouped together, in which case the server may perform content selection procedures for content items in a given group at least partially in parallel with each other.

[0017] These and other aspects may improve the efficiency of the content selection procedure (e.g., use fewer processing resources), reduce latency (e.g., reduce web page load time), prevent "crashes" on client devices (e.g., when the client device is a particularly resource-constrained device), and / or provide other technical advantages.

[0018] For example, in the first aspect (in which the server conditionally performs or does not perform one or more additional content selection procedures for an information resource), (1) the server uses fewer processing resources than if the server had automatically performed the content selection procedures for all content slots of a multi-slot information resource (e.g., based on a single request from a client device), (2) latency or delay at the client device is reduced compared to if the client device had to send a series of individual requests to the server for various slots (e.g., when loading a web page with multiple content slots), and / or (3) the client device is less likely to crash compared to if the client device had to process the responses to each of a series of individual requests sent to the server to determine whether to send the next request.

[0019] In the second aspect (in which the server conditions the manner in which it performs one or more additional content selection procedures based on the results of one or more previous content selection procedures for the same information resource), (1) the fill rate of the content item slots is typically improved compared to when the server automatically performs the content selection procedures for all content slots of a multi-slot information resource (e.g., based on a single request from the client device), (2) the latency or delay at the client device is reduced (e.g., when loading a web page containing multiple content slots) compared to when the client device must send a series of individual requests to the server for the various slots, and / or (3) the likelihood of the client device crashing is reduced compared to when the client device must process the responses to each of a series of individual requests sent to the server to determine whether to send the next request.

[0020] Furthermore, in both of these aspects, more information is generally available to the server than if the client device made multiple content requests and analyzed each request result / response for a given information resource slot. Such information may help the server make more informed decisions (e.g., for future content selection) and / or may enable the server to provide useful analytical or problem-solving feedback (e.g., information indicating advertisement performance, or information indicating why an advertisement was not served, etc.).

[0021] 1 illustrates an example system 100 in which one or more techniques for efficiently serving content to client devices 102 may be implemented. The example system 100 includes a client device 102, a content server 104, several publishers 106, several content providers 108, and a network 110. The content server 104 is remote from the client device 102 and communicatively coupled to the client device 102 via the network 110. The communication / network connections illustrated in FIG. 1 for the publishers 106 and content providers 108 represent communication / network connections with computing devices or systems associated with the publishers 106 and content providers 108, respectively.

[0022] Network 110 may be a single communications network (e.g., the Internet), or in some implementations, may also include one or more additional networks. As just one specific example, network 110 may include a cellular network, the Internet, and a server-side local area network (LAN). While Figure 1 shows only a single client device 102, it will be understood that content server 104 may also communicate with several other client devices that are generally similar to client device 102.

[0023] Generally, client devices 102 can access information resources provided or published by publishers 106, and content servers 104 select content items provided by content providers 108 to fill content slots within those information resources. For example, the information resources can be web pages of a website hosted by the publisher 106, and the content providers 108 can provide advertisements or other content items that the content server 104 can then select to fill content slots within those web pages. As another example, the information resources can be screens or user interfaces of applications (e.g., mobile apps) provided by the publisher 106, and the content providers 108 can provide advertisements or other content items that the content server 104 can use to fill content slots within those user interfaces. As yet another example, the information resource can be a video, and content slots can be distributed in time throughout the video. However, in some implementations, multiple (some or all) content slots of an information resource exist simultaneously in the information resource without temporal distribution. The content items themselves may be text, images, or video, or any combination thereof, and may or may not include an audio component.

[0024] Client device 102 may be or include any stationary, mobile, or portable computing device with wired and / or wireless communication capabilities (e.g., a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart wearable device such as smart glasses or a smart watch, a vehicle head unit computer, etc.). In the exemplary implementation of FIG. 1, client device 102 includes a network interface 120, a processor 122, a memory 124, and a display 126. Processor 122 may be a single processor (e.g., a central processing unit (CPU)) or may include a set of processors (e.g., multiple CPUs, or one or more CPUs and one or more graphics processing units (GPUs)).

[0025] Memory 124 includes one or more computer-readable, non-transitory storage units or devices, which may include persistent (e.g., hard disk) and / or non-persistent memory components. Memory 124 stores instructions executable by processing unit 122 to perform various operations, including instructions for various software applications and data generated and / or used by such applications. In the exemplary implementation of FIG. 1, memory 124 stores at least application 130, which may be, for example, a web browser application (e.g., Chrome) or a mobile application downloaded from an app store.

[0026] Generally, application 130 is executed by processor 122 to present information resources to a user of client device 102 via display 126, at least one of which includes multiple content slots for dynamically presenting content items. In embodiments in which application 130 is a web browsing application, for example, the information resource may be a web page hosted by one of publishers 106, and the browser may cause client device 102 to download HTML, scripts, and / or other code for the web page for presentation to the user via display 126. As another example, application 130 may be a video sharing application, such as YouTube, and the information resource may be a user interface generated by the video sharing application and presented via display 126. As yet another example, application 130 may be a video playback application, and the information resource may be a video played by the video playback application.

[0027] Display 126 includes hardware, firmware, and / or software configured to allow a user to view the visual output of client device 102 and may use any suitable display technology (e.g., LED, OLED, LCD, etc.). In some implementations, display 126 is incorporated into a touchscreen that has both display and manual input capabilities. Furthermore, in some implementations where client device 102 is a wearable device, display 126 is a transparent visibility component of an integrated electronic component (e.g., a lens of smart glasses). For example, display 126 may include microLED or OLED electronics embedded in the lens of smart glasses.

[0028] Network interface 120 includes hardware, firmware, and / or software configured to enable client device 102 to exchange electronic data with content server 104 over network 110. For example, network interface 120 may include a cellular communication transceiver, a WiFi transceiver, and / or a transceiver for one or more other wired and / or wireless communication technologies.

[0029] 1 depicts client device 102 as a single component that communicates directly with content server 104 (i.e., via network 110), in some implementations, the subcomponents of client device 102 depicted in FIG. 1 are instead split into two or more user-side devices. As just one example, smart glasses may include processor 122, memory 124, and display 126, while a smartphone may include another processing unit, another memory, another display, and network interface 120. The smart glasses (or smart helmet, etc.) may then communicate with the smartphone (e.g., via Bluetooth) as needed to enable the operations described herein.

[0030] The content server 104 includes a network interface 140, a processor 142, and memory 144. The network interface 140 includes hardware, firmware, and / or software configured to enable the content server 104 to exchange electronic data with the client device 102 and other similar client devices over the network 110. For example, the network interface 140 may include a wired or wireless router and modem. The processor 142 may be a single processor or may include multiple processors. As used herein, the term "server" may refer to a single computing device at a single location or multiple coordinating computing devices that are either co-located or remotely distributed.

[0031] Memory 144 is a computer-readable, non-transitory storage unit or device, or collection of units / devices, which may include persistent and / or non-persistent memory components. Memory 144 stores instructions for content selector 150 that may be executed by processor 142. Content selector 150 includes conditional content selection procedure (CSP) module 152, which includes slot priority module 154, content filter 156, and relevance module 158. One, some, or all of modules 154, 156, and 158 may optionally be omitted from system 100.

[0032] Generally, content selector 150 selects content items that content server 104 serves / transmits to client device 102 (or other client devices) over network 110. Content selector 150 selects particular content items using one or more content selection procedures implemented by conditional CSP module 152. Content filter 156 can apply one or more constraints to limit which content items can serve as candidates for selection by conditional CSP module 152 based on the results of one or more previous content selection procedures (described further below) and / or based on one or more other factors (e.g., user preferences).

[0033] The content selection procedure performed by the conditional CSP module 152 may, for example, be an auction, with the conditional CSP module 152 selecting a winner based on the bids that the various content providers 108 communicate to the content server 104 or that the content server 104 otherwise generates for the various content providers 108 (e.g., based on input from the content providers 108, such as a preferred maximum bid). As a more specific example, each content provider 108 may submit a maximum bid for placement of its content item, and the conditional CSP module 152 may select the content item associated with the highest-bidding content provider 108. The content server 104 may then charge the winning content provider 108 the highest bid (or the bid of the second-highest bidder, etc.) and select the content item associated with that content provider 108 to fill the content slot under consideration.

[0034] The content selection procedure may additionally or alternatively take into account selection parameters other than the bid amount, such as the relevance of a particular content item to the user of the client device 102 (or relevance to an information resource, etc.), as determined by the relevance module 158 and / or user preferences, etc. The content selection procedure may be entirely rules-based or may have one or more random components (e.g., random selection from a candidate pool after filtering by the content filter 156), etc.

[0035] At least some of the content selection procedures performed by the conditional CSP module 152 have conditional aspects. In some implementations, for example, execution of a content selection procedure for a given content slot(s) is conditioned on the results of content selection procedures for one or more other content slots of the same information resource. Additionally or alternatively, in some implementations, the content filter 156 may conditionally apply or not apply one or more constraints that limit which content items may be candidates for selection for one or more content slots based on the results of content selection procedures for one or more other content slots of the same information resource. In some implementations, the slot priority module 154 determines a priority / ordering for individual content slots or groups / subsets of content slots, and the conditional CSP module 152 executes the conditional content selection procedure for one, some, or all of the content slots or content slot subsets according to the determined order.

[0036] The operation of the conditional CSP module 152 and its constituent modules is described in further detail below in connection with various exemplary implementations.

[0037] Content items provided (e.g., created and / or sourced) by content providers 108 and available for selection by content selector 150 may, in some implementations, be stored in content database 160. Content database 160 may be stored in memory 144 or, for example, in one or more memories that are remote from content server 104.

[0038] In some implementations, publishers 106 and / or content providers 108 maintain accounts associated with services provided by content server 104. For example, publishers 106 may create such accounts in order to monetize information resources that the publishers publish or otherwise make available (e.g., by selling advertisements in content slots on web pages that the publishers host), and / or content providers 108 may create such accounts to locate and purchase content slots where presenting their content (e.g., advertisements) would be particularly advantageous. In these implementations, information related to publisher and / or content provider accounts may be stored in account database 162. Account database 162 may be stored in memory 144 or, for example, in one or more memories that are remote from content server 104. Account information may include information such as entity name, subscription level, entity preferences (e.g., brand control preferences), etc. In some implementations, the account information includes selection parameters (e.g., bid amounts or maximum bid amounts) associated with different content providers 108 that are used by the conditional CSP module 152 in selecting content as described elsewhere herein.

[0039] In some implementations, system 100 is arranged differently than shown in Figure 1. For example, content server 104 may be associated with the same entity that provides information resources, in which case system 100 may omit publisher 106 (e.g., activities described herein for publisher 106 may instead be performed by content server 104). Additionally or alternatively, content server 104 may be associated with the same entity that sources content items to fill content slots in information resources, in which case system 100 may omit content provider 108 (e.g., activities described herein for content provider 108 may instead be performed by content server 104).

[0040] Figures 2 and 3 show exemplary information resources having content slots from which content selector 150 of Figure 1 is configured to select content items. The information resource of Figure 2 or 3 may be sourced (e.g., hosted) by one of publishers 106 and presented on display 126 by application 130 after client device 102 downloads the information resource (e.g., from publisher 106) via network 110, network interface 120, etc.

[0041] 2 illustrates information resource 200, a user interface for a video sharing application (e.g., YouTube), which may be provided as a web page (e.g., if application 130 is a web browser) or as a screen of a dedicated application (e.g., if application 130 is a dedicated video sharing application). Information resource 200 includes a primary or "masthead" content slot 202 and secondary content slots 204A-D, with primary content slot 202 being more prominent than each of secondary content slots 204A-D both by virtue of its higher position within information resource 200 and by virtue of having larger dimensions (height and / or width).

[0042] In some implementations where application 130 is a web browser application and information resource 200 is a web page hosted by one of publishers 106, application 130 downloads and executes the web page code (e.g., HTML code) to present the web page via display 126. The web page code may also include script (e.g., JavaScript) that, when interpreted by application 130, causes client device 102 to request content from content server 104 to fill or conditionally fill at least some (i.e., two or more) of content slots 202 and 204A-D. In some implementations where application 130 is a dedicated application that instead generates information resource 200, application 130 causes client device 102 to request content from content server 104 when information resource 200 is initially generated (e.g., when application 130 is launched). In either case (web page or dedicated application screen), the request sent to content server 104 may be a single remote procedure call (RPC), as described below in connection with FIG. 4.

[0043] FIG. 3 illustrates an information resource 300 that is a search engine user interface, which may be provided as a web page (e.g., if the application 130 is a web browser) or as a screen of a dedicated application (e.g., if the application 130 is a search engine mobile application). The information resource 300 includes multiple sponsored content slots 320A-C, with each successive slot further down in the list presented in the information resource 300, followed by a number of content slots 330A-F representing the results of a search. The search may be based on a text string query entered by a user in the query field 302. In various implementations, the search results (e.g., uniform resource locator (URL) links) used to fill the content slots 330A-F may be selected by the content server 104 or a server other than the content server 104 (e.g., a dedicated search engine server) using any suitable search technique or algorithm.

[0044] In some implementations in which application 130 is a web browser application and information resource 300 is a web page, application 130 downloads and executes the web page's code (e.g., HTML code) to present the web page via display 126. The web page code may also include script (e.g., JavaScript) that, when interpreted by application 130, causes client device 102 to request content from content server 104 to fill or conditionally fill at least content slots 320A-C. In some implementations in which application 130 is a dedicated application that instead generates information resource 300, application 130 causes client device 102 to request content from content server 104 when information resource 300 is initially generated (e.g., when application 130 is launched). In either case (web page or dedicated application screen), the request sent to content server 104 may be a single RPC, as described below in connection with FIG. 4.

[0045] In some implementations, information resource 300 presents results in a manner different from that shown in Figure 3. For example, a search engine may utilize generative artificial intelligence (e.g., based on large language models (LLMs)), and information resource 300 may provide paragraph-style text responses to a single user's query instead of supplying an ordered list of search results. In one such implementation, sponsored content slots 320A-C are interspersed with the generated text responses at fixed or dynamically changing locations.

[0046] In some implementations, content slots 202 and 204A-D of Figure 2, or content slots 320A-C of Figure 3, are all configured to display the same general type of content (e.g., video only, images only, text only, text and images, etc.). In other implementations, different content slots of the same information resource may display different types of content depending on what content is selected for the slot by content server 104. For example, content selector 150 may be constrained (by content filter 156) to select only video content items for content slot 202, but may be allowed to select image (or image and text, etc.) or video (or video and text, etc.) content items for the remaining content slots 204A-D.

[0047] In some implementations, when one of content slots 202, 204A-D, 320A-C is filled with a content item, a user of client device 102 can select the content item using an input device of client device 102 (e.g., by tapping a touchscreen, clicking a mouse button, etc.) and be redirected to a page or other information resource associated with the selected content item. For example, if the content item is an advertisement filling content slot 202 or 204A of FIG. 2 or content slot 320A of FIG. 3, when the user selects the content item, application 130 can direct the user to a landing page hosted by the content provider's server. The landing page may enable the user to navigate to a page where, for example, an item advertised by the content item can be purchased. In implementations where the initial information resource (200 or 300) is a web page, this may include redirecting the user from the URL of the initial web page to the content provider's URL / web page. In embodiments where the initial information resource (200 or 300) is a dedicated application screen, this may involve automatically opening a web browser application and opening the URL of the content provider's web page.

[0048] 2 and 3 are for illustrative purposes only, and it should be understood that the techniques described herein may instead be applied to information resources having different numbers of content slots, different sized content slots, different relative positions of content slots, etc. Additionally, in some implementations and / or scenarios, a user may need to navigate (e.g., by scrolling or panning) through the entire information resource to view a particular content slot. In some implementations, multiple content slots are delivered over time (e.g., if the information resource is a video, multiple slots are interrupted for video advertisement breaks).

[0049] A manner in which the content selector 150 selects content items to fill / populate the content slots 202, 204A-D of FIG. 2 or the content slots 320A-C of FIG. 3, according to some implementations, will now be described with reference to FIG. 4. FIG. 4 illustrates an exemplary scenario 400 in which the content server 104 of FIG. 1 selects and serves content to the client device 102 of FIG. 1. In scenario 400, the client device 102 accesses (402) an information resource having multiple content slots (e.g., information resource 200 or 300). In implementations in which the information resource is a web page hosted by one of the publishers 106, event 402 may include the client device 102 accessing the web page (perhaps after a user has navigated the publisher's 106's website) by using the web page's URL and receiving, at least in part, the web page's code (e.g., HTML, JavaScript, etc.) from the publisher 106 over the network 110. In an embodiment where the information resource is a user interface / screen of a (dedicated) application 130, the event 402 may include the client device 102 launching the application 130 and possibly navigating to a particular user interface provided by the application 130.

[0050] After (e.g., automatically) accessing (402) an information resource (and in response to) the access, the client device 102 (e.g., application 130) requests (404) content from the content server 104 to fill a content slot in the information resource. The request may consist of a single RPC. In some implementations where the information resource is a web page, the event 404 may be automatically triggered by the application 130 in response to the application 130 interpreting script embedded in the web page code received from the server hosting the web page. In some implementations where the information resource is a user interface / screen of a dedicated application (application 130), the event 404 may be automatically triggered by the application 130 in response to the user opening or launching the application 130 or in response to the user navigating to a particular user interface / screen of the application 130. Other triggers for the event 404 are also possible.

[0051] In response to receiving the request, the content server 104 (e.g., the content selector 150) processes the request (410). Event 410 may include, for example, reformatting the request message and / or determining the capabilities of the client device 102 based on the request message. In some implementations, event 410 is omitted. The slot priority module 154 sorts (412) the content slots of the information resource (e.g., slots 202 and 204A-D of FIG. 2 or slots 320A-C of FIG. 3) according to slot priority. Sorting (412) functions to establish the order in which the conditional CSP module 152 executes or considers executing a content selection procedure for a particular content slot or a particular subset of content slots. The slot priority module 154 may sort (412) the slots or slot subsets of the information resource based on priority information (e.g., in the request message of event 404) sent by the client device 102 to the content server 104. The client device 102 may have obtained the priority information from, for example, an information resource, from information associated with each content slot or slot subset (e.g., a flag or field value), and / or from the application 130. In some implementations, the order / priority is fixed and event 412 is omitted.

[0052] The priority information may reflect the desirability of the content slot (e.g., salience of location, size, etc.). For information resource 200, for example, content slot 202 may have the highest priority, content slot 204A may have the second highest priority, content slot 204B may have the third highest priority, content slot 204C may have the fourth highest priority, and content slot 204D may have the fifth highest priority. Alternatively, content slot 202 may have the highest priority, but content slots 204A-D may be grouped together and all have the same second highest priority. For information resource 300, content slot 320A may have the highest priority, content slot 320B may have the second highest priority, and content slot 320C may have the third highest priority. Other priorities and / or groupings are possible.

[0053] At event 414, the conditional CSP module 152 performs a content selection procedure (e.g., an auction or a relevance-based selection procedure, etc.) on the highest priority content slot (e.g., content slot 202 or 320A), or performs a first set of content selection procedures for each slot of a subset of the highest priority content slot (e.g., content slots 302A and 302B). The conditional CSP module 152 then processes (416) the results of the content selection procedure(s) performed at event 414 to determine how or whether to perform a conditional selection procedure on the next highest priority content slot or slot subset at event 420.

[0054] In some implementations, at event 416, conditional CSP module 152 determines whether or not to perform event 420 (and possibly any other subsequent similar operations, such as event 424) based on the results. For example, conditional CSP module 152 may (1) determine not to perform a content selection procedure (e.g., an auction) for content slot 204A if the content selection procedure at event 414 results in any type of content item being selected for content slot 202, but (2) determine that the content selection procedure should be performed if the content selection procedure at event 414 does not result in any type of content item being selected for content slot 202. As another example, conditional CSP module 152 may (1) determine that if the content selection procedure at event 414 results in a content item being selected for content slot 202, then the content selection procedure should not be performed for any of content slots 204A-D, but (2) determine that if the content selection procedure at event 414 does not result in a content item being selected for content slot 202, then the content selection procedure should be performed for each of content slots 204A-D. As yet another example, conditional CSP module 152 may (1) determine that if the content selection procedure at event 414 results in a content item being selected for content slot 320A, then the content selection procedure should not be performed for content slot 320B, but (2) determine that if the content selection procedure at event 414 does not result in a content item being selected for content slot 320A, then the content selection procedure should be performed.

[0055] The content selection procedure performed by the conditional CSP module 152 may not result in a content item being selected for a particular content slot for various reasons, depending on the implementation and scenario. For example, the content selection procedure may result in the content item being rejected by the content server 104 due to security concerns associated with the content item (e.g., suspicious scripts in the content item), or due to incompatibility with the preferences of the user of the client device 102 or the publisher 106 (e.g., if the content filter 156 applies certain constraints after the content selection procedure is complete).

[0056] In other embodiments, the processing at event 416 analyzes other aspects of the results of the content selection procedure(s) at event 414 when determining whether to execute event 420 (and / or subsequent events in FIG. 4 ). For example, conditional CSP module 152 may determine that if the content selection procedure for content slot 202 at event 414 results in a content item being selected with a bid above a certain minimum amount, then the content selection procedure will not be executed for any of content slots 204A-D. In another embodiment, conditional CSP module 152 may determine whether to execute the content selection procedure(s) at event 414 based on one or more characteristics of the one or more content items selected at event 414. For example, conditional CSP module 152 may determine that if the content selection procedure at event 414 results in a content item of a particular type (e.g., video) being selected for content slot 202, then the content selection procedure will not be executed for any of content slots 204A-D at event 420.

[0057] In another exemplary implementation, the conditional CSP module 152 conditionally executes or does not execute the content selection procedure(s) at event 420 based on the relevance score of at least one content item selected during the content selection procedure(s) at event 414. For example, the relevance module 158 (or another module / software, or possibly another server) may use a machine learning model (e.g., a neural network) to determine a relevance score for each of the one or more content items selected at event 414, and the relevance module 158 may execute the content selection procedure(s) at event 420 only if the determined relevance score (or each of a certain number of relevance scores) exceeds some predetermined threshold. The relevance score may indicate, for example, relevance to a user's search query (e.g., a text string), relevance to an information resource, etc.

[0058] As a more specific example, in a scenario in which information resource 200 is presented to a user of client device 102 in response to the user's video search query, relevance module 158 may use a machine learning model to calculate a relevance or search score for a content item selected by content selector 150 (at event 414) to fill content slot 202. If the relevance / search score is greater than some threshold (meaning the content item is particularly relevant to the search), conditional CSP module 152 may perform another content selection procedure (e.g., an auction) at event 420 to fill content slot 204A (or perform such a procedure for each of slots 204A-D). However, if the content item's search score is not greater than the threshold, conditional CSP module 152 may not perform the content selection procedure(s) at event 420, thereby reducing the likelihood of confusing the user with irrelevant or useless information.

[0059] In some implementations, at event 416, conditional CSP module 152 alternatively (or additionally) determines the manner in which event 420 is executed based on the results of the content selection procedure(s) at event 414. In particular, content filter 156 may apply one or more constraints to limit which content items (from content database 160) can serve as candidates in the content selection procedure(s) at event 420 based on the format or content of the content item(s) selected at event 414.

[0060] For example, the conditional CSP module 152 can exclude content items from consideration at event 420 based on one or more categories of the content items selected at event 414, a user cost associated with one or more of the selected content items, one or more formats of the selected content items (e.g., excluding content items of the same format as the selected content item), and / or a level of similarity to one or more of the selected content items. The level of similarity can be indicated by a similarity score. The content server 104 (e.g., the content selector 150) can determine the similarity score for a pair of content items using machine learning techniques, such as a neural network or other machine learning model trained on the content item pair using supervised or unsupervised learning techniques. The machine learning model can operate on (accept as input) specific parameters indicative of the characteristics of the content items (e.g., the content item's format, category, length, etc.) or can operate directly on, for example, image, text, or video frame information. In some implementations, the content server 104 uses generative artificial intelligence (e.g., a large-scale language model (LLM)) to determine the level of similarity between two content items, both of which contain text.

[0061] Applying constraints based on similarity to previously selected content items and / or based on the category or format of previously selected content items can avoid scenarios in which content items of a particular type / format, etc., are over-selected, providing various benefits. For example, if a content slot corresponds to an in-stream video advertising break, it may be desirable to prevent a client device user from being subjected to multiple long video ads (e.g., spanning a threshold length, such as 60 seconds). Thus, if a long video ad is selected in the first break, content filter 156 may exclude (as candidates for the content selection procedure(s) at event 420) any long video ads (e.g., those exceeding a threshold length) in one or more subsequent advertising breaks. As another example, content filter 156 may be useful in enforcing brand safety constraints related to advertisements or content providers. For example, content filter 156 may exclude (as candidates for the content selection procedure(s) at event 420) any ads associated with a competitor of the content provider whose advertisement was selected in event 414. A list of competitors may be associated with each content provider's 108 account in account database 162, or content server 104 may access other information (e.g., categories of goods or services sold as indicated in databases 160 or 162) to identify likely competitors.

[0062] In some implementations, content filter 156 uses a machine learning model to apply its constraints to the content items. For example, event 420 may include applying one or more parameters indicative of the type or content of the content item(s) selected in event 414 as input to a machine learning model (such as a neural network) and deriving at least one of the constraints from the output of the machine learning model.

[0063] 4, additional events similar to events 416 and / or 420 may occur or conditionally occur if there is a subsequent priority level for the content slot (or slot subset) as determined by slot priority module 154. For example, conditional CSP module 152 may process the results of the content selection procedure(s) for event 420 in event 422 (e.g., similar to event 416), conditionally perform or not perform (or conditionally apply constraints on the execution of) one or more further content selection procedures in event 424 (e.g., similar to event 420), etc.

[0064] In some implementations where certain content slots (e.g., slots 204A-D) are grouped together into subsets for prioritization / sorting purposes, the conditional CSP module 152 performs content selection procedures on all content slots within the group / subset in parallel (i.e., simultaneously) or at least partially in parallel (i.e., at least partially overlapping in time) to reduce latency associated with filling content slots of an information resource.

[0065] After events 410-424 are completed (or, as noted above, may not be executed at all, in the case of events 420-424), the content server 104 responds (e.g., returns an RPC response) to the client device 102 (430). The response for event 430 may include the content item(s) selected during one or more of the content selection procedures for events 414, 420, and 424, or may include links to the selected content item(s). At event 430, the client device 102 stores the selected content item(s) in the appropriate content slot(s) of the information resource. In some implementations, the client device 102 does not need to make any RPCs other than the single RPC for event 404 to obtain the content item(s) to fill the slots of the information resource.

[0066] Exemplary methods for efficiently selecting content for a client device will now be described with reference to Figures 5 and 6. Either or both of the methods of Figures 5 and 6 may be implemented as instructions stored on one or more computer-readable media and executed by one or more processors of one or more computing devices. For example, the method(s) may be performed by processor 142 of content server 104 of Figure 1 when executing instructions of content selector 150.

[0067] Referring initially to method 500 of FIG. 5 , a request for content to be included in an information resource having a plurality of content slots is received from a client device (e.g., client device 102) at block 502 (e.g., event 404). One or more content items are selected at block 504 in response to the request. Block 504 includes determining a first content selection result for a first subset of the plurality of content slots (e.g., event 414) by performing a first one or more content selection procedures using at least selection parameters (e.g., bid amount, relevance score, etc.) associated with the candidate content items. Block 504 also includes conditionally performing or not performing a second one or more content selection procedures for a second subset of the plurality of content slots based on the first content selection result. In some implementations, the first subset of the plurality of content slots consists of only a single content slot, and / or the second subset of the plurality of content slots consists of only a single content slot. In other implementations, one or both subsets include multiple content slots.

[0068] The client device is caused to fill one or more of the plurality of content slots with the selected one or more content items at block 506. Block 506 may correspond to, for example, event 430.

[0069] In some implementations, block 504 includes refraining from (not performing) the second content selection procedure(s) when selecting a first set of one or more content items to fill a first subset of the plurality of content slots, and instead performing the second content selection procedure(s) when no content items are selected to fill the first subset of the plurality of content slots.

[0070] In some implementations, determining the first content selection result at block 504 includes selecting a first set of one or more content items to fill a first subset of the plurality of content slots, and selecting the one or more content items at block 504 includes performing or not performing a second content selection procedure(s) conditionally based on one or more characteristics of the first set of content items. The one or more characteristics of the first set of content items may include one or more relevance scores (e.g., determined by the relevance module 158 using a machine learning model) associated with the first set of content items.

[0071] In some implementations, method 500 also includes determining that the first subset has a higher priority than the second subset before selecting one or more content items, and performing a first content selection procedure(s) at block 504 in response to this determination.

[0072] In some embodiments, the first subset of the plurality of content slots includes at least a first content slot and a second content slot, the first content selection procedure(s) includes a third content selection procedure for the first content slot and a fourth content selection procedure for the second content slot, and performing the first content selection procedure(s) includes performing the third content selection procedure at least partially in parallel with performing the fourth content selection procedure.

[0073] Referring now to method 600 of FIG. 6, a request for content to be included in an information resource having a plurality of content slots is received from a client device (e.g., client device 102) at block 602 (e.g., event 404). At block 604, a plurality of content items are selected in response to the request. Block 604 includes selecting a first set of one or more content items for a first subset of the plurality of content slots (e.g., slot 202, slot 320A, or slots 320A and 320B, etc.) by performing a first one or more content selection procedures (e.g., event 414) using at least selection parameters (e.g., bids, relevance scores, etc.) associated with the candidate content items. Block 604 also includes applying one or more constraints that limit which content items may be candidates in a second one or more content selection procedures for a second subset of the plurality of content slots based on the format or content of the first set of content items. In some embodiments, the first subset of the plurality of content slots consists of only a single content slot and / or the second subset of the plurality of content slots consists of only a single content slot, while in other embodiments, one or both subsets include multiple content slots.

[0074] The client device is caused to fill at least some of the plurality of content slots with the selected content item at block 606. Block 606 may correspond to, for example, event 430.

[0075] In some implementations, applying one or more constraints includes excluding content items from the second content selection procedure(s) based on a category of the content item (e.g., excluding content items associated with at least one category of content items in the first set of content items), excluding content items from the second content selection procedure(s) based on a cost to a user associated with at least one content item in the first set of content items, excluding content items having the same format as at least one content item in the first set of content items, and / or excluding content items based on a similarity score indicating a level of similarity to at least one content item in the first set of content items. In the latter case, method 600 may further include determining the similarity score using a machine learning model.

[0076] In some embodiments, applying the constraint(s) at block 604 includes applying one or more parameters indicative of the format or content of the first set of content items as input to a machine learning model, and deriving at least one of the constraint(s) from the output of the machine learning model.

[0077] In some implementations, method 600 also includes determining that the first subset has a higher priority than the second subset before selecting the content items at block 604, and performing the first content selection procedure(s) at block 604 is in response to determining that the first subset has the higher priority.

[0078] In some embodiments, the first subset of content slots includes at least a first content slot and a second content slot, the first content selection procedure(s) includes a third content selection procedure for the first content slot and a fourth content selection procedure for the second content slot, and executing the first content selection procedure(s) in block 604 includes executing the third content selection procedure at least partially in parallel with executing the fourth content selection procedure.

[0079] In some embodiments, the techniques disclosed herein use artificial intelligence to facilitate efficient selection of content items to fill content slots in information resources. Artificial intelligence (AI) is a segment of computer science that focuses on creating models that can perform tasks with little or no human intervention. Artificial intelligence systems can utilize, for example, machine learning, natural language processing, and computer vision. Machine learning and its subsets, such as deep learning, focus on developing models that can infer outputs from data. Outputs can include, for example, predictions and / or classifications. Natural language processing focuses on analyzing and generating human language. Computer vision focuses on analyzing and interpreting images and videos. Artificial intelligence systems can include generative models that generate new content, such as images, videos, text, audio, and / or other content, in response to input prompts and / or based on other information.

[0080] Exemplary machine learning models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some exemplary machine learning models can utilize attention mechanisms such as self-attention. For example, some machine learning models can include multi-head self-attention models (e.g., Transformer models).

[0081] The model(s) may be trained using various training or learning techniques. Training may involve supervised learning, unsupervised learning, reinforcement learning, etc. Training may involve techniques such as backpropagation of errors. For example, a loss function may be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent may be used to iteratively update parameters over several training iterations. Several generalization techniques (e.g., weight decay, dropout, etc.) may be used to improve the generalization ability of the trained model.

[0082] Prior to domain-specific alignment, the model(s) can be pre-trained. For example, the model can be pre-trained on a general corpus of training data and fine-tuned to a more targeted corpus of training data. The model can be aligned using prompts designed to elicit domain-specific outputs. The prompts can be designed to include learned prompt values ​​(e.g., soft prompts). The trained model(s) can be validated before use using input data other than the training data and can be further updated or refined during use based on additional feedback / input.

[0083] In some implementations, the content server 104 (e.g., the conditional CSP module 152) may use any one or more of the above machine learning models to perform any one or more of the operations discussed herein in connection with machine learning. For example, the content server 104 may use one or more such machine learning models to determine relevance scores for content items, determine similarity scores between content items, and / or determine constraints applied by the content filter 156, as described above.

[0084] While the foregoing text describes in detail many different aspects and embodiments of the present invention, it should be understood that the scope of the patent is defined by the language of the claims at the end of this patent. The detailed description is to be construed as illustrative only and does not describe every possible embodiment, as describing every possible embodiment would be impractical, if not impossible. Using current technology, or technology developed after the filing date of this patent, numerous alternative embodiments can be implemented and still fall within the scope of the claims. The disclosure herein contemplates at least the following examples:

[0085] Example 1. A method for efficiently providing content to a client device, comprising: receiving, by a server and from the client device, a request for content to be included in an information resource having a plurality of content slots; in response to the request, selecting, by the server, one or more content items, including (i) determining a first content selection result for a first subset of the plurality of content slots by performing a first one or more content selection procedures using selection parameters associated with at least candidate content items; and (ii) conditionally performing or not performing a second one or more content selection procedures for a second subset of the plurality of content slots based on the first content selection result; and causing the server to cause the client device to fill one or more content slots of the plurality of content slots with the selected one or more content items.

[0086] Example 2. The method described in Example 1, wherein selecting the one or more content items includes not performing the second one or more content selection procedures when selecting a first set of one or more content items to fill the first subset of the plurality of content slots, and performing the second one or more content selection procedures when not selecting any content items to fill the first subset of the plurality of content slots.

[0087] Example 3. The method of Example 1, wherein determining the first content selection result includes selecting a first set of one or more content items to fill the first subset of the plurality of content slots, and selecting the one or more content items includes conditionally performing or not performing the second one or more content selection procedures based on one or more characteristics of the first set of content items.

[0088] Example 4. The method of example 3, wherein the one or more characteristics of the first set of content items include one or more relevance scores associated with the first set of content items.

[0089] Example 5. The method of Example 4, further comprising determining, by the server, the one or more relevance scores using a machine learning model.

[0090] Example 6. The method of any one of Examples 1 to 5, further comprising, prior to selecting the one or more content items, determining by the server that the first subset has a higher priority than the second subset, and performing the first one or more content selection procedures in response to determining that the first subset has a higher priority than the second subset.

[0091] Example 7. The method of Example 6, wherein the first subset of the plurality of content slots includes at least a first content slot and a second content slot, the first one or more content selection procedures include a third content selection procedure for the first content slot and a fourth content selection procedure for the second content slot, and executing the first one or more content selection procedures includes executing the third content selection procedure at least partially in parallel with executing the fourth content selection procedure.

[0092] Example 8. A method according to any one of Examples 1 to 6, wherein one or both of the first subset of the plurality of content slots consists of a single content slot, and the second subset of the plurality of content slots consists of a single content slot.

[0093] Example 9. The method of any one of Examples 1 to 8, wherein the request is a remote procedure call (RPC).

[0094] Example 10. A method for efficiently supplying content to a client device, comprising: receiving, by a server and from the client device, a request for content to be included in an information resource having a plurality of content slots; in response to the request, selecting, by the server, a first set of one or more content items for a first subset of the plurality of content slots by performing a first one or more content selection procedures using selection parameters associated with at least candidate content items, and applying, based on the format or content of the first set of content items, one or more constraints that limit which content items may be candidates in a second one or more content selection procedures for a second subset of the plurality of content slots; and causing the server to cause the client device to fill at least some of the plurality of content slots with the selected plurality of content items.

[0095] Example 11. The method of Example 10, wherein applying the one or more constraints includes excluding content items from the second one or more content selection procedures based on content item category.

[0096] Example 12. The method of Example 11, wherein excluding content items from the second one or more content selection procedures based on content item category includes excluding content items associated with at least one content item category from the first set of content items.

[0097] Example 13. The method of any one of Examples 10 to 12, wherein applying the one or more constraints includes excluding content items from the second one or more content selection procedures based on a cost to the user associated with at least one content item of the first set of content items.

[0098] Example 14. The method of any one of Examples 10 to 13, wherein applying the one or more constraints includes excluding content items that have the same format as at least one content item in the first set of content items.

[0099] Example 15. The method of any one of Examples 10 to 14, wherein applying the one or more constraints includes excluding content items based on a similarity score indicating a level of similarity to at least one content item of the first set of content items.

[0100] Example 16. The method of any one of Examples 10-14, further comprising determining, by the server, the similarity score using a machine learning model.

[0101] Example 17. A method according to any one of Examples 10 to 16, wherein applying the one or more constraints comprises applying one or more parameters indicative of the format or the content of the first set of content items as input to a machine learning model, and deriving at least one of the one or more constraints from the output of the machine learning model.

[0102] Example 18. The method of any one of Examples 10 to 17, wherein the request is a remote procedure call (RPC).

[0103] Example 19. The method of any one of Examples 10 to 18, further comprising, before selecting the plurality of content items, determining by the server that the first subset has a higher priority than the second subset, and performing the first one or more content selection procedures is in response to determining that the first subset has a higher priority than the second subset.

[0104] Example 20. The method of Example 19, wherein the first subset of the plurality of content slots includes at least a first content slot and a second content slot, the first one or more content selection procedures include a third content selection procedure for the first content slot and a fourth content selection procedure for the second content slot, and executing the first one or more content selection procedures includes executing the third content selection procedure at least partially in parallel with executing the fourth content selection procedure.

[0105] Example 21. A method according to any one of Examples 10 to 20, wherein one or both of the first subset of the plurality of content slots consists of a single content slot, and the second subset of the plurality of content slots consists of a single content slot.

[0106] Example 22. A server including one or more processors and one or more memories, wherein the one or more memories store instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of Examples 1 to 21.

[0107] The following additional considerations apply to the foregoing discussion: Throughout this specification, multiple examples may implement a component, operation, or structure that is described as a single example. While individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed simultaneously, and nothing requires that the operations be performed in the order illustrated. Structures and functionality depicted as separate components in an illustrative configuration may be implemented as a combined structure or component. Similarly, structure and functionality depicted as a single component may be implemented as a separate component. These and other variations, modifications, additions, and improvements are within the scope of the subject matter of this disclosure.

[0108] Unless otherwise expressly stated, discussions in this disclosure using words such as "processing," "operating," "calculating," "determining," "representing," "displaying," and the like may refer to machine (e.g., computer) acts or processes that operate on or transform data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0109] As used in this disclosure, any reference to "one implementation" or "an implementation" means that a particular feature, structure, or characteristic described in connection with the implementation is included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same implementation.

[0110] As used in this disclosure, "comprises," "comprising," "includes," "including," "has," "having," or any other variation thereof, is intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent in such process, method, article, or apparatus. Furthermore, unless expressly stated to the contrary, "or" refers to an inclusive or, not an exclusive or. For example, condition A or B can be satisfied by any one of A being true (or present) and B being false (or absent), A being false (absent) and B being true (or present), and both A and B being true (or present).

[0111] After reading this disclosure, those skilled in the art will recognize, through the principles described herein, still additional structural and functional design alternatives for efficiently selecting and delivering content to client devices to fill content slots in information resources. Thus, while particular implementations and applications have been illustrated and described, it should be understood that the disclosed implementations are not limited to the exact structure and components disclosed in this disclosure. Various modifications, changes, and variations apparent to those skilled in the art may be made in the arrangement, operation, and details of the methods and apparatus disclosed in this disclosure without departing from the spirit and scope of the appended claims.

Claims

1. 1. A method for efficiently serving content to a client device, comprising: receiving, by a server and from the client device, a request for content to be included in an information resource having a plurality of content slots; by the server in response to said request: determining a first content selection result for a first subset of the plurality of content slots by performing a first one or more content selection procedures using selection parameters associated with at least the candidate content items; and conditionally performing or not performing a second one or more content selection procedures for a second subset of the plurality of content slots based on the first content selection result; selecting one or more content items including: causing, by the server, the client device to fill one or more content slots of the plurality of content slots with the selected one or more content items; The method comprising:

2. Selecting the one or more content items comprises: not performing the second one or more content selection procedures when selecting a first set of one or more content items to fill the first subset of the plurality of content slots; and performing the second one or more content selection procedures when no content items are selected to fill the first subset of the plurality of content slots; The method of claim 1 , comprising:

3. determining the first content selection results includes selecting a first set of one or more content items to fill the first subset of the plurality of content slots; selecting the one or more content items includes conditionally performing or not performing the second one or more content selection procedures based on one or more characteristics of the first set of content items; The method of claim 1 , comprising:

4. The method of claim 3 , wherein the one or more characteristics of the first set of content items include one or more relevance scores associated with the first set of content items.

5. The method of claim 4 , further comprising determining, by the server, the one or more relevance scores using a machine learning model.

6. determining, by the server, before selecting the one or more content items, that the first subset has a higher priority than the second subset; 6. The method of claim 1, wherein performing the first one or more content selection procedures is in response to determining that the first subset has a higher priority than the second subset.

7. the first subset of the plurality of content slots includes at least a first content slot and a second content slot; the first one or more content selection procedures include a third content selection procedure for the first content slot and a fourth content selection procedure for the second content slot; 7. The method of claim 6, wherein performing the first one or more content selection procedures includes performing the third content selection procedure at least partially in parallel with performing the fourth content selection procedure.

8. the first subset of the plurality of content slots consists of a single content slot; and the second subset of the plurality of content slots consisting of a single content slot; The method of any one of claims 1 to 6, wherein the method is one or both of:

9. The method of any one of claims 1 to 8, wherein the request is a remote procedure call (RPC).

10. 1. A method for efficiently serving content to a client device, comprising: receiving, by a server and from the client device, a request for content to be included in an information resource having a plurality of content slots; by the server in response to said request: selecting a first set of one or more content items for a first subset of the plurality of content slots by performing a first one or more content selection procedures using selection parameters associated with at least the candidate content items; and applying one or more constraints that limit which content items may be candidates in a second one or more content selection procedures for a second subset of the plurality of content slots based on the format or content of the first set of content items; selecting a plurality of content items including: causing the server to cause the client device to fill at least some of the plurality of content slots with the selected plurality of content items; The method comprising:

11. applying the one or more constraints The method of claim 10 , comprising excluding content items from the second one or more content selection procedures based on content item category.

12. Excluding content items from the second one or more content selection procedures based on content item category includes: excluding content items associated with at least one content item category from the first set of content items; The method of claim 11 , comprising:

13. applying the one or more constraints excluding content items from the second one or more content selection procedures based on a cost to the user associated with at least one content item of the first set of content items; The method according to any one of claims 10 to 12, comprising:

14. applying the one or more constraints filtering out content items having the same format as at least one content item in the first set of content items; The method according to any one of claims 10 to 13, comprising:

15. applying the one or more constraints filtering out content items based on a similarity score indicative of a level of similarity to at least one content item of the first set of content items. The method according to any one of claims 10 to 14.

16. The method of any one of claims 10 to 14, further comprising determining, by the server, the similarity score using a machine learning model.

17. applying the one or more constraints applying one or more parameters indicative of the format or the content of the first set of content items as input to a machine learning model; and deriving at least one of the one or more constraints from an output of the machine learning model; The method according to any one of claims 10 to 16, comprising:

18. The method of any one of claims 10 to 17, wherein the request is a remote procedure call (RPC).

19. determining, by the server, before selecting the plurality of content items, that the first subset has a higher priority than the second subset; 19. The method of any one of claims 10 to 18, wherein performing the first one or more content selection procedures is in response to determining that the first subset has a higher priority than the second subset.

20. the first subset of the plurality of content slots includes at least a first content slot and a second content slot; the first one or more content selection procedures include a third content selection procedure for the first content slot and a fourth content selection procedure for the second content slot; 20. The method of claim 19, wherein performing the first one or more content selection procedures includes performing the third content selection procedure at least partially in parallel with performing the fourth content selection procedure.

21. the first subset of the plurality of content slots consists of a single content slot; and the second subset of the plurality of content slots consisting of a single content slot; The method of any one of claims 10 to 20, wherein the method is one or both of:

22. A server comprising one or more processors and one or more memories, the one or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to perform a method according to any one of claims 1 to 21.

Images