Restricting the Provision and Display of Redundant Digital Components on Client Devices

Abstract

A method, system, and apparatus for restricting the provision and display of redundant digital components on a client device, including a computer program encoded on a computer storage medium. The method may include storing, by the client device, a list of digital components specifying digital components available for provision to the client device. For a first digital component received within a first application, the client device detects a set of signals specifying a first user interaction with the first digital component and a second user interaction with content provided in response to the first user interaction. Based on whether a positive user action has been performed, the client device may modify the list of digital components. When a request to access a content page within a second application is received, the client device may receive a second digital component, which may be selected from among the digital components included in the modified list of digital components.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority to RO Application No. A / 10018 / 2020, filed on April 27, 2020, and RO Application No. A / 00282 / 2020, filed on May 22, 2020, the disclosures of which are incorporated herein by reference. Background Art

[0003] This specification relates to restricting the provision and display of such digital components on a client device based at least on previous user interactions and / or user actions with redundant digital components and / or any additional content associated therewith.

[0004] A client device can use an application (e.g., a web browser, a native application) to access a content platform (e.g., a search platform, a social media platform, or another platform hosting content). The content platform can display digital components (discrete units of digital content or digital information, such as, for example, video clips, audio clips, multimedia clips, images, text, or another content unit) that can be provided by one or more content sources / platforms within the application launched on the client device. For example, if a browser application running on the client device is used to perform an internet search for "rental cars", the content source and / or platform can provide a search results page that includes digital components that provide information about rental cars from rental car companies and links to the companies' websites. Summary of the Invention

[0005] Generally speaking, an innovative aspect of the subject matter described in this specification can be embodied in a method including the following operations: storing, by a client device, a list of digital components that specifies a set of digital components available for providing to applications running on the client device; receiving, within a first application running on the client device, a first digital component provided by a first content provider; detecting, by the client device, a set of signals specifying (i) a first user interaction with the first digital component and (ii) a second user interaction with content provided in response to the first user interaction with the first digital component; determining, by the client device and based on the set of signals, that an affirmative user action has been performed by a user of the client device, where the affirmative user action represents the execution of a specified target action by the user after the first user interaction with the first digital component; modifying, by the client device, the list of digital components based on the affirmative user action performed by the user after the first user interaction with the first digital component; receiving a request to access a content page within a second application running on the client device, where the first application is different from the second application; in response to receiving the request to access the content page, transmitting, to a second content provider where the first content provider is different from the second content provider, a content request that includes a portion of the modified list of digital components that prevents the selection of the first digital component in response to the content request; receiving, in response to the content request and within the second application by the client device, a second digital component from the second content provider, where the second digital component is selected from among the digital components included in the modified list of digital components; and providing the second digital component for display on the content page within the second application.

[0006] Other embodiments of this aspect include corresponding methods, apparatuses, and computer programs that are configured to perform the actions of the method encoded on a computer storage device. These and other embodiments may each optionally include one or more of the following features.

[0007] The method may further include inputting the set of signals associated with the first digital component into a machine learning model that predicts whether a user has an affirmative user action with respect to a particular digital component based on a particular set of signals associated with the particular digital component, where the machine learning model is trained using training data of multiple training digital components, and where the training data of each training digital component includes the set of signals associated with the training digital component and a corresponding label indicating whether the user has an affirmative user action with respect to the training digital component; and obtaining, from the machine learning model and in response to the set of signals associated with the first digital component input into the machine learning model, an indication specifying whether the user has an affirmative user action with respect to the first digital component.

[0008] The method may include storing a ranked list of digital components and modifying the list of digital components by decreasing the rank corresponding to the first digital component in the ranked list or removing the first digital component from the list of digital components.

[0009] The method may include transmitting a content request including a portion of the modified list to a second content provider, where the portion of the modified list of digital components includes the top N ranked digital components in the ranked list.

[0010] The method may further include: in response to the content request, receiving, by a client device and within a second application, a third digital component from the second content provider, where the third digital component is not among the digital components included in the modified list of digital components; suppressing the display of the third digital component on a content page; and in response to the suppression, modifying the content layout of the content page and providing a message indicating that the third digital component has been suppressed to the second content provider.

[0011] Particular embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. For example, the techniques described in this specification are capable of filtering out (e.g., preventing retrieval and / or display) certain digital components (or certain types of digital components) for which the user of the client has had a positive user action (as further described in this specification), thereby saving significant computational resources required in providing and rendering these digital components. This in turn can also promote an improved user experience and user engagement across multiple content platforms by avoiding the repeated presentation of the same / similar digital components. Traditional systems do not include the ability to prevent the presentation of the same or similar digital components provided by multiple different content sources / platforms. In contrast, the techniques described in this specification can prevent the provision and / or rendering of (multiple) digital components for which the user has had a positive user action, and can thus provide and display digital components for which the user has not had a positive user action, rather than presenting content for which the user has already experienced and taken action. By preventing the presentation of content for which the user has already experienced and taken action, the system reduces the amount of wasted computational resources (e.g., processing resources, network bandwidth, limited display space, etc.) due to providing redundant content to the user. Additionally, the limited display space of the client device is used more effectively because the space occupied by the content that has been prevented can instead be beneficially used for other content or other purposes.

[0012] The techniques described in this document also facilitate an improvement in the security and privacy of data associated with handling, analyzing, and / or maintaining interactions with digital components and / or any associated device actions on a client device. In some traditional implementations, when data is to be shared with a third-party system and is thus exposed to the third-party system, the security and / or privacy of such data may not be maintained or may even be infeasible. In contrast, the techniques described in this specification can be implemented on a client device such that the detection, processing, and storage of a set of signals for inferring affirmative user actions and the determination of affirmative user actions can be performed and stored entirely on the client device. Additionally, the techniques described in this specification do not require sharing such data with a third-party system.

[0013] Furthermore, the techniques described in this specification are capable of dynamically modifying the interface displayed on a client device. The techniques described in this specification can enable a client device to suppress, remove, and / or prevent the display of certain digital components for which an affirmative user action has occurred. In such cases, the techniques described herein can dynamically modify the interface such that the location where the suppressed / removed digital component was to be displayed is replaced by other content (e.g., content already included on the page that can be adjusted (e.g., reorganized / moved / resized)) or other content available from a content source / platform.

[0014] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of an example environment in which digital content is distributed and presented to a client device.

[0016] Figure 2 is a table showing an example list of digital components stored on a client device.

[0017] Figure 3 is a flowchart of an example process for providing one or more digital components to a client device based on user actions on the client device.

[0018] Figure 4 is a block diagram of an example computer system that can be used to perform the described operations. DETAILED DESCRIPTION

[0019] This specification generally relates to restricting the provision and display of such digital components on a client device based at least on previous user interactions and / or user actions with redundant digital components and / or any additional content associated therewith.

[0020] Client devices connected to the Internet (e.g., smart phones, tablets, and personal computers) can be provided with digital content that includes various digital components. As used throughout this document, the phrase "digital component" refers to discrete units of digital content or digital information (e.g., video clips, audio clips, multimedia clips, images, text, or another content unit).

[0021] User interaction and / or user actions with digital components can ultimately result in an affirmative user action, which represents the execution of a specific target event / action after the presentation of and / or interaction with the digital component. For example, a user who encounters a digital component about an endangered species can interact with the component (e.g., by selecting or clicking on the component) and be directed to a web page about a specific endangered species, where the user can sign up for a newsletter aimed at helping to save that endangered species. In this example, signing up for the newsletter can be considered a specific target action. Examples of target actions can also include, but are not limited to, registering with a website / service, adding an item to an online shopping cart, downloading a white paper, obtaining a product, navigating to at least a given depth of a website, viewing at least a certain number of web pages, spending at least a predetermined amount of time on a website or web page, completing a website registration process, and subscribing to a digital service. In other words, a specific target action can be an interaction by the user with the content provided to the user after the user's interaction with the digital component. The user can perform a first interaction with a first piece of content of the digital component and then, after that first interaction, perform a second interaction with a second piece of content provided to the user as a result of the first interaction. The first and second interactions can be the generation of signals caused by one or more inputs from the user. Inputs from the user can be via a touch screen, keyboard, microphone, video, or any other means of interacting with the client device.

[0022] When a client device re-accesses the same or a different content platform within an application running on the client device, digital components that are the same as or similar to the digital components previously provided to the client device can be provided to the client device again. Such redundant provision of such digital components can occur even if the client device had a positive user action with respect to the same / similar digital components previously. A positive user action is indicated by the execution of a specific target action after a user interaction with the digital component. Additionally, in some systems, even if one content source / platform determines that a user of the client device has had a positive user action with respect to a specific digital component, that determination of the positive user action may be unknown to another content source / platform that the client device may subsequently access. As a result, the other content source / platform can provide the same or a similar type of digital component that has encountered a positive user action. For example, assume that a specific digital component has been submitted to two different content distribution systems that do not share data. In such a scenario, even if one content distribution system is aware that a specific user has completed a target action after interacting with the specific digital component, another different content distribution system will be completely unaware of the execution of the target action, such that the specific digital component can continue to be presented to the specific user.

[0023] In contrast, the techniques described in this document can be implemented entirely (or mostly) on the client device (i.e., independent of a specific content source / platform). As further described in this specification, the client device can use multiple signals to determine a positive user action with respect to a digital component based on past user activity on the client device, user interactions with the digital component on the client device regardless of the content source / platform of the digital component, the content platform on which the (multiple) digital components are provided, or the (multiple) applications within which the digital components are obtained. In some embodiments, a machine learning model (or a heuristic-based approach, or another suitable model-based or rule-based technique) is used to analyze / process the multiple signals, and the machine learning model determines whether the client device has had a positive user action with respect to the digital component.

[0024] Based on whether the client device has a positive user action on a specific digital component, the techniques described herein can modify a stored (i.e., stored on the client device) list of digital components (e.g., a cookie list) that specifies a set of digital components available for providing to an application running on the client device. For example, in response to a user's interaction (e.g., selecting, clicking, viewing) with a digital component of a rental car company, the list of digital components can be updated to include a specific entry (e.g., the name of the digital component, the category / type of the digital component, the activity the user was performing while interacting with the digital component) that indicates the provided digital component. However, if a machine learning model (or another suitable means / model) determines that the client device has a positive user action on a digital component, the machine learning model (or other suitable means / model) can update the list of digital components, e.g., by removing a specific digital component from the list or decreasing its ranking in the list (if it is already in the list) or by not adding the specific digital component to the list (if it is not already in the list).

[0025] Subsequently, when the client device uses the application to access the same or a different content platform, the techniques described herein can send a portion of the modified list of digital components (e.g., the entire list or a subset of the list, such as the top N components included on the list) to the content platform (and / or the (one or more) content sources providing content for the content platform). The content platform and / or the (one or more) content sources can use the modified list of digital components to provide content for display within the application running on the client device. For example, the content source and / or the content platform can only provide one or more digital components that are included on the received portion of the list of digital components. In this case, the application running on the client device renders the digital components provided by the content source / platform. As another example, the (one or more) content sources and / or the (one or more) content platforms can provide any one or more digital components, regardless of whether they are on the list. In this case, the client device can determine whether the provided digital component is on the modified list of digital components (e.g., whether the provided digital component is one of the top N digital components on the list of digital components). If so, the application running on the client device can render / display the provided digital component. Otherwise, the application can suppress such content and instead modify the interface such that the location where the suppressed / removed digital component was to be displayed is replaced by other content (e.g., content that is already included on the page and can be adjusted (e.g., rearranged / moved / resized)) or other content available from the content source / platform).

[0026] The following references Figures 1 - 4 further describe these features and additional features.

[0027] In addition to the description throughout this document, controls can be provided to users to allow them to select whether and when the systems, programs, or features described herein can collect user information (e.g., information about a user's social network, social actions or activities, occupation, user preferences, or user's current location) and whether to transmit content or communications to the user from a server. Additionally, certain data may be processed in one or more ways before being stored or used so that personally identifiable information is removed. For example, a user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographical location may be generalized to the place where the location information was obtained (such as city, ZIP code, or state level) so that the user's specific location cannot be determined. Thus, the user can control what information about the user is collected, how that information is used, and what information is provided to the user.

[0028] Figure 1 FIG. 4 is a block diagram of an example environment 100 in which content is distributed and presented to client devices. Example environment 100 includes a network 104, such as a local area network (LAN), wide area network (WAN), the Internet, or a combination thereof. Network 104 connects client devices 102, content platform 106, and content sources 110. Example environment 100 may include many different content sources 110, content platforms 106, and client devices 102.

[0029] Content platform 106 is a computing platform capable of distributing content. Example content platforms 106 include search engines, social media platforms, news platforms, data aggregator platforms, or other content sharing platforms. Each content platform 106 may be operated by a content platform service provider.

[0030] Content platform 106 may publish and provide its own content on the platform. For example, content platform 106 may be a news platform that publishes its own news articles. Content platform 106 may also present content provided by one or more content sources 110. In the above example, the news platform may also present content created by different authors and provided by one or more content sources 110. As another example, content platform 106 may be a data aggregator platform that does not publish any of its own content but aggregates and presents news articles provided by different news websites (i.e., content sources 110).

[0031] Client device 102 is an electronic device capable of requesting and receiving content via network 104. Example client devices 102 include personal computers, mobile communication devices, digital assistant devices, and other devices capable of transmitting and receiving data via network 104.

[0032] Client device 102 typically includes an operating system 112 that is primarily responsible for managing the device's hardware resources and software resources, such as applications. Client device 102 also includes device storage 120 for storing data temporarily or permanently based on specific embodiments, applications, and use cases. Client device 102 typically includes user applications 116 and 117 (such as a web browser) to facilitate the transmission and reception of data over network 104, but local applications run by client device 102 can also facilitate the transmission and reception of content over network 104. Examples of content presented at client device 102 include web pages, word processing documents, Portable Document Format (PDF) documents, images, videos, and search result pages and digital advertisements.

[0033] Generally, client device 102 interacts with applications running on client device 102 to access digital content, such as search results, web pages, news articles, and social media posts. When accessing digital content, the client device can also receive digital components from one or more content providers.

[0034] For example, assume that a user uses application A 116 (which is a web browser) to perform an Internet search for "rental cars" and view the search results returned in response to submitting the search query "rental cars". In this case, a third party (i.e., an entity other than the client device, such as a content platform or content source) can provide (e.g., by selecting or clicking on a digital component, or viewing the content presented in the digital component for a specific period of time) digital components (e.g., videos, text) that may be relevant to the search query and with which the user can interact to the application.

[0035] This interaction with these digital components can cause the application to open (or redirect to) another content page with additional content / digital components that may be related to the theme of the selected digital component. For example, assume that while viewing the search results returned in response to submitting the search query "rental cars", the user interacts with a digital component by selecting or clicking on a link. In this example, client device 102 can open application B 117, which is an application (e.g., a local application) installed on client device 102 and provided by a specific car rental company. In another example, selecting or clicking on a link can redirect the user within application A 116 to another website that includes digital content related to the search query "rental cars", such as reviews of different car rental services or web pages of car rental services.

[0036] The client device 102 also includes a content evaluation device 114 (which may be a data processing device, as described in this specification). In some embodiments, the content evaluation device 114 is implemented as a machine learning model that includes a plurality of trainable parameters and is trained to determine a positive user action of a user with respect to digital components presented on the client device 102 by analyzing a set of signals (as further described below). The machine learning model can be any model considered suitable for a particular embodiment, such as a decision tree, artificial neural network, genetic programming, logic programming, support vector machine, clustering, reinforcement learning, Bayesian inference, and the like. The machine learning model can also include methods, algorithms, and techniques for natural language processing for analyzing signals including text data.

[0037] In some embodiments, a machine learning model implemented within the content evaluation device 114 is trained using training data of a plurality of training digital components. Each training digital component in the training data is associated with a corresponding set of signals of the training digital component and a label indicating whether the user has a positive user action with respect to the training digital component. For example, the training digital components include a set of signals generated by user interactions with a particular digital component (and / or actions / interactions with other content / applications / digital components on the client device 102), and a label stating whether the user action with respect to the particular digital component is positive.

[0038] In some embodiments, training the machine learning model involves adjusting the trainable parameters of the machine learning model such that the machine learning model can analyze a plurality of signals generated by user activities and user interactions with digital components to predict whether the user action with respect to the digital components provided by the content provider is positive. Depending on the specific embodiment, the training process of the machine learning model can be supervised, unsupervised, or semi-supervised.

[0039] In some embodiments, in addition to user interactions and / or user actions with specific digital components, the machine learning model can also process multiple additional signals generated from user activities on the client device. These signals include, but are not limited to, clicks on different links provided in the digital components, Internet search keywords, and the time spent viewing different digital components. For example, if a digital component with a link to a website belonging to a specific product (or service or other content) is provided to the client device during the operation of an application, the user can click on the link and access the linked website. In this case, the OS detects the user interaction (click or selection) and generates a signal that is provided to the content evaluation device. In another example, when a digital component including a video is provided to the client device, the OS can detect that the user has skipped the video (e.g., by scrolling past the digital component) or may have spent some time viewing the video (e.g., by not scrolling the page for a certain threshold amount of time). Thus, the OS generates a signal representing the amount of time spent on the video digital component and provides this signal for further processing by the machine learning model.

[0040] In some embodiments, in addition to the above user interaction signals, the machine learning model implemented within the content evaluation device 114 can also predict a positive user action of the user on a digital component based on signals generated by the user's contact with the digital components presented on the client device 102. Such embodiments can use techniques well known in the fields of digital image processing and natural language processing, such as optical character recognition (OCR). For example, assume that the client device is interacting with a third-party application (such as an application for booking a rental car provided by a car rental company), and an image of the rendered user interface can be used to identify the user's positive user action on the digital component. In another example, assume that the user receives a web push notification about the booking confirmation of a rental car in a browser application running on the client device 102. Depending on the type of embodiment, the OS can generate multiple signals from the web push notification, such as an image of the user interface presenting the web push notification on the client device 102 or the text of the web push notification. These signals can be analyzed by the machine learning model to determine the user's positive user action on the digital component.

[0041] In some embodiments, as described above, the machine learning model implemented within the content assessment device 114 processes a set of signals generated through user interactions with digital components and other actions on the client device (e.g., a first user interaction with a digital component and one or more additional user interactions and / or user actions with the content provided in response to the first user interaction) to determine whether a positive user action has been performed. In some embodiments, the machine learning model can generate a score or likelihood of a positive user action based on the set of input signals. For example, the likelihood or score of a positive user action can be a number in the range from 0 to 10, where a number closer to 0 indicates a lower likelihood or score of a positive user action, and a number closer to 10 indicates a higher likelihood / score of a positive user action. In such an embodiment, there may be a preset threshold that may imply a positive user action when the likelihood of the positive user action is greater than the preset threshold. For example, assume that the preset threshold for likelihood is set to 5. If the likelihood of the user action on the digital component generated by the machine learning model is a value greater than 5, then the action is determined to be positive. Otherwise, the interaction is determined not to be positive.

[0042] In some embodiments, the content assessment device 114 implements a heuristic-based approach (instead of a machine learning model) for determining positive user actions on digital components. In such an embodiment, the content assessment device 114 analyzes the same set of signals (as described above) generated through user interactions with the digital components provided to the client device 102 by the content source / platform.

[0043] In some embodiments, the content assessment device 114 operates at the operating system (OS) level on the client device 102, rather than at the application level of a specific application. As used in this specification, an OS-level operation is either an operation that accesses higher permissions than an application-level operation, and / or an operation performed by the operating system.

[0044] As another example, an OS-level operation can access data processed by multiple different applications and provide that data to the machine learning model (or another suitable model or heuristic-based method) such that it can determine whether a positive user action has been performed. Thus, by operating at the OS level, the machine learning model is agnostic to the application and can determine positive user actions by analyzing the data accessed by the client device 102.

[0045] Although Figure 1Although not shown in the figure, in some embodiments, the content evaluation device 114 may be implemented at the application level (instead of at the OS level). In such an embodiment, the application implementing the content evaluation device 114 may be provided with higher permissions than other applications running at the application level of the operating system 112. In such an embodiment, the application implementing the content evaluation device 114 may have the permission to access the content of another application. When the application implementing the content evaluation device 114 accesses the content, a machine learning model (or another suitable model or heuristic-based approach) determines whether a positive user action has occurred.

[0046] In some embodiments, the client device 102 stores a digital component list 130 in the device storage 120. The digital component list 130 (e.g., a cookie list) includes a list of digital components that can be provided to applications running on the client device 102. The digital component list 130 will be further explained with reference to Figure 2 below.

[0047] As further described with reference to Figure 3 below, the digital component list is modified based on a determination as to whether the client device has a positive user action with respect to a particular digital component. As further described with reference to Figure 3 below, the modified digital component list is then used by the content platform and / or the content source(s) to provide digital components for display within the application such that the provided digital components are different from the digital component(s) (or type of digital component(s)) for which the client device has had a positive user action.

[0048] Figure 2 is a table showing an example digital component list 130 stored on the client device 102.

[0049] In some embodiments, each entry in the digital component list 130 includes a particular digital component that can be presented to the client device. Such digital components are identified on the list based on previous user interactions and / or user actions with the digital components. Based on the type of digital component and user activity, each entry in the digital component list 130 may also include multiple features. For example, columns Product 202, Category 204, User Action 206, and Ranking 208 represent different features or characteristics of user activity or user interaction with the digital component.

[0050] In some embodiments, a feature or characteristic is directly associated with, or inferred from, user activity or user interaction with a digital component. For example, assume that a user searches for "rental cars" and obtains a content page that includes search results returned in response to submitting the search query "rental cars" and digital components depicting specific rental car services. In this example, an entry representing the topic of the specific digital component with which the user initially interacted is created in the digital component list 130 on the client device. For example, entry 5 in the table indicates the user action of performing an Internet search for rental cars. Additionally, in this example, the user may interact with one of these digital components (e.g., by selecting or clicking), which can redirect the user to a specific third-party rental car website. In this example, an entry representing the user's interaction with the digital component (web page) of a specific car rental service is created in the digital component list 130 on the client device 102. For example, entry 6 in the table indicates the interaction where the user accesses the web page of rental car company Q, and entry 7 in the table indicates the interaction where the user accesses the web page of rental car company R.

[0051] In some embodiments, a portion of the digital component list 130 (e.g., in response to being provided the list when a blind pixel on the client device is triggered, or in response to a request for the list) is accessed by a third party such as a content platform or content source. Based on a specific entry in the digital component list, the third party provides digital components to the user. For example, assume that a user is accessing digital content provided by a content provider using the browser application 117 installed on the client device 102. The content source / platform may transmit a request for the digital component list to the client device 102. In response to the request for the digital component list, the client device 102 may send a portion of the digital component list to the content source. Upon receiving the portion of the digital component list, the content source may provide digital components to the client device 102 for presentation to the user based on the entries listed in the portion of the digital component list received by the content source / platform.

[0052] In some embodiments, each entry in the digital component list 130 is ranked based on the probability that a user may interact with a specific digital component. For example, column 208 includes the ranking of each entry representing the digital component list in table 130. In some embodiments, the ranking of each entry in the digital component can be as described in the following example and with reference to Figure 1 and Figure 3Modify as further described. For example, if a machine learning model implemented within the content assessment device 114 determines that the user has reserved a rental car by analyzing other digital content accessed by a browser application, the ranking of a specific entry in the digital component list 130 can be decreased. When a third party such as an advertiser or publisher receives the modified digital component list 130 from the client device 102, digital components are provided based on the updated rankings of the digital components, thereby providing the user with digital components having a higher ranking (or among the top N ranked digital components).

[0053] In some embodiments, more than one content source / platform can provide digital content to the user on the client device 102. In such an embodiment, the client device 102 can store a digital component list 130, which can be modified based on affirmative user actions identified using signals generated by user interactions with digital components supplied by a first content source / platform. Subsequently, a second content source / platform can access a portion of the digital component list 130 stored on the client device 102 and supply digital components based on the modified digital component list 130.

[0054] For example, assume that an Internet search for "camera" is performed using browser application A 116 running on the client device 102, which results in a search result page being provided within application A 116 and digital components being provided by a first content source. If the client device 102 detects an interaction with the digital component, the client device 102 adds an entry for the digital component to the digital component list 130. Later, during another online session, a second content source / platform can access a portion of the digital component list 130 from the client device 102 and provide digital components related to "camera" based on that entry. If the user performs a target action (e.g., reads a review of a specific camera on a website linked to by the digital component) in response to an interaction with the digital component (as well as additional content provided in response to that interaction and additional content for which the client device detects additional interactions / actions), the content assessment device 114 implemented on the client device 102 can analyze the corresponding signals detected by the OS based on such interaction / action and modify the digital component list 130 on the client device, for example, by removing entry 212 or decreasing the ranking of a specific entry 212. In this scenario, the first content source / platform can access the digital content list and not provide an advertisement related to "camera".

[0055] Figure 3 is a flowchart of an example process 300 for providing one or more digital components to a client device based on user actions and / or operations of the client device. The operations of process 300 are described below as being performed by Figure 1 andFigure 2 performed by the components of the system described and depicted herein. The operations of process 300 are described below for illustrative purposes only. The operations of process 300 can be performed by any suitable device or system (e.g., any suitable data processing device). The operations of process 300 can also be implemented as instructions stored on a non-transitory computer-readable medium. The execution of the instructions causes one or more data processing devices to perform the operations of process 300.

[0056] The digital component list 130 is stored on the client device and specifies a set of digital components (310) available for providing to an application running on the client device. In some embodiments, and as described above with reference to Figure 2 each entry in the digital component list 130 describes / represents a digital component available for providing to an application running on the client device. The entries in the digital component list 130 are created based at least on user interactions and / or user actions with the digital components provided to the client device 102 by the content source / platform. For example, using Figure 2 the digital component list 130 represented by the table in is stored in the device storage 120 of the client device 102. In some embodiments, the entries in the digital component list 130 are ranked according to their relative importance to the user. For example, column 208 shows the ranking of each entry in the list. In some embodiments, the entries in the digital component list can be ranked relative to type / category.

[0057] A first digital component is provided by a first content source to an application running on the client device (320). In some embodiments, the first content source can provide the first digital component for display within an application running on the client device 102. For example, a browser-based application A 116 running on the client device 102 can be used to perform an Internet search for "rental cars". In response to the search query, the content source / platform provides search results related to rental cars to the client device 102.

[0058] The client device detects a set of signals, where the set of signals specifies a first user interaction with a first digital component and a second user interaction with content provided in response to the first user interaction (330). In some embodiments, the OS of client device 102 detects a first user interaction with the first digital component. Examples of such first user interactions and / or user actions include a user selecting the first digital component, clicking on the first digital component, and viewing the first digital component (e.g., for a specific period of time). The OS can detect such user interactions and / or user actions by using a set of rules that capture different device events (e.g., time elapsed in a specific application, information about scrolling content on the client device, selection / click of a digital component, opening of presented content / app or page in response to selection or click of a digital component), and combining one or more of these rules to determine if the first interaction has been performed. Alternatively, the captured device events can be fed into a model (e.g., a machine learning model or another suitable statistical model) trained to predict whether the first interaction has been performed based on a set of device actions rather than a set of rules. For example, such a model can be trained using the actually detected first interactions and their respective sets of device events.

[0059] In response to the first user interaction, additional content can be provided within an application running on client device 102 by the first content source (or by another content source). Examples of such additional content include content pages to which a user is redirected after selecting a link in search results provided by a content provider in response to a submitted search query. The OS can use signals such as click counts or time spent on a web page to detect a second user interaction with the additional content (which can include, for example, purchasing a product / service, leaving a review, or signing up for a content delivery service). Similarly, the OS can detect such user interactions and / or user actions by using a set of rules that capture different device events (obtained after the first user interaction is performed) of the displayed content and / or parsing / analyzing (e.g., using OCR, image processing, etc.), and combining one or more of these rules to determine if the second interaction has been performed. Alternatively, the captured device events and / or the parsed / analyzed image content can be fed into a model (e.g., a machine learning model or another suitable statistical model) trained to predict whether the second interaction has been performed based on a set of device actions rather than a set of rules. For example, such a model can be trained using the actually detected second user interactions and / or user actions and their respective corresponding sets of device events.

[0060] In addition to the above interactions, the OS can also detect additional signals regarding other device actions and / or data from other applications / services on the device (as referenced above Figure 1(as described). For example, if a user receives a clickable URL in an email or SMS text after opening a web page that displays a confirmation of a rental car reservation on a browser application running on the client device 102, the OS will detect signals such as text from the web page and provide the set of signals to the content evaluation device 114.

[0061] The set of signals is used to determine whether the user has performed an affirmative user action (340) with respect to the first digital component. In some embodiments, the content evaluation device 114 implements a machine learning model or a heuristic-based approach on the client device 102 that analyzes the set of signals (detected as described above in operation 330) to determine whether an affirmative user action has occurred (as referenced Figure 1 and further described). For example, if a digital component is provided for display and includes a link to a website regarding a particular product, the user may click on the link and access the linked website. In this case, the user activity generates multiple signals such as the number of clicks and the context of the content presented in the web page. The content evaluation device 114 analyzes the signals to determine the user's affirmative user action with respect to the digital component provided to the client device 102 by the content source / platform.

[0062] In some embodiments, and as referenced Figure 1 and described, the machine learning model implemented within the content evaluation device 114 processes the set of signals generated by user interactions with the digital component (and other actions on the client device) and performs a classification of whether an affirmative user action has resulted based on the set of signals associated with the user interaction with the digital component (e.g., the first user interaction with the digital component and one or more additional user interactions and / or user actions with respect to the content provided in response to the first user interaction). In some embodiments, the machine learning model may generate a score or likelihood of an affirmative user action based on the input set of signals (as referenced Figure 1 and described).

[0063] The digital component list 130 is modified, and the modified list is stored on the client device 102 (350). If the content evaluation device 114 implemented within the client device 102 identifies a positive user action for a particular digital component, it modifies the digital component list 130 stored on the client device 102. In some embodiments, if the content evaluation device determines that a first digital component has encountered a positive user action, the content evaluation device 114 reduces the ranking of a particular entry corresponding to the first digital component in the digital component list, e.g., such that the particular entry is no longer among the top N ranked components. For example, if a machine learning model implemented within the content evaluation device 114 determines that the user has booked a rental car by analyzing other digital content accessed by the same browser application or by other applications, the ranking of a particular entry in the digital component list can be reduced. Alternatively, the content evaluation device 114 can remove the reference to the digital component from the digital component list 130 instead of reducing the ranking.

[0064] A request to access a content page is received from an application running on the client device 102 (360). To enable an application running on the client device 102 to access a content page provided by a content source / platform, the application generates a request to access the content page sent to the content source / platform over the network 104. The content source, upon receiving the request to access the content page, sends the corresponding content page to the client device 102. For example, a user can use the browser application A 116 running on the client device 102 to perform an Internet search, view the search results provided in response to the Internet search, and access multiple digital components by clicking on a link provided as a search result. In such a scenario, the client device 102 generates a request to access the content page including the search results. Additionally, when the user clicks on a link provided as a search result, for example, the client device 102 generates a corresponding request to access the digital content sent to the content source over the network 104.

[0065] A content request including a portion of the modified digital component list is transmitted to a second content source (370). In some embodiments, in response to a request by an application running on the client device 102 to access a content page, a script on the content page runs and causes the application to provide a content request to the second content source. In some embodiments, the content request includes a portion of the modified digital component list (e.g., the entire list or a subset of the list, such as the top N ranked digital components). The content request can also include event data specifying content characteristics, such as the requested electronic document, the name or network location of the server from which the digital components are requested, the name or network location of the requesting device (e.g., the client device 102).

[0066] In some embodiments, a portion of the digital component list 130 provided to the content source / platform includes the top N entries in the digital component list 130, where the top N entries are selected based on the ranking of each individual digital component based on the relative interests of the user. For example, in Figure 2 entries 1-7 are ranked relative to the type / category of the digital component. In this case, the client device 102 may provide to the content source / platform a portion of the digital component list 130 that includes the entry ranked 1.

[0067] In some embodiments, a portion of the digital component list 130 provided to the content source / platform includes a combination of one or more features of the digital components listed in the top N entries of the digital component list 130, where the top N entries are selected based on the ranking of each individual digital component based on the relative interests of the user. For example, in Figure 2 the client device 102 may provide to the content source / platform a portion of the digital component list 130 that includes one or more features (such as product 202 or category 204 of the entry ranked 1).

[0068] Receive a second digital component (380) from the content source / platform based on a portion of the modified digital component list 130. In some embodiments, when receiving a portion of the digital component list 130, the content source / platform provides to the client device 102 those digital components among the digital components listed on the received list 130 (e.g., provides digital components from the top N identified digital components on the list 130). In certain scenarios, the digital component list may be a modified digital component list, where the digital component list is modified based on positive user actions of the user with respect to other digital components.

[0069] For example, assume that a portion of the digital component list 130 provided to the content source / platform includes the top N entries in the digital component list 130, where the top N entries are selected based on the ranking of each individual digital component. In this case, the portion of the digital component list 130 accessed by the content source / platform will include entries 1, 2, and 5, which indicate the user's interest in cameras, hotels, and rental cars.

[0070] In some embodiments, the (multiple) content sources and / or content platforms may provide any digital components, regardless of whether they are on a list. In such a case, the client device may determine whether the provided digital component is on the digital component list (e.g., whether the provided digital component is one of the top N digital components on the digital component list). If so, the application running on the client device may render / display the provided digital component. Otherwise, the application may suppress it, e.g., by not displaying such content, and optionally, the application may instead modify the interface such that the location where the suppressed / removed digital component was to be displayed is replaced by other content (e.g., content already included on the page that can be adjusted (e.g., reorganized / moved / resized), or other content), which may be obtained from the content source or content platform.

[0071] In some embodiments, the client device 102 may notify the content platform or content source about a particular digital component that was suppressed and not presented / displayed on the client device 102. The content platform or content source may use such information to avoid subsequently providing such digital components on the client device 102. The content platform and / or content source may maintain / store information about the suppressed / removed content, which may be used to perform an analysis that may inform the types of digital components to be provided to one or more client devices.

[0072] Digital components received from the content source / platform are provided for display (390) within an application running on the client device 102. In some embodiments, the client device 102 provides the digital components for display within the application when it receives the digital components from a second content source. For example, assume that the content source / platform provides digital components, such as content related to cameras, hotels, and rental cars, to the client device 102 when it receives a portion of the digital component list.

[0073] In summary, the above operations limit the provision and display on the client device of redundant digital components for which the client device has had a positive user action. Relatedly, when there has been no positive user action associated with these digital components previously, the above operations provide and display the digital components (regardless of whether they were previously provided) without any restrictions.

[0074] Figure 4FIG. 0 is a block diagram of an example computer system 400 that can be used to perform the operations described above. System 400 includes a processor 410, a memory 420, a storage device 430, and an input / output device 440. Each of the components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. The processor 410 is capable of processing instructions for execution within the system 400. In some embodiments, the processor 410 is a single-threaded processor. In another embodiment, the processor 410 is a multi-threaded processor. The processor 410 is capable of processing instructions stored in the memory 420 or on the storage device 430.

[0075] The memory 420 stores information within the system 400. In one embodiment, the memory 420 is a computer-readable medium. In some embodiments, the memory 420 is a volatile memory unit. In another embodiment, the memory 420 is a non-volatile memory unit.

[0076] The storage device 430 is capable of providing mass storage for the system 400. In some embodiments, the storage device 430 is a computer-readable medium. In various different embodiments, the storage device 430 can include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices (e.g., a cloud storage device), or some other large-capacity storage device.

[0077] The input / output device 440 provides input / output operations for the system 400. In some embodiments, the input / output device 440 can include one or more network interface devices, such as an Ethernet card, a serial communication device (e.g., an RS-232 port), and / or a wireless interface device (e.g., an 802.11 card). In another embodiment, the input / output device can include a drive device configured to receive input data and transfer output data to a peripheral device 460 (e.g., a keyboard, a printer, and a display device). However, other embodiments can also be used, such as mobile computing devices, mobile communication devices, set-top box TV client devices, etc.

[0078] Although an example processing system has been described in Figure 4 , embodiments of the subject matter and functional operations described in this specification can be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them.

[0079] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs encoded on a computer storage medium for running by, or to control the operation of, a data processing apparatus, i.e., one or more modules of computer program instructions. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver apparatus for running by the data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, although a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0080] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0081] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or multiple or combinations of the foregoing. The apparatus can include dedicated logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus can also include code that creates an execution environment for the computer programs being discussed, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and the execution environment can implement various different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0082] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can, but need not, correspond to a file in a file system. The program can be stored in a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program being discussed, or in multiple cooperating files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0083] The processes and logical flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. These processes and logical flows can also be performed by dedicated logic circuitry, and the apparatus can also be implemented as dedicated logic circuitry, e.g., an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0084] By way of example, processors suitable for running a computer program include both general and special purpose microprocessors. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for performing actions in accordance with the instructions and one or more memory devices for storing the instructions and data. Generally, a computer will also include or be operatively coupled to one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, to receive data therefrom, or to transfer data thereto, or both. However, a computer need not have such devices. In addition, a computer can be embedded in another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus (USB) flash drive), etc. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example: semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, dedicated logic circuitry.

[0085] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user may provide input to the computer. Other kinds of devices may also be used to provide for interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, the computer may interact with the user by transmitting documents to and receiving documents from the devices used by the user; for example, by sending a web page to a web browser on a user client device in response to a request received from the web browser.

[0086] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer having a graphical user interface or a web browser through which the user may interact with the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an intranet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network).

[0087] The computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on respective computers and having a client-server relationship to each other. In some embodiments, the server transmits data (e.g., an HTML page) to the client device (e.g., for displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., the results of user interaction) may be received at the server from the client device.

[0088] Although this specification contains many specific implementation details, these should not be construed as limitations on any invention or the scope of the claims, but rather as descriptions of specific features of particular embodiments of a particular invention. Certain features described in the context of separate embodiments in this specification can also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. Additionally, although a feature may be described above as acting in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination can be deleted from the combination, and the claimed combination can be directed to a sub-combination or a variant of the sub-combination.

[0089] Similarly, although operations are described in a particular order in the figures, this should not be understood as requiring that the operations be performed in the particular order shown or in sequential order, or that all of the operations shown be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the above embodiments should not be understood as required in all embodiments, and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged into multiple software products.

[0090] Accordingly, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the acts recited in the claims can be performed in a different order and still obtain the desired result. Additionally, the processes described in the figures do not necessarily require the particular order or sequential order shown to obtain the desired result. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method, comprising: A client device stores a list of digital components that specifies a set of digital components available for providing to an application running on the client device; Receive a first digital component provided by a first content provider within a first application running on the client device; The client device detects a set of signals that specify (i) a first user interaction with the first digital component and (ii) a second user interaction with content provided in response to the first user interaction with the first digital component; The client device determines, based on the set of signals, that a positive user action has been performed by a user of the client device, where the positive user action represents the execution of a specified target action by the user after the first user interaction with the first digital component; The client device modifies the list of digital components based on the positive user action performed by the user after the first user interaction with the first digital component; Receive a request to access a content page within a second application running on the client device; In response to receiving the request to access the content page, transmit a content request to a second content provider that includes a portion of the modified list of digital components that prevents the first digital component from being selected in response to the content request; In response to the content request, the client device receives a second digital component from the second content provider within the second application, where the second digital component is selected from among the digital components included in the modified list of digital components; and Provide the second digital component for display on the content page within the second application.

2. The computer-implemented method according to claim 1, wherein, Determining that a positive user action has been performed by a user of the client device includes: Inputting the set of signals associated with the first digital component into a machine learning model that predicts whether the user has a positive user action with respect to the first digital component based on the set of signals associated with the first digital component, where: The machine learning model is trained using training data of multiple training digital components, where the training data for each training digital component includes a set of signals associated with the training digital component and a corresponding label indicating whether the user has a positive user action with respect to the training digital component; and Obtain an indication specifying whether the user has a positive user action with respect to the first digital component from the machine learning model and in response to the set of signals associated with the first digital component input into the machine learning model.

3. The computer-implemented method according to claim 1 or 2, wherein, Storing the list of digital components includes storing a ranked list of digital components.

4. The computer-implemented method according to claim 3, wherein, Modifying the list of digital components includes decreasing the rank corresponding to the first digital component in the ranked list.

5. The computer-implemented method according to claim 4, wherein, A portion of the modified list of digital components includes the top N ranked digital components in the ranked list.

6. The computer-implemented method according to claim 1 or 2, wherein, Modifying the list of digital components includes removing the first digital component from the list of digital components.

7. The computer-implemented method according to claim 1 or 2, wherein: In response to the content request, a third digital component is received by the client device and within the second application from the second content provider, where the third digital component is not among the digital components included in the modified digital component list; and Suppress displaying the third digital component on the content page.

8. The computer-implemented method according to claim 7, wherein: In response to the suppression, modify the content layout of the content page.

9. The computer-implemented method according to claim 8, further comprising: Provide a message indicating that the third digital component is suppressed to the second content provider.

10. The computer-implemented method according to claim 1 or 2, wherein, The first content provider is different from the second content provider, and the first application is different from the second application.

11. A system for restricting the provision and display of redundant digital components on a client, comprising: A digital component list is stored by the client device, the digital component list specifying a set of digital components available for providing to applications running on the client device; A first digital component provided by a first content provider is received within a first application running on the client device; A set of signals is detected by the client device that specify (i) a first user interaction with the first digital component and (ii) a second user interaction with content provided in response to the first user interaction with the first digital component; Based on the set of signals, the client device determines that a positive user action has been performed by a user of the client device, where the positive user action represents the execution of a specified target action by the user after the first user interaction with the first digital component; Based on the positive user action performed by the user after the first user interaction with the first digital component, the client device modifies the digital component list; A request to access a content page within a second application running on the client device is received; In response to receiving the request to access the content page, a content request including a portion of the modified digital component list that prevents selection of the first digital component in response to the content request is transmitted to the second content provider; In response to the content request, a second digital component is received by the client device and within the second application from the second content provider, where the second digital component is selected from among the digital components included in the modified digital component list; and Provide the second digital component for display on the content page within the second application.

12. The system according to claim 11, wherein, Determining that a positive user action has been performed by a user of the client device includes: Inputting the set of signals associated with the first digital component into a machine learning model, the machine learning model predicting whether the user has a positive user action with respect to the first digital component based on the set of signals associated with the first digital component, where: The machine learning model is trained using training data of a plurality of training digital components, where the training data for each training digital component includes the set of signals associated with the training digital component and a corresponding label indicating whether the user has a positive user action with respect to the training digital component; and Obtain an indication specifying whether a user has a positive user action with respect to the first digital component from the machine learning model and in response to a set of signals associated with the first digital component input into the machine learning model.

13. The system according to claim 11 or 12, wherein, Storing the list of digital components includes storing a ranked list of digital components.

14. The system according to claim 11 or 12, wherein: In response to the content request, receive, by the client device and within the second application, a third digital component from the second content provider, where the third digital component is not among the digital components included in the modified list of digital components; and Suppress displaying the third digital component on the content page.

15. The system according to claim 11 or 12, wherein, The first content provider is different from the second content provider, and the first application is different from the second application.

16. A non - transitory computer - readable medium storing instructions that, when executed by one or more data processing devices, cause the one or more data processing devices to perform operations, the operations including: Store, by the client device, a list of digital components that specifies a set of digital components available for providing to applications running on the client device; Receive, within a first application running on the client device, a first digital component provided by a first content provider; Detect, by the client device, a set of signals specifying (i) a first user interaction with the first digital component and (ii) a second user interaction with content provided in response to the first user interaction with the first digital component; Determine, by the client device and based on the set of signals, that a positive user action has been performed by a user of the client device, where the positive user action represents the execution of a specified target action by the user after the first user interaction with the first digital component; Modify, by the client device, the list of digital components based on the positive user action performed by the user after the first user interaction with the first digital component; Receive a request to access a content page within a second application running on the client device; In response to receiving the request to access the content page, transmit, to the second content provider, a content request including a portion of the modified list of digital components that prevents selection of the first digital component in response to the content request; In response to the content request, receive, by the client device and within the second application, a second digital component from the second content provider, where the second digital component is selected from among the digital components included in the modified list of digital components; and Provide the second digital component for display on the content page within the second application.

17. The non - transitory computer - readable medium according to claim 16, wherein, Determining that a positive user action has been performed by a user of the client device includes: Inputting the set of signals associated with the first digital component into a machine learning model that predicts, based on the set of signals associated with the first digital component, whether the user has a positive user action with respect to the first digital component, where: The machine learning model is trained using training data of multiple training digital components, where the training data for each training digital component includes a set of signals associated with the training digital component and a corresponding label indicating whether the user has a positive user action with respect to the training digital component; and Obtain an indication of whether a user has a positive user action with respect to the first digital component from the machine learning model and in response to a set of signals input into the machine learning model that are associated with the first digital component.

18. The non - transitory computer - readable medium according to claim 16 or 17, wherein, Storing the list of digital components includes storing a ranked list of digital components.

19. The non-transitory computer-readable medium according to claim 16 or 17, wherein: In response to the content request, receive, by the client device and within the second application, a third digital component from the second content provider, where the third digital component is not among the digital components included on the modified list of digital components; and Suppress displaying the third digital component on the content page.

20. The non-transitory computer-readable medium according to claim 16 or 17, wherein The first content provider is different from the second content provider, and the first application is different from the second application.

Images