Self-improving artificial intelligence system

Abstract

One example method includes generating, by an artificial intelligence (AI) system, a ranking of a plurality of digital components based on performance data of the plurality of digital components; creating, by the AI system based on the ranking, a digital component pair that includes a first digital component and a second digital component; generating, by the AI system, first attribute scores of the first digital component and second attribute scores of the second digital component; computing, by the AI system, differences between the first attribute scores and the second attribute scores; identifying one or more target attributes whose corresponding one or more differences are greater than other attributes of the first digital component and the second digital component; and generating, based on the one or more target attributes, a summary indicating at least one reason the first digital component and the second digital component have different performance data.

Description

SELF-IMPROVING ARTIFICIAL INTELLIGENCE SYSTEMBACKGROUND

[0001] This specification relates to data processing and self-improving artificial intelligence system.

[0002] Advances in machine learning are enabling artificial intelligence to be implemented in more applications. For example, a generative model is a type of machine learning model that aims to learn and mimic the underlying distribution of a given dataset. Unlike discriminative models that focus on classifying data into predefined categories, generative models are designed to generate new data that resembles the original training data. Generative models are used in various applications, such as image generation, text synthesis, and data augmentation.SUMMARY

[0003] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of generating, by an artificial intelligence (Al) system, a ranking of a plurality of digital components based on performance data of the plurality of digital components; creating, by the Al system based on the ranking, a digital component pair that includes a first digital component and a second digital component; generating, by the Al system, first attribute scores of the first digital component and second attribute scores of the second digital component, the first attribute scores and the second attribute scores representing respective attributes of the first digital component or the second digital component; computing, by the Al system, differences between the first attribute scores and the second attribute scores, each difference representing a difference of a first attribute score corresponding to an attribute and a second attribute score corresponding to the attribute; identifying, by the Al system, one or more target attributes whose corresponding one or more differences are greater than other attributes of the first digital component and the second digital component; and generating, by the Al system and based on the one or more target attributes, a summary indicating at least one reason the first digital component and the second digital component have different performance data.

[0004] These and other embodiments can each optionally include one or more of the following features.

[0005] In some implementations, performance data of a digital component indicates an acceptance level of the digital component, the plurality of digital components are ranked from the highest acceptance level to the lowest acceptance level, the first digital component has the highest acceptance level, and the second digital component has the lowest acceptance level.

[0006] In some implementations, the performance data includes at least one of clickthrough rate (CTR) or conversion rate (CVR), and the acceptance level is determined based on at least one of the CTR or CVR.

[0007] In some implementations, generating, by the Al system, the first attribute scores of the first digital component and the second attribute scores of the second digital component includes: identifying, by the Al system, a predetermined set of attributes; and for each attribute of the predetermined set of attributes, generating, by the Al system and using a quality model, a first attribute score corresponding to the attribute for the first digital component and a second attribute score corresponding to the attribute for the second digital component.

[0008] In some implementations, methods include transmitting the first digital component, the second digital component, and the one or more target attributes to one or more computing devices associated with one or more evaluators; receiving feedback associated with the one or more target attributes from the one or more computing devices; and updating, based on the feedback, the one or more target attributes.

[0009] In some implementations, a machine learning model is used to generate summaries, and where the computer-implemented method includes refining the machine learning model using the feedback.

[0010] In some implementations, methods include generating, by the Al system and based on the feedback, training data; and refining, by the Al system and using the training data, the machine learning model.

[0011] In some implementations, the machine learning model is a supervised machine learning model, and generating, by the Al system and based on the feedback, the training data includes: including the one or more target attributes as a feature of the training data; and including, in a label of the training data, at least a part of the feedback.

[0012] In some implementations, the machine learning model is trained using a reinforcement learning (RL) algorithm; and generating, by the Al system and based on the feedback, the training data includes: including, in the training data, at least one of the oneor more target attributes, at least a part of the feedback, or a reward of the one or more target attributes.

[0013] In some implementations, methods include updating, based on the one or more target attributes, the second digital component to generate an updated second digital component; inputting the updated second digital component into a performance prediction model to generate predicted performance data of the updated second digital component; and updating the one or more target attributes based on the predicted performance data of the updated second digital component.

[0014] In some implementations, methods include classifying, by the Al system based on the performance data, the plurality of digital components into a first group of digital components and a second group of digital components; generating, by the Al system, attribute scores for each digital component of the first group of digital components and the second group of digital components; determining, by the Al system and based on the attribute scores, a first set of common attributes for the first group of digital components and a second set of common attributes for the second group of digital components; computing, by the Al system, additional differences between attribute scores of the first set of common attributes and attribute scores of the second set of common attributes; identifying, by the Al system, one or more additional target attributes whose corresponding one or more additional differences are greater than other attributes of the first set of common attributes and the second set of common attributes; and generating, by the Al system and based on the one or more additional target attributes, an additional summary indicating at least one reason the first group of digital components and the second group of digital components have different performance data.

[0015] In some implementations, methods include generating, using a machine learning model, a prompt for each digital component of the first group of digital components and the second group of digital components, where the prompt indicates one or more attributes of the digital component, the first set of common attributes are identified based on the attribute scores and a first set of prompts for the first group of digital components, and the second set of common attributes are identified based on the attribute scores and a second set of prompts for the second group of digital components.

[0016] In some implementations, the at least one reason indicates that the one or more target attributes cause performance disparity of the first digital component and the second digital component.

[0017] In some implementations, methods include generating, by the Al system and based on the summary, training data; and refining, by the Al system and using the training data, a generative model that generated the plurality of digital components.

[0018] In some implementations, generating, by the Al system and based on the summary, training data includes: determining, by the Al system, a training digital component whose one or more attribute scores corresponding to the one or more target attributes satisfy one or more conditions; and generating, by the Al system and based on the training digital component, the training data.

[0019] The techniques described herein can be implemented to achieve the following advantages. In some cases, an artificial intelligence (Al) system can determine attribute(s) / attribute score(s) of a digital component that potentially causes good performance of the digital component. Using the identified attribute(s), the Al system can generate — without human intervention — training data to refine the generative model that is used to generate the digital component. These feedback loops enable the generative model to generate more digital components similar to the ones that received positive outcomes and to avoid generating digital components similar to the ones that received negative outcomes. This can reduce the rejections of undesirable, low-quality digital components, and thus reduce wasted computing resources that would be used to, for example, generate and evaluate the low-quality digital components and / or regenerate digital components.

[0020] Additionally, when compared to refining the generative model using a wellperforming digital component without determining the specific attribute(s) leading to its good performance, the techniques described herein enable to determine and focus on a subset of attributes within the digital component. This smaller subset is then used to refine the generative model. By precisely determining attributes associated with good performance, these techniques enhance the training efficiency of generative models. Specifically, they reduce the training iterations required to converge on optimization objectives, thus avoiding the substantial consumption of computing and network resources needed for refining generative models based on no or incorrect attributions of good performance.

[0021] In some implementations, the Al system can verify the initially identified attribute(s) that potentially results in good performance by updating a digital component based on the initially identified attribute(s) and testing the updated digital component. In some cases, the Al system can use a performance prediction model to predict the performance of the updated digital component and use the predicted performance to verifywhether the initially identified attribute(s) is correct (e.g., a positive predicted performance can indicate correct identification of attribute(s)). This can improve the accuracy of determining attributes that contribute to good performance, subsequently enhancing the efficiency of refining generative models.

[0022] In some implementations, the Al system can use a machine learning model to generate a summary indicating the identified attribute(s) that potentially results in good performance. Evaluators can assess the identified attribute(s) and provide feedback on the identified attribute(s). The Al system can use evaluators’ feedback to refine the machine learning model used to generate the summan . These feedback loops enable the machine learning model to generate more summaries similar to the ones that are consistent with the evaluators’ feedback and to avoid generating summaries similar to the ones that are inconsistent with the evaluators’ feedback. This can enhance the accuracy of the attributions of good performance, which in turn can avoid the substantial consumption of computing and network resources needed for re-training generative models based on incorrect attributions of good performance.

[0023] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG. 1 is a block diagram of an example environment for examining digital components and determining attributes that cause performance disparities among different digital components.

[0025] FIG. 2 is a block diagram illustrating interactions between an Al system, a generative model, and a client device implementing innovative aspects of this specification.

[0026] FIG. 3 is a flow chart of an example process for determining causes of performance disparities among digital components.

[0027] FIG. 4 is a block diagram of an example computer sy stem that can be used to perform operations disclosed by this specification.

[0028] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0029] This specification describes techniques for examining atributes leading to performance disparities of digital components and optionally using the identified atributes to refine generative models. Various modifications, alterations, and permutations of the disclosed implementations can be made and will be readily apparent to those of ordinary skill in the art, and the general principles defined can be applied to other implementations and applications, without departing from the scope of the present disclosure. In some instances, one or more technical details that are unnecessary to obtain an understanding of the described subject mater and that are within the skill of one of ordinary skill in the art may be omited so as to not obscure one or more described implementations. The present disclosure is not intended to be limited to the described or illustrated implementations, but to be accorded the widest scope consistent with the described principles and features.

[0030] Artificial intelligence (Al) is a segment of computer science that focuses on the creation of models that can perform tasks act autonomously (e.g., with litle to no human intervention). Artificial intelligence systems can utilize, for example, one or more of machine learning, natural language processing, or computer vision. Machine learning, and its subsets, such as deep learning, focus on developing models that can infer outputs from data. The outputs can include, for example, predictions and / or classifications. Natural language processing, focuses on analyzing and generating human language. Computer vision focuses on analyzing and interpreting images and videos. Artificial intelligence systems can include generative models that generate new content, such as images, videos, text, audio, and / or other content, in response to input prompts and / or based on other information.

[0031] A generative model that generates digital components can be refined using previously generated digital components having good performance. In some cases, a wellperforming digital component can be used to refine the generative model without determining the specific atribute(s) leading to its good performance. However, making no or incorrect attributions of good performance can result in an increased number of training iterations required to converge on optimization objectives. This, in turn, necessitates a substantial consumption of computing and network resources needed for refining generative models.

[0032] To avoid such wasted consumption of computing and network resources, the techniques described throughout this specification enable an Al system to examine atributes of digital components having different performance metrics to determine thoseatribute(s) of the digital components that are the cause of the performance differences, and generate a text / audio output explaining why the digital components have different performance metrics. In some implementations, this information can be used, for example, in a feedback loop to adjust generative models that create new digital components so that the adjusted generative models generate and output new content that has better performance than the content previously generated. In this way, the generative model is improved by not wasting significant computing resources and power generating low quality content.

[0033] As an example, the Al system can examine / compare attribute scores of two digital components — one having good performance and the other having bad performance — and determine / isolate the attribute(s) having large score disparity. Using the identified attribute(s), the Al system can generate — e.g., without human intervention — training data to refine the generative model that is used to generate digital components. By precisely determining attribute differences that cause performance disparities and / or attributes / attribute values associated with good performance, these techniques enhance the training efficiency of generative models. Specifically, they reduce the training iterations required to converge on optimization objectives, thus avoiding the substantial consumption of computing and network resources needed for refining generative models based on no or incorrect attributions of good performance.

[0034] As used throughout this document, the phrase “digital component’' refers to a discrete unit of digital content or digital information (e.g., a video clip, audio clip, multimedia clip, gaming content, image, text, bullet point, Al output, language model output, or another unit of content). A digital component can electronically be stored in a physical memory device as a single file or in a collection of files, and digital components can take the form of video files, audio files, multimedia files, image files, or text files and include advertising information, such that an advertisement is a type of digital component.

[0035] FIG. 1 is a block diagram of an example environment 100 for examining digital components and determining attributes that cause performance disparities among different digital components, according to an implementation of the present disclosure. The example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects electronic document servers 104, client devices 106, digital component servers 108, and a service apparatus 110. The example environment 100 may include many different electronic document servers 104, client devices 106, and digital component servers 108.

[0036] A client device 106 is an electronic device capable of requesting and receiving online resources over the network 102. Example client devices 106 include personal computers, gaming devices, mobile communication devices, digital assistant devices, augmented reality devices, virtual reality devices, and other devices that can send and receive data over the network 102. A client device 106 typically includes a user application, such as a web browser, to facilitate the sending and receiving of data over the network 102, but native applications (other than browsers) executed by the client device 106 can also facilitate the sending and receiving of data over the network 102.

[0037] A gaming device is a device that enables a user to engage in gaming applications, for example, in which the user has control over one or more characters, avatars, or other rendered content presented in the gaming application. A gaming device ty pically includes a computer processor, a memory’ device, and a controller interface (either physical or visually rendered) that enables user control over content rendered by the gaming application. The gaming device can store and execute the gaming application locally or execute a gaming application that is at least partly stored and / or served by a cloud server (e.g.. online gaming applications). Similarly, the gaming device can interface with a gaming server that executes the gaming application and “streams” the gaming application to the gaming device. The gaming device may be a tablet device, mobile telecommunications device, a computer, or another device that performs other functions beyond executing the gaming application.

[0038] Digital assistant devices include devices that include a microphone and a speaker. Digital assistant devices are generally capable of receiving input by way of voice, and respond with content using audible feedback, and can present other audible information. In some situations, digital assistant devices also include a visual display or are in communication with a visual display (e.g., by way of a wireless or wired connection). Feedback or other information can also be provided visually7when a visual display is present. In some situations, digital assistant devices can also control other devices, such as lights, locks, cameras, climate control devices, alarm systems, and other devices that are registered with the digital assistant device.

[0039] As illustrated, the client device 106 is presenting an electronic document 150. An electronic document is data that presents a set of content at a client device 106. Examples of electronic documents include webpages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources. Native applications (e.g., “apps” and / or gaming applications), such as applications installedon mobile, tablet, or desktop computing devices are also examples of electronic documents. Electronic documents can be provided to client devices 106 by electronic document servers 104 (‘'Electronic Doc Servers”).

[0040] For example, the electronic document servers 104 can include servers that host publisher websites. In this example, the client device 106 can initiate a request for a given publisher webpage, and the electronic document server 104 that hosts the given publisher webpage can respond to the request by sending machine executable instructions that initiate presentation of the given webpage at the client device 106.

[0041] In another example, the electronic document servers 104 can include app servers from which client devices 106 can download apps. In this example, the client device 106 can download files required to install an app at the client device 106, and then execute the downloaded app locally (i.e., on the client device). Alternatively, or additionally, the client device 106 can initiate a request to execute the app, which is transmitted to a cloud server. In response to receiving the request, the cloud server can execute the application and stream a user interface of the application to the client device 106 so that the client device 106 does not have to execute the app itself. Rather, the client device 106 can present the user interface generated by the cloud server’s execution of the app and communicate any user interactions with the user interface back to the cloud server for processing.

[0042] Electronic documents can include a variety of content. For example, an electronic document 150 can include native content 152 that is within the electronic document 150 itself and / or does not change over time. Electronic documents can also include dynamic content that may change over time or on a per-request basis. For example, a publisher of a given electronic document (e.g., electronic document 150) can maintain a data source that is used to populate portions of the electronic document. In this example, the given electronic document can include a script, such as the script 154, that causes the client device 106 to request content (e.g., a digital component) from the data source when the given electronic document is processed (e g., rendered or executed) by a client device 106 (or a cloud server). The client device 106 (or cloud server) integrates the content (e.g., digital component) obtained from the data source into the given electronic document to create a composite electronic document including the content obtained from the data source.

[0043] In some situations, a given electronic document (e.g., electronic document 150) can include a digital component script (e.g., script 154) that references the service apparatus 1 10, or a particular service provided by the sendee apparatus 110. In these situations, thedigital component script is executed by the client device 106 when the given electronic document is processed by the client device 106. Execution of the digital component script configures the client device 106 to generate a request for digital components 112 (referred to as a “component request”), which is transmitted over the network 102 to the service apparatus 110. For example, the digital component script can enable the client device 106 to generate a packetized data request including a header and payload data. The component request 112 can include event data specifying features such as a name (or network location) of a server from which the digital component is being requested, a name (or network location) of the requesting device (e.g., the client device 106), and / or information that the service apparatus 110 can use to select one or more digital components, or other content, provided in response to the request. The component request 112 is transmitted, by the client device 106, over the network 102 (e.g., a telecommunications network) to a server of the service apparatus 110.

[0044] The component request 112 can include event data specifying other event features, such as the electronic document being requested and characteristics of locations of the electronic document at which digital component can be presented. For example, event data specifying a reference (e.g., a Uniform Resource Locator (URL)) to an electronic document (e.g., webpage) in which the digital component will be presented, available locations of the electronic documents that are available to present digital components, sizes of the available locations, and / or media types that are eligible for presentation in the locations can be provided to the service apparatus 1 10. Similarly, event data specifying keywords associated with the electronic document (“document keywords”) or entities (e.g., people, places, or things) that are referenced by the electronic document can also be included in the component request 112 (e.g., as payload data) and provided to the service apparatus 110 to facilitate identification of digital components that are eligible for presentation with the electronic document. The event data can also include a search query that was submitted from the client device 106 to obtain a search results page.

[0045] Component requests 112 can also include event data related to other information, such as information that a user of the client device has provided, geographic information indicating a state or region from which the component request was submitted, or other information that provides context for the environment in which the digital component will be displayed (e.g., a time of day of the component request, a day of the week of the component request, a type of device at which the digital component will be displayed, such as a mobile device or tablet device). Component requests 112 can betransmitted, for example, over a packetized network, and the component requests 112 themselves can be formatted as packetized data having a header and payload data. The header can specify a destination of the packet and the pay load data can include any of the information discussed above.

[0046] The sendee apparatus 110 chooses digital components (e.g., third-party' content, such as video files, audio files, images, text, gaming content, augmented reality content, and combinations thereof, which can all take the form of advertising content or nonadvertising content) that will be presented with the given electronic document (e.g., at a location specified by the script 154) in response to receiving the component request 112 and / or using information included in the component request 112.

[0047] In some implementations, a digital component is selected in less than a second to avoid errors that could be caused by delayed selection of the digital component. For example, delays in providing digital components in response to a component request 112 can result in page load errors at the client device 106 or cause portions of the electronic document to remain unpopulated even after other portions of the electronic document are presented at the client device 106.

[0048] Also, as the delay in providing the digital component to the client device 106 increases, it is more likely that the electronic document will no longer be presented at the client device 106 when the digital component is delivered to the client device 106, thereby negatively impacting a user's experience with the electronic document. Further, delays in providing the digital component can result in a failed delivery of the digital component, for example, if the electronic document is no longer presented at the client device 106 when the digital component is provided.

[0049] In some implementations, the service apparatus 110 is implemented in a distributed computing system that includes, for example, a server and a set of multiple computing devices 114 that are interconnected and identify and distribute digital component in response to requests 112. The set of multiple computing devices 114 operate together to identity’ a set of digital components that are eligible to be presented in the electronic document from among a corpus of millions of available digital components (DCi-x). The millions of available digital components can be indexed, for example, in a digital component database 116. Each digital component index entry' can reference the corresponding digital component and / or include distribution parameters (DPi-DPx) that contribute to (e.g., trigger, condition, or limit) the distribution / transmission of the corresponding digital component. For example, the distribution parameters can contributeto (e.g., trigger) the transmission of a digital component by requiring that a component request include at least one criterion that matches (e.g., either exactly or with some prespecified level of similarity) one of the distribution parameters of the digital component.

[0050] In some implementations, the distribution parameters for a particular digital component can include distribution keywords that must be matched (e.g., by electronic documents, document keywords, or terms specified in the component request 112) in order for the digital component to be eligible for presentation. Additionally, or alternatively, the distribution parameters can include embeddings that can use various different dimensions of data, such as website details and / or consumption details (e.g., page viewport, user scrolling speed, or other information about the consumption of data). The distribution parameters can also require that the component request 112 include information specifying a particular geographic region (e.g., country or state) and / or information specifying that the component request 112 originated at a particular type of client device (e.g., mobile device or tablet device) in order for the digital component to be eligible for presentation. The distribution parameters can also specify an eligibility value (e.g., ranking score, or some other specified value) that is used for evaluating the eligibility of the digital component for distribution / transmission (e.g., among other available digital components).

[0051] The identification of the eligible digital component can be segmented into multiple tasks 117a-l 17c that are then assigned among computing devices within the set of multiple computing devices 114. For example, different computing devices in the set 114 can each analyze a different portion of the digital component database 116 to identify various digital components having distribution parameters that match information included in the component request 112. In some implementations, each given computing device in the set 114 can analyze a different data dimension (or set of dimensions) and pass (e.g., transmit) results (Res 1 -Res 3) 118a- 118c of the analysis back to the service apparatus 110. For example, the results 118a-118c provided by each of the computing devices in the set 114 may identify a subset of digital components that are eligible for distribution in response to the component request and / or a subset of the digital component that have certain distribution parameters. The identification of the subset of digital components can include, for example, comparing the event data to the distribution parameters, and identify ing the subset of digital components having distribution parameters that match at least some features of the event data.

[0052] The service apparatus 110 aggregates the results 118a- 118c received from the set of multiple computing devices 114 and uses information associated with the aggregatedresults to select one or more digital components that will be provided in response to the request 112. For example, the service apparatus 110 can select a set of winning digital components (one or more digital components) based on the outcome of one or more content evaluation processes, as discussed below. In turn, the service apparatus 110 can generate and transmit, over the network 102, reply data 120 (e.g., digital data representing a reply) that enable the client device 106 to integrate the set of winning digital components into the given electronic document, such that the set of winning digital components (e.g., winning third-party content) and the content of the electronic document are presented together at a display of the client device 106.

[0053] In some implementations, the client device 106 executes instructions included in the reply data 120, which configures and enables the client device 106 to obtain the set of winning digital components from one or more digital component servers 108. For example, the instructions in the reply data 120 can include a network location (e.g., a URL) and a script that causes the client device 106 to transmit a server request (SR) 121 to the digital component server 108 to obtain a given winning digital component from the digital component server 108. In response to the request, the digital component server 108 will identify the given winning digital component specified in the server request 121 (e.g., within a database storing multiple digital components) and transmit, to the client device 106, digital component data 122 that presents the given winning digital component in the electronic document at the client device 106.

[0054] When the client device 106 receives the digital component data 122, the client device will render the digital component (e.g., third-party content), and present the digital component at a location specified by, or assigned to, the script 154. For example, the script 154 can create a walled garden environment, such as a frame, that is presented within, e.g., beside, the native content 152 of the electronic document 150. In some implementations, the digital component is overlaid over (or adjacent to) a portion of the native content 152 of the electronic document 150, and the service apparatus 110 can specify the presentation location within the electronic document 150 in the reply 120. For example, when the native content 152 includes video content, the service apparatus 110 can specify a location or object within the scene depicted in the video content over which the digital component is to be presented.

[0055] The service apparatus 110 can also include an Al system 160 configured to autonomously generate digital components, either prior to a request 112 (e.g., offline) and / or in response to a request 1 12 (e.g., online or real-time). As described in more detailthroughout this specification, the Al system 160 can collect online content about a specific entity (e.g.. digital component provider or another entity) and generate digital components based on the collected online content using one or more generative models 170.

[0056] Generative models are designed to generate new data that resembles a given training dataset and operate by learning underlying patterns, structures, and relationships present in the training dataset, enabling them to create new samples that share similar characteristics. The primary goal of generative models is to capture inherent complexity of a data distribution, allowing them to produce outputs that exhibit the same diversity and variability found in the original dataset.

[0057] One of the fundamental concepts in generative models is generation of data from random noise or latent variables. The generative models create a mapping between a latent space and data space, permitting generation of entirely novel instances that possess meaningful features. Generative models can be broadly categorized into two main types: likelihood-based and adversarial-based.

[0058] Likelihood-based generative models, such as Variational Autoencoders (VAEs) and Autoregressive Models, focus on learning the probability distribution of the data. V AEs, for instance, employ an encoder-decoder architecture to map data points into a latent space and then decode them back into the data space. This process encourages the model to learn a more structured and continuous representation of the data distribution.

[0059] Adversarial-based generative models, most notably Generative Adversarial Networks (GANs), leverage a different approach. GANs consist of two neural networks: a generator and a discriminator. The generator aims to produce data that is indistinguishable from real data, while the discriminator tries to distinguish between real and generated data. This adversarial process results in the generator improving over time and producing increasingly convincing outputs.

[0060] In some situations, content distributed by the service apparatus 110 and / or generated by the Al system 160 may not be optimal for the environment in which the generated content is presented. In these situations, the sub-optimal nature of the digital component will be evidenced by low performance metrics for the digital component. While the exact performance metrics used are not critical for the implementation of the present innovations, easily understandable metrics include engagement metrics with the content presents, such as presentation time of the digital component, rates of interaction with the digital component, and submission of affirmative user feedback regarding presentation of the digital component (e.g., a “+1”, “like”, or positive survey response regarding the digitalcomponent). Whatever the chosen metric, less optimal digital components will generally have a lower performance measure with respect to the chosen metric, and more optimal digital components will generally have a higher performance measure with respect to the chosen metric.

[0061] Although it can be relatively straight forward to measure performance of digital components with respect to one or more specified metrics, determining why some digital components have higher performance measure than other digital components is often far beyond the capability of human evaluation because of the sheer number of variables involved, and the fact that two different digital components can have tens, hundreds, or even thousands of attributes that can vary widely, or in nuanced ways, such that determining which attribute differences, or combination of attribute differences, are actually causing differences in performance measures between different digital components is generally outside of the ability of humans, particularly in the context of online content distribution where the service apparatus 110 is making digital component serving decisions in less than a second (e.g., in less than a millisecond or microsecond), the number of digital components to be evaluated can be in the tens, hundreds, thousands, or millions, and adjustments to the Al system 160 to reduce the number of suboptimal digital components generated may need to be made on an ongoing basis to prevent wasting resources responding to millions of content requests.

[0062] For example, search systems can respond 100,000+ user search requests every second, and digital component serving decisions, such as those made by the service apparatus 110, are generally made for all, or at least most, of those search requests. It impractical for a human to perform the analysis required to evaluate digital components served in response to those search requests, identify attribute differences that led differences in performance, and adjust the Al model 160 that generates digital components in the time needed to effectively modify7distribution of digital components in the manner discussed herein. Moreover, adjusting the Al model 160 is a real-world adjustment / modification of the model, which cannot practically be performed in the human mind.

[0063] As discussed with reference to the figures that follow, the techniques described herein, which can be implemented in methods, devices, systems, and computer readable medium, provide solutions to the problems discussed above. In some implementations, an Al system can be trained to evaluate pairs of digital components having disparate performance measures for a given metric, and determine which attribute / attribute value, orcombination of attributes / attribute values that (i) differ among the two digital components and (ii) are the cause of the disparate performance of the digital components. The Al system can also be trained to generate a visual / audible summary explaining one or more reasons why one of the digital components in the pair has a higher performance measure than the other digital component in the pair. That summary' can be delivered / presented to a creator of the digital component and / or provided as feedback to a generative model that creates new digital components and / or modifies existing digital components to change the manner in which the generative model creates / modifies digital components in the future. Furthermore, the Al system can output a recommendation of how a digital component creator and / or generative model can modify attributes of the lower performing digital component in the pair (or another digital component) to increase the performance of the lower performing digital component. As such the output of the Al system can be implemented in a “self-improving” Al system that continually improves the content generated by the Al system independent of (e.g., without or with minimal) human intervention.

[0064] FIG. 2 is a block diagram of an example system 200 illustrating interactions between an Al system 160, a generative model 202, and a client device 204. The system 200 is configured to determine sets of attributes / attribute values that cause performance disparities between digital components. The system 200 can use the determined attributes / attribute values to refine generative models, according to an implementation of the present disclosure. In some situations, generative model 202 and client device 204 can, respectively, be the same or similar to the generative model 170 and client device 106 of FIG. 1.

[0065] The generative model 202 can be, for example, a text-to-text generative model, a text-to-image generative model, a text-to-video generative model, an image-to-image generative model, or any other ty pe of generative model. Although a single generative model 202 is depicted in FIG. 2, the generative model 202 can be a set of different generative models that can be invoked for different tasks for which the different generative models are specially trained. For example, one generative model within the set of generative models may be specially trained to perform content summary tasks, while another model may be specially trained to generate digital components, for example, using the output of the specially trained generative model. Furthermore, the set of models can include a generalized generative model that is larger is size, and capable of generating large amounts of diverse datasets, but this generalized model may have higher latency than thespecialized models, which can make it less desirable for use in real-time operations, depending on time latency constraints required to generate content.

[0066] The Al system 160 includes a data collection apparatus 206, a prompt apparatus 208, a digital component serving apparatus 210, a summary generation apparatus 212, a training data generation apparatus 214, and a model refine apparatus 216. The following description refers to these different apparatuses as being implemented independently and each configured to perform a set of operations, but any of these apparatuses could be combined to perform the operations discussed below.

[0067] The Al system 160 is in communication with a memory7structure 232. The memory' structure 232, can include one or more databases. As shown, the memory structure 232 includes a collected data database 218, a digital components database 220, and a training data database 222. Each of these databases 218, 220, and 222 can be implemented in a same hardware memory7device, separate hardware memory devices, and / or implemented in a distributed cloud computing environment.

[0068] At a high level, the client device 204 transmits a query 246 to the Al system 160. In some examples, a user can submit the query using a front-end interface of the Al system 160 (e g., a website, or an application of a computing device). In some cases, the query7246 can be, for example, a request for the Al system 160 to generate a digital component (e.g., an advertisement). For example, a user can input a prompt to request the Al system 160 to generate an image, video, audio, or another digital component, any' of which can take the form of an advertisement.

[0069] In some cases, the user can upload, to the Al system 160, one or more original input digital components (e.g., images, text, and videos) associated with the query (whether as a part of the query' or not), and the original digital component(s) can be used to create output digital components. For example, the original digital component(s) can be image(s) of a product, and the image(s) of the product can be included in one or more new or modified output digital components generated by the Al system 160.

[0070] In some implementations, the user can submit additional query data to the Al system 160, where the additional query data can include data not in the query and can limit digital components generated by the Al system 160. For example, the additional query data can include but not limited to, the geographic location(s) to which the output digital component will be distributed, a language used in the digital component, and / or a vertical industry that will be used to distribute the output digital component. For example, an advertiser can indicate that the digital component is aimed for distribution in NorthAmerican markets, should be in English language, and / or is aimed at the fashion clothing vertical industry. In some examples, the user provides the additional query data in the same prompt that requests to generate the digital component. In other examples, the additional uery data is input separately from the prompt. For example, the Al system 160 can generate one or more follow-up questions in response to the user’s prompt, where the one or more follow-up questions are used to solicit input of the additional query data from the user. For example, the follow-up question(s) can be "which geographic location(s) will the digital component be distributed in,” “which language should the digital component be in,” and / or “which vertical market(s) should the digital component be directed to?”

[0071] In some examples, the Al system 160 can collect, using the data collection apparatus 206. additional query data not input directly by the user. The data collection apparatus 206 is implemented using at least one computing device (e.g., one or more processors), and can include one or more machine learning models. In some cases, the data collection apparatus 206 can obtain an identity of an entity7associated with the query. The identity can include at least one identifier, such as a company or corporation name, a URL, a telephone number, employer ID number, or other means of identifying an entity. The data collection apparatus 206 can obtain the at least one identifier using, for example, an account of the user who submitted the query7or from a partner system. The data collection apparatus 206 can automatically identify, based on the identity of the entity, a data source including information about the entity. These data sources can be, but are not limited to. web pages (e.g., the entity’s landing page), review- compilation pages (e.g., google.com, yelp.com, and crunchbase.com), federal and / or state registries (e.g., the Delaware entity7search tool), private databases, news articles, or other suitable sources. In some implementations, a data crawler application automatically queries a plurality7of databases, performs searches, and extracts information from the results in response to the process being triggered. The information obtained from these data sources can be bulk text data, a combination of text and images, metadata, or other suitable data and / or media.

[0072] In some examples, the data collection apparatus 206 can perform a semantic analysis of the collected information for at least one data source. In some implementations, a single data source is analyzed using semantic analysis. In some implementations, all collected information is analyzed. The semantic analysis can be performed by one or more machine learning algorithms with an overall objective of generating one or more entity attributes associated with the entity. In some cases, the data collection apparatus 206 can perform the semantic analysis using an array of neural networks that operate in series orcan include machine learning algorithms that operate in parallel, or otherwise independently of each other. In some implementations, traditional data analysis can be performed in addition to, or separately from, the machine learning processes. Similar to the additional query data, the one or more entity attributes can include, for example, the geographic location(s) that a digital component will be distributed, a language used in the digital component, and / or a vertical industry that will be used to distribute the digital component. In some examples, the data collection apparatus 206 can include the one or more entity’ attributes in the additional query data.

[0073] The data collection apparatus 206 can store the collected data in the collected data database 218. For example, the data collection apparatus 206 can index the collected data to the query used to collect the data and / or an entity characterized by the collected data so that the collected data can be retrieved from the collected data database 218 for additional operations performed by the data collection apparatus 206 and / or any operations performed by the Al system 160.

[0074] The Al system 160 can generate, using the prompt apparatus 208, an input prompt 242 using the query 246 and / or additional query data. The prompt apparatus 208 can be implemented using at least one computing device (e.g., a device including one or more processors), and can include one or more language models. In some cases, the input prompt 242 can include the query 246 and a set of constraints generated based on, for example, the additional query data. For example, the prompt apparatus 208 can insert, into the input prompt 242, one or more of the entity attribute(s) corresponding to the entity as identified by the data collection apparatus 206. In some implementations, the one or more of the entity attribute(s) inserted into the prompt operates as a contextual constraint that limits content created by the generative model 202 responsive to the input prompt 242. For example, the entity attribute(s) can limit the content created by the generative model to subject matter specified by the entity attribute(s) that is included in the prompt as a contextual constraint.

[0075] The Al system 160 can transmit the input prompt 242 to the generative model 202. The generative model 202 can then generate, based on the input prompt 242. one or more digital components, and transmit the one or more digital components to the Al system 160 as model output 244. In some cases, the Al system 160 can receive a plurality’ of original digital components (e.g., original images) associated with the query. The generative model 202 can generate a plurality of digital components (e.g., advertisements)using the original digital components, where each of the plurality of digital components includes at least one of the plurality of original digital components.

[0076] The Al system 160 can store the generated digital components in the digital components database 220. For example, the Al system 160 can index the generated digital component(s) to the query used to generate the digital component(s) and / or an entity associated with the digital component(s), so that the digital component(s) can be retrieved from the digital components database 220 for additional operations performed by the Al system 160.

[0077] For example, assume that the query 246 is “Generate for a digital component that includes sunglasses and a corresponding description’' and the user uploaded an image of the sunglasses. Also assume that the additional query data indicates that the digital component is intended for distribution to users in Japan who are interested in the fashion clothing vertical market. The input prompt 242 can take the following form:

[0078] Generate a good output: a digital component where the query is “Generate a digital component for sunglasses.” The good_output should be directed to the fashion clothing vertical market in Japan.

[0079] The generative model 202 can generate multiple candidate digital components including the image of the sunglasses and having different backgrounds. For example, a digital component can include a Fuji Mountain scene in the background, a digital component can include a snow scene in the background, a digital component can include a backyard scene in the background, and a digital component can include an Eiffel Tower scene in the background.

[0080] The Al system 1 0 can serve, using the digital component serving apparatus 210, the digital components. The digital component serving apparatus 210 can be implemented using at least one computing device (e.g., a device including one or more processors), and can include one or more machine learning models. In some cases, the digital component serving apparatus 210 can perform rendering, including rendering and formatting the digital component to visually match / blend with the publisher’s website or app layout. The digital component serving apparatus 210 can generate the necessary HyperText Markup Language (HTML), images, or video components to display the digital component. In some examples, the digital component serving apparatus 210 can perform digital component delivery — the rendered digital component is transmitted to the publisher’s website or app, where it is displayed to the user in the designated digital component space.

[0081] In some examples, the digital component serving apparatus 210 can collect performance data of the digital components. In some implementations, the performance data can indicate acceptance levels of the digital components and can be used to evaluate and rank the digital components. The performance data can be based on, for example, user interactions with the digital components. For example, users may interact with the digital component by clicking on it, watching a video, purchasing a product promoted by the digital component, or taking other actions. Examples of performance data include but not limited to, clickthrough rate (CTR), conversion rate (CVR), cost per day (CPD), and other user actions.

[0082] In some implementations, the digital component serving apparatus 210 can operate at the exploration mode or the exploitation mode. The exploration mode can be operated, for example, when the performance data needs to be collected for evaluations of a plurality of candidate digital components generated in response to a query. When the digital component serving apparatus 210 operates at the exploration mode, the digital component serving apparatus 210 can randomly select, in each serving of a candidate digital component, one of the candidate digital components to deliver to users. After a number of deliveries, each candidate digital component has had a chance to be delivered to the users (e.g., audiences of digital components), and the performance data of each candidate digital component has had a chance to be monitored and recorded. On the other hand, the exploitation mode can be operated, for example, when the generative model has been trained to a certain extent (e g., when a predetermined amount of performance data has been received, w hen a predetermined number of training iterations has been performed, or other suitable conditions). When the digital component serving apparatus 210 operates at the exploitation mode, the digital component serving apparatus 210 does not test a plurality of candidate digital components in response to a query. Instead, a digital component generated by the digital component serving apparatus 210 can be directly delivered to users and / or the querier who requested to generate the digital component.

[0083] In some cases, based on the served digital components and the performance data of the digital components, the summary generation apparatus 212 can generate a summary indicating at least one reason why different (e.g., two or more) digital components have different performance data. In some implementations, the summan' generation apparatus 212 can generate the summary' periodically (e.g., even,' seven days, thirty days, or other suitable periods) and / or generate the summary upon the occurrence of one or more predetermined conditions, such as a predetermined amount of performance data has beencollected, a predetermined number of digital components have been served, or other suitable conditions.

[0084] More specifically, the summary generation apparatus 212 can first obtain a plurality of digital components and performance data of the plurality of digital components from, for example, the memory' structure 232. The summary generation apparatus 212 can rank the digital components based on the performance data of the digital components (e.g., highest performance measure to lowest performance measure). In some cases, the summary generation apparatus 212 can create a pair of digital components, based on the ranking of the digital components, where the pair includes a first digital component having good performance (e.g., the digital component having the best performance measure, a performance measure within a specified amount of the best performance, or a performance measure of at least a specified amount) and a second digital component having bad performance (e.g., the digital component having the worst performance measure, a performance measure within a specified amount of the worst performance, or a performance measure below a specified amount).

[0085] The summary generation apparatus 212 can generate first attribute scores of the first digital component and second attribute scores of the second digital component, the first attribute scores and the second attribute scores representing respective attributes / attribute values of the first digital component or the second digital component. The summary- generation apparatus 212 can compute differences between the first attribute scores and the second attribute scores, each difference representing a difference of a first attribute score corresponding to an attribute and a second attribute score corresponding to the attribute. The summary generation apparatus 212 can determine one or more target attributes whose score differences are greater than other attribute scores of the first digital component and the second digital component. The summary generation apparatus 212 can generate, based on the one or more target attributes, a summary indicating at least one reason the first digital component and the second digital component have different performance data. More details are described with respect to FIGS. 3.

[0086] For purposes of example, assume that the high performing digital component of a given digital component pair does not include animation, and the low performing digital component does include animation, such that the attribute scores for animation for the tyvo digital components are 0 and 1, respectively. Further assume that this is the highest disparity of attribute scores among the pair of digital components. In this example, the summary generation apparatus can generate a summary explaining that the inclusion ofanimation in the lower performing digital component is the cause of the disparity of performance between the two digital components. Of course, more complex relationships between multiple attributes can be examined by the system 200, and the summary generated by the summary generation apparatus 212 would explain the combination of attribute / attribute values that are causing the performance disparity.

[0087] In some implementations, the Al system 160 generates, using the training data generation apparatus 214 (which can be implemented using at least one computing device (e.g., a device including one or more processors, and can include one or more machine learning models), training data based on the summaries generated by the summary generation apparatus 212. For example, a summary' can indicate one or more target attributes resulting in the performance disparities. In some cases, the training data generation apparatus 214 can determine or generate a training digital component which performs well on the one or more target attributes (e.g., having high attribute score(s) on the target attribute(s)). In some cases, the training data generation apparatus 214 can determine or generate training digital component based on determining whether attribute score(s) on the target attribute(s) of the training digital component satisfies one or more conditions. The one or more conditions can be, for example, that each attribute score of the target attribute(s) of the training digital component meets or exceeds a predetermined threshold. The training data generation apparatus 214 can then generate training data using the digital component. By using this feedback loop to adjust the generative model 202 that creates new digital components, the adjusted generative model 202 can more precisely generate new digital components having better performance than the digital components previously generated. In this way, the generative model 202 is improved by not wasting significant computing resources and power generating low quality’ content.

[0088] The contents included in the training data can vary based on whether the generative model 202 is an unsupervised machine learning model or a supervised machine learning model. In some implementations, the generative model 202 is an unsupervised machine learning model trained using reinforcement learning algorithm(s). Reinforcement learning, also known as RL, is a machine learning approach used to solve problems by maximizing rewards or achieving specific targets through interactions between an agent and an environment, modeled as a Markov decision process (MDP). RL is an unsupervised learning method that relies on sequential feedback (e.g., rewards) from the environment. During the learning process, the agent observes the state of the environment, selects actions based on a policy, and receives feedback in the form of rewards or scores.

[0089] Through trial and error, the agent iteratively interacts with the environment, aiming to obtain a maximum reward or reach a specific target. Reward signals from the environment are used to evaluate a quality of the agent’s actions rather than guiding the agent on how to perform correct actions. As the environment provides limited feedback, the agent learns through experience, acquires knowledge during interactions, and enhances an action selection policy to adapt to the environment.

[0090] More specifically, the learning process can involve the agent repeatedly observing the state of the environment, making decisions based on behavior, and receiving feedback. The objective of this learning can be to achieve an ideal state value function or policy. In some cases, the state value function can represent the expected cumulative rewards attainable by following the policy.

[0091] In one example, a state value function can be defined as:

[0092] In this equation, Rtrepresents a long-term cumulative reward obtained through executing actions based on the policy 7i. The state value function represents an expectation of a cumulative reward brought by using the policy 7t starting from the state s.

[0093] As an example, assume that the digital components are images and the generative model 202 is trained to generate images based on the query and / or additional query data. The environment’s state can include the following elements:- Query and / or additional query data.Feedback on previously generated images: the feedback can be based on, for example, a summary' indicating one or more target attributes resulting in the performance disparities.The state evolves as the generative model 202 iteratively generates images, receives feedback, and updates its policy.

[0094] The agent’s action can be what the generative model 202 does in response to its current state. In this example, the agent’s action can be to generate an image based on its current policy, the query and / or additional query data, and the feedback on previous generated images. The agent aims to leam a policy that leads to generating images that receive positive feedback, thus maximizing the received rewards while minimizing the penalties. The rewards and / or penalties can be determined based on a reward function.

[0095] The generative model 202 's objective is to leam from these rewards and penalties to improve its digital component generation capabilities iteratively. Over time, the generative model 202 should generate digital components that are more likely to receivepositive feedback, leading to better digital component generation performance. The reward function, in this case, acts as the '’reinforcement signal’" that guides the generative model 202’s learning process.

[0096] In some implementations, when the machine learning model is an unsupervised machine learning model trained using RL algorithm(s), the Al system 160 can include, in the training data, at least one of a digital component (e.g., a digital component having high score(s) on the target attribute(s) identified in the summary generated by the summary generation apparatus 212), an algorithm for generating the digital component, or a reward of the digital component. In some cases, the training data can include other digital component(s) and / or their corresponding data (e.g., other digital component(s), algorithm(s) for generating the other digital component(s). and / or reward(s) of the other digital component(s)).

[0097] The algorithm for generating the digital component can include, for example, one or more steps associated with generating background images for original images. In some cases, the algorithm for generating the digital component can occupy a smaller memory space than the digital component itself. So. in some cases, including the algorithm for generating the digital component in the training data can save storage space compared to including the entire digital component in the training data.

[0098] In some implementations, a reward of a digital component can be generated using a reward function. To output the reward of a digital component, the input(s) to the reward function can include, for example, at least one of acceptance level or performance data of the digital component.

[0099] In some examples, the generative model 202 is a supervised machine learning model. The input(s) to the supervised machine learning model can include one or more features, such as an input prompt (e.g., the input prompt 242), a query (e.g., the query 246), and / or the additional uery data. The output of the supervised machine learning model can be, for example, a digital component (e.g., an image). The supervised machine learning model can be trained using a set of training data and a corresponding set of labels, where the training data can include multiple sets of data relating to multiple queries and the generated digital components for the multiple queries. For example, a piece of training data can include, as feature(s) of a sample, an input prompt, a uery, and / or the additional query data. The label of the piece of training data can be, for example, a digital component corresponding to the feature(s) and having high score(s) on the target attribute(s) identified in the summary generated by the summary generation apparatus 212. The machine learningmodel can be trained by optimizing a loss function based on a difference between the model’s output during training and the corresponding label.

[0100] The training data generation apparatus 214 can store the generated training data in the training data database 222. For example, the training data database 222 can index the generated training data to the query for which the training data is generated and / or an entity associated with the generated training data, so that the generated training data can be retrieved from the training data database 222 for additional operations performed by the training data generation apparatus 214 and / or the Al system 160.

[0101] In some cases, the Al system 160 can refine, using the model refine apparatus 216, the generative model 202 using the training data. The model refine apparatus 216 can be implemented using at least one computing device (e.g., a device including one or more processors), and can include one or more machine learning models. In some cases, the model refine apparatus 216 can refine the generative model 202 immediately upon the occurrence of a particular event. For example, the generative model 202 can be re-trained when an accuracy of the generative model 202 satisfies (meets or below) a predetermined threshold. In some cases, the generative model 202 can be re-trained periodically (e.g.. every seven days or thirty days) and / or re-trained when a certain amount of training data has been generated.

[0102] In some implementations, the retraining can be performed based on feedback data (e.g., the summary) obtained from the analysis of high performing digital components versus low performing digital components. The feedback data can specify the attributes / attribute values that cause digital components to perform better and / or the attributes / attribute values that cause digital components to perform worse. For example, the techniques discussed in this specification can be used to obtain the feedback data, which can include an explanation as to why the performance measure of digital components differs, which is then used to retrain the model to generate higher performing digital components.

[0103] In some implementations, after a period of training and / or refining, the generative model 202 can satisfy one or more predetermined conditions. The one or more predetermined conditions can include, for example, an accuracy of the generative model 202 satisfies (meets or exceeds) a predetermined threshold (e.g., the CTRs of images generated by the generative model 202 satisfy predetermined threshold(s)). When the generative model 202 satisfies the one or more predetermined conditions, the Al system160 can enter the exploitation mode where the Al system 160 can return an output digital component 248 to the client device 204 in response to a query from the client device 204.

[0104] In some implementations, the Al system 160 can use the summaries to limit the contents created by the generative model 202. For example, a summary can indicate that an attribute causes good performance of digital components. The Al system 160 can, for example, indicate to enhance the use associated with the attribute in the input prompt 242. On the other hand, a summary can indicate that an attribute causes low performance of digital components. The Al system 160 can, for example, indicate to avoid or limit the use associated with the attribute in the input prompt 242. The input prompt 242 can then limit the generative model 202 in generating the digital component.

[0105] As an example, assume that the summary indicates that animation causes good performance of digital components. The Al system 160 can generate, for example, using the prompt apparatus 208, an input prompt 242 including a constraint to include or enhance the use of animation in the output digital component. The generative model 202 can then follow the input prompt 242 to generate digital components including or enhancing the use of animation.

[0106] FIG. 3 is a flow chart of an example process 300 for determining causes of performance disparities among digital components, according to an implementation of the present disclosure. Operations of the process 300 can be performed, for example, by the service apparatus 110 of FIG. 1. the Al system 160 of FIG. 2, or another data processing apparatus. The operations of the process 300 can also be implemented as instructions stored on a computer readable medium, which can be non -transitory'. Execution of the instructions, by one or more data processing apparatus, causes the one or more data processing apparatus to perform operations of the process 300.

[0107] At 302, an Al system (e.g., the Al system 160) generates a ranking of aplurality of digital components based on performance data (e.g., CTR, CVR, and / or CPD) of the plurality7of digital components. The Al system can obtain the performance data by collecting non-predicted performance data and / or generating predicted performance data. With respect to collecting the non-predicted performance data, in some implementations, the Al system can serve the plurality of digital components and collect the performance data of the plurality' of digital components based on, for example, the operations associated with the digital component serving apparatus 210 described with respect to FIG. 2.

[0108] With respect to generating predicted performance data, in some implementations, the Al system can generate the predicted performance data of the pluralityof digital components using a machine learning model. The machine learning model can, for example, take a digital component as input and output the predicted performance data of the digital component. The Al system can generate the ranking periodically (e.g., every seven days, thirty days, or other suitable periods) and / or generate the ranking upon the occurrence of one or more predetermined conditions, such as a predetermined amount of performance data has been collected, a predetermined number of digital components have been served, or other suitable conditions.

[0109] In some examples, the performance data of a digital component indicates an acceptance level of the digital component. The Al system can determine the acceptance level based on the performance data in various ways. In some cases, the acceptance level is determined based on one metric of the performance data. For example, the digital component having a higher CTR, higher CVR, or higher CPD has a higher acceptance level. In some cases, the performance data includes at least two different metrics (e.g., at least two of the CTR, CVR, and CPD), and the acceptance level is determined based on a combination of the at least two different metrics. For example, the acceptance level can be determined based on a weighted sum of the at least two different metrics.

[0110] In some implementations, when acceptance levels of the plurality of digital components are determined, the plurality of digital components can be ranked from the highest acceptance level to the lowest acceptance level. Alternatively, in some implementations, the plurality of digital components can be ranked without determining the acceptance levels of the plurality of digital components. In one example, the plurality of digital components can be ranked from the highest CTR, CVR, and / or CPD to the lowest CTR, CVR, and / or CPD. In another example, the plurality of digital components can be ranked using at least two different metrics of the performance data. For example, the at least tw o different metrics can be ranked from a most important metric to a least important metric. First, the digital components can be ranked based on the most important metric from highest to lowest. For digital components that are equal for a metric, the digital components can be ranked based on a lower-ranking metric (i.e., a tiebreaker) to determine their ranking.

[0111] At 304, the Al system creates, based on the ranking, a digital component pair that includes a first digital component and a second digital component. In some cases, the plurality of digital components are ranked from the highest acceptance level to the low est acceptance level, the first digital component has the highest acceptance level, an acceptance level within a specified amount of the highest acceptance level, or at least a specifiedacceptance level. The second digital component can be selected based on it having the lowest acceptance level, an acceptance level within a specified amount of the lowest acceptance level, or less than a specified acceptance level. In some cases, the plurality of digital components are ranked based on CTR, CVR, and / or CPD, the first digital component has the highest CTR, CVR, and / or CPD, whereas the second digital component has the lowest CTR, CVR, and / or CPD.

[0112] At 306. the Al system generates first attribute scores (e.g., attribute values) of the first digital component and second attribute scores of the second digital component, where the first attribute scores represent attributes (e.g., visual or audio attributes) of the first digital component and the second attribute scores represent attributes of the second digital component. In some cases, the Al system identifies a predetermined set of attributes. For each attribute of the predetermined set of attributes, the Al system generates, using a quality model, a first attribute score corresponding to the attribute for the first digital component and a second attribute score corresponding to the attribute for the second digital component. As an example, a predetermined set of attributes includes attribute A, attribute B, and attribute C. For each of attribute A, attribute B. and attribute C. the Al system generates a respective attribute score corresponding to the attribute for each of the first digital component and the second digital component. For example, the first digital component can have the attribute scores of 1, 0. and 1 for attribute A, attribute B, and attribute C, respectively, whereas the second digital component can have the attribute scores of 0, 1 , and 1 for attribute A, attribute B, and attribute C, respectively.

[0113] Examples of the predetermined set of attributes include but not limited to, engagingness of an image in the digital component, aesthetic quality of an image in the digital component, whether collage image is present in the digital component, whether racy and / or sexually content is present in the digital component, whether inappropriate content (e.g., racy, offensive, disgusting, violent, or sensitive) is present in the digital component, whether bad quality image (e.g., blurry, distorted, incomplete, unclear, or cropped) is present in the digital component, whether image that is badly cropped is present in the digital component, whether glare image is present in the digital component, and an overlay ratio of an element (text, logo, or button) inside an image.

[0114] In some examples, the attributes are not predetermined but the Al system can determine the attributes using a machine learning model. The machine learning model can be, for example, a large language model (LLM) trained to generate an output describing attributes of a digital component based on taking the digital component as an input. Forexample, after receiving a digital component, the machine learning model can output that the digital component has high aesthetic quality, includes collage image, and includes inappropriate content. The attributes of the digital components can be predetermined, determined by a machine learning model, or a combination thereof.

[0115] The Al system can compute the attribute scores using, for example, a quality model, such as a quality model specially trained for determining qualities of images and / or videos. In some cases, the Al system can compute the attribute scores of a digital component using more than one machine learning models. For example, the Al system can use a regression model to compute an attribute score for engagingness of an image in the digital component, and use a classification model for determining whether inappropriate content is present in the digital component.

[0116] In some cases, the attribute score can be a numerical value. In one example, the attribute score can be a binary value. For example, for the attribute of whether collage image is present in the digital component, an attribute score of 1 indicates that collage image is present in the digital component, whereas an attribute score of 0 indicates that collage image is absent in the digital component. In another example, the attribute score can be a numerical value falling in a predetermined range (e g., from 0 to 1, from 1 to 10, or any suitable range). For example, for the attribute of aesthetic quality of an image in the digital component, a digital component having an attribute score of 10 has greater aesthetic quality than another digital component having an attribute score of 1.

[0117] At 308, the Al system computes differences between the first attribute scores and the second attribute scores, each difference representing a difference of a first attribute score corresponding to an attribute and a second attribute score corresponding to the attribute. In some examples, the difference can be calculated as the absolute value between the attribute score of the first digital component and the attribute score of the second digital component, with both scores corresponding to the same attribute. For example, assume that the attributes include attribute A, attribute B, and attribute C, the first digital component has the attribute scores of 1, 0, and 1 for attribute A, attribute B, and attribute C, respectively, and the second digital component has the attribute scores of 0, 1, and I for attribute A, attribute B, and attribute C, respectively. The differences between the first attribute scores and the second attribute scores are 1, 1, and 0 corresponding to attribute A, attribute B, and attribute C, respectively.

[0118] In some examples, the difference can be calculated as a percentage difference between the attribute score of the first digital component and the attribute score of thesecond digital component, with both scores corresponding to the same attribute. The percentage difference can be calculated as a ratio of the absolute value between two attribute scores and the range of the attribute scores. For example, assume that the attributes include attribute A, attribute B, and attribute C, the first digital component has the attribute scores of 0.3, 2, and 1 for attribute A, attribute B, and attribute C, respectively, and the second digital component has the attribute scores of 1, 8, and 0 for attribute A, attribute B, and attribute C. respectively. Also assume that the value ranges of attribute A and attribute B are [0, 1] and [0, 10], respectively, and attribute score of attribute C is a binary value. The differences between the first attribute scores and the second attribute scores are 70%, 60%, and 100% corresponding to attribute A, attribute B, and attribute C, respectively. This allows for the comparability of attribute scores, even when they have different value ranges.

[0119] In some implementations, the Al system does not identify an attribute in the first digital component but identifies the attribute in the second digital component. For example, when the Al system identifies attributes based on the output of a machine learning model, the output may not mention the attribute for the first digital component but may mention the attribute for the second digital component. In such case, a default attribute score (e g., 0) can be set for the attribute of the first digital component, so as to allow for the comparability of attribute scores of the attribute, even when the attribute is missing in one of the digital components.

[0120] At 310, the Al system identifies one or more target attributes whose corresponding one or more differences are greater than other attributes of the first digital component and the second digital component. In some cases, the Al system can determine a target attribute having the greatest difference (or multiple target attributes if they all have the same greatest difference) among all attributes. For example, assume that the differences between the first attribute scores and the second attribute scores are 1, 1, and 0 corresponding to attribute A, attribute B, and attribute C, respectively. The target attributes can include attribute A and attribute B, both of which have a difference of 1 and is greater than the difference of attribute C. In some cases, the Al system can determine a predetermined number of target attribute(s) whose corresponding difference(s) are greater than other attributes. For example, assume that the differences between the first attribute scores and the second attribute scores are 70%, 60%, and 100% corresponding to attribute A, attribute B, and attribute C, respectively. Also assume that the predetermined number of target attribute(s) is two. Then, attribute C and attribute A are the target attributes.

[0121] In some implementations, the Al system transmits an assessment, including, for example, the first digital component, the second digital component, and / or the one or more target attributes to one or more computing devices associated with one or more evaluators, where the one or more evaluators can further evaluate the Al system’s assessment. In some cases, the Al system’s identified target attribute(s) may not be desirable attribute(s) to pursue in a digital component, and the human evaluators can recommend excluding such target attribute(s) from being used to generate the summary. For example, in some cases, a digital component having the best performance data is not necessarily a desirable output. An example is a clickbait advertisement, which is an online advertisement that is designed to entice viewers to click on it. Clickbait advertisements often use intriguing or sensationalist headlines, images, or phrases to attract users’ attention and encourage them to click on the advertisements to leam more. Therefore, a clickbait advertisement may excel in one metric of performance data such as CTR. However, clickbait advertisement may not be a desirable output because, for example, it can perform poorly on CVR. By comparing a clickbait advertisement having good performance data and another advertisement that does not include clickbait feature and has bad performance data, the Al system may determine that the clickbait attribute is a target attribute contributing to the good performance data. A human evaluator can evaluate the Al system’s assessment and decide to exclude the clickbait attribute from the summary — to the contrary of the Al system’s assessment, thereby discouraging the creation of future digital components with clickbait.

[0122] In some cases, the computing device(s) associated with the evaluator(s) can transmit the evaluator(s)’s feedback associated with the one or more target attributes to the Al system. The Al system can update, based on the feedback, the one or more target attributes. In one example, the feedback can indicate an undesirable attribute to remove, and the Al system can remove the attribute from the one or more target attributes. In another example, the feedback can indicate a desirable attribute to add, and the Al system can add the attribute to the one or more target attributes.

[0123] In some implementations, the Al system can verify whether the one or more target attributes result in the performance disparities by updating at least one of the first digital component or the second digital component based on the one or more target attributes and testing the updated first digital component and / or the updated second digital component. For example, the Al system can update the second digital component (i.e., the digital component having worse performance data than the first digital component) based on the one or more target attributes to generate an updated second digital component.Examples of such update can include adding a target attribute to the second digital component, such as adding collage feature to the images. The update can also include, for example, enhancing a target attribute of the second digital component, such as enhancing engagingness of an image in the digital component, enhancing aesthetic quality of an image in the digital component, or adjusting cropping of an image in the digital component. Additionally, the update can include, for example, removing a target attribute of the second digital component, such as removing inappropriate (e.g., racy, offensive, disgusting, violent, or sensitive) content in the digital component or removing bad quality image (e.g., blurry, distorted, incomplete, unclear, or cropped) in the digital component. In some cases, the Al system can input the updated second digital component into a performance prediction model to generate predicted performance data of the updated second digital component. The performance prediction model can, for example, take a digital component as input and output predicted performance data (e.g., predicted CTR, CVR, and / or CPD) of the digital component. Alternatively, the Al system can serve the updated second digital component and collect the performance data (i.e.. non-predicted performance data) of the updated second digital component based on, for example, the operations associated with the digital component serving apparatus 210 described with respect to FIG. 2. Using the predicted and / or non-predicted performance data, the Al system can update the one or more target attributes. For example, if the predicted and / or non-predicted performance data of the updated digital component indicates an improved performance, the Al system can determine that the target attribute(s) that was updated in generating the updated digital component likely results in good performance, and can include the target attribute(s) in the summary.

[0124] At 312, the Al system generates, based on the one or more target attributes, a summary indicating at least one reason the first digital component and the second digital component have different performance data. In some cases, the Al system generates the summary using a machine learning model (e.g., an LLM). For example, the machine learning model can take the one or more target attributes as input, and output a narrative indicating at least one reason the first digital component and the second digital component have different performance data. In some cases, the at least one reason indicates that the one or more target attributes cause performance disparity7of the first digital component and the second digital component. For example, the at least one reason can be the first digital component possesses the one or more target attributes, while the second digital component does not.

[0125] In some cases, the Al system can use the evaluators’ feedback to refine the machine learning model that is used to generate the summaries. In some implementations, the Al system generates, based on the evaluators’ feedback, training data, and refines, using the training data, the machine learning model. For example, the evaluators’ feedback can highlight undesirability7of an attribute, such as clickbait. Subsequently, the machine learning model can learn that a preferred summary should either exclude the attribute or address it in a manner that discourages its future use. Therefore, using the evaluators’ feedback as training data allows the machine learning model to leam how to handle attributes identified in the evaluators’ feedback.

[0126] More specifically, when the machine learning model is trained using RL algorithm(s). the training data can include at least one of the one or more target attributes, at least a part of the evaluators’ feedback, or a reward of the one or more target attributes. The reward of the one or more target attributes can be generated using a reward function. In an example binary7reward function, the binary reward function can provide a positive reward (e.g., +1) for a summary that addresses an attribute in a way consistent with the evaluators’ feedback. On the other hand, the binary reward function can provide a negative reward (e.g., -1) for a summary that addresses an attribute in a way inconsistent with the evaluators’ feedback.

[0127] In this example, the machine learning model (trained using RL algorithm(s)) would aim to maximize the cumulative reward it receives over time. If a summary addresses an attribute in a way consistent with the evaluators’ feedback, the model receives a positive reward. This encourages the model to generate more summaries similar to the one that is consistent with the evaluators’ feedback. On the other hand, if a summary addresses an attribute in a way inconsistent with the evaluators’ feedback, the model receives a penalty. This feedback signals the model to avoid generating similar summaries in the future and strive for better results.

[0128] When the machine learning model is a supervised machine learning model, the input(s) to the supervised machine learning model can include one or more features, such as one or more target attributes. The output of the supervised machine learning model can be, for example, a narrative discussing the one or more target attributes. The supervised machine learning model can be trained using a set of training data and a corresponding set of labels, where the training data can include multiple sets of data relating to target attributes and their corresponding narrative. For example, a piece of training data can include, as feature(s) of a sample, one or more target attributes. The label of the piece oftraining data can be, for example, evaluators’ feedback discussing the one or more target attributes. The machine learning model can be trained by optimizing a loss function based on a difference between the model’s output during training and the corresponding label.

[0129] In some implementations, instead of comparing two digital components having different performance data, the Al system can compare two groups of digital components — a first group having good performance data and a second group having bad performance data — and generate a summary indicating at least one reason the first group of digital components and the second group of digital components have different performance data. For example, the Al system can classify, based on the performance data, the plurality of digital components into a first group of digital components and a second group of digital components. The classification can be based on, for example, a ranking of the plurality of digital components. Specifically, the Al system can generate, based on operations associated with step 302 of FIG. 3, a ranking of the plurality of digital components based on performance data of the plurality of digital components. The Al system can then determine one or more digital components on the top of the ranking as the first group of digital components and determine one or more digital components on the bottom of the ranking as the second group of digital components. The Al system can generate attribute scores for each digital component of the first group of digital components and the second group of digital components using, for example, operations associated with step 306 of FIG. 3.

[0130] The Al system can determine, based on the attribute scores, a first set of common attributes for the first group of digital components and a second set of common attributes for the second group of digital components. A common attribute of a group can be, for example, an attribute that is present in most of the digital components in the group. For example, if the quantity of digital components having the attribute satisfies (e.g., meets or exceeds) a predetermined threshold, the attribute can be identified as a common attribute.

[0131] Whether a digital component has an attribute can be determined in various ways. In one example, whether a digital component has an attribute can be determined by, for example, whether the attribute score of the digital component’s attribute satisfies (e.g.. meets or exceeds) a predetermined threshold. In another example, the Al system can generate, using a machine learning model, a prompt for each digital component of the first group of digital components and the second group of digital components, the prompt indicating one or more attributes of the digital component. For example, the prompt can indicate that the digital component has high aesthetic quality, includes collage image, andincludes inappropriate content. Accordingly, the attributes of the digital component can include high aesthetic quality, collage image, and inappropriate content. In some cases, additional attributes can be identified by the prompts generated using the machine learning model that may not have been initially included in the predetermined set of attributes. For each group of the first group of digital components and the second group of digital components, the Al system can determine a set of common attributes based on the attribute scores and / or a set of prompts for the group of digital components. For example, if at least one of the attribute scores or the set of prompts indicate that an attribute is common among the digital components in the group, the attribute can be identified as a common attribute.

[0132] The Al system can compute differences between attribute scores of the first set of common attributes and attribute scores of the second set of common attributes, and determine one or more target attributes whose corresponding difference(s) are greater than other attributes of the first set of common attributes and the second set of common attributes. The Al system can generate, based on the one or more target attributes, a summary indicating at least one reason the first group of digital components and the second group of digital components have different performance data. These operations can be similar to the steps 308, 310, and 312 described with respect to FIG. 3, and the details are omitted here for brevity.

[0133] FIG. 4 is a block diagram of an example computer system 400 that can be used to perform described operations, according to an implementation of the present disclosure. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input / output device 440. Each of the components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. The processor 410 is capable of processing instructions for execution within the system 400. In one implementation, the processor 410 is a single-threaded processor. In another implementation, the processor 410 is a multi-threaded processor. The processor 410 is capable of processing instructions stored in the memory 420 or on the storage device 430.

[0134] The memory’ 420 stores information within the system 400. In one implementation, the memory 420 is a computer-readable medium. In one implementation, the memory 420 is a volatile memory unit. In another implementation, the memory 420 is a non-volatile memory unit.

[0135] The storage device 430 is capable of providing mass storage for the system 400. In one implementation, the storage device 430 is a computer-readable medium. In various different implementations, the storage device 430 can include, for example, a hard diskdevice, an optical disk device, a storage device that is shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large capacity storage device.

[0136] The input / output device 440 provides input / output operations for the system 400. In one implementation, the input / output device 440 can include one or more network interface devices, e.g., an Ethernet card, a serial communication device, e.g., and RS-232 port, and / or a wireless interface device, e.g., and 802.11 card. In another implementation, the input / output device can include driver devices configured to receive input data and send output data to other devices, e.g., keyboard, printer, display, and other peripheral devices 460. Other implementations, however, can also be used, such as mobile computing devices, mobile communication devices, and set-top box television client devices.

[0137] Although an example processing system has been described in FIG. 4, implementations of the subject matter and the functional operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0138] An electronic document (which for brevity will simply be referred to as a document) does not necessarily correspond to a file. A document may be stored in a portion of a file that holds other documents, in a single file dedicated to the document in question, or in multiple coordinated files.

[0139] For situations in which the systems discussed here collect and / or use personal information about users, the users may be provided with an opportunity to enable / disable or control programs or features that may collect and / or use personal information (e.g., information about a user’s social network, social actions or activities, a user’s preferences, or a user’s cunent location). In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information associated with the user is removed. For example, a user’s identity7may be anonymized so that the no personalty identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined.

[0140] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of thesubject matter described in this specification can be implemented as one or more computer programs, i.e.. one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively, or in addition, the program instructions can be encoded on an artificially -generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0141] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0142] The term "data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a crossplatform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web sendees, distributed computing and grid computing infrastructures.

[0143] This document refers to a service apparatus. As used herein, a service apparatus is one or more data processing apparatus that perform operations to facilitate the distribution of content over a network. The service apparatus is depicted as a single block in block diagrams. However, while the service apparatus could be a single device or single set of devices, this disclosure contemplates that the service apparatus could also be a groupof devices, or even multiple different systems that communicate in order to provide various content to client devices. For example, the service apparatus could encompass one or more of a search system, a video streaming service, an audio streaming service, an email service, a navigation service, an advertising sendee, a gaming service, or any other service.

[0144] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0145] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0146] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory (RAM) or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus(USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory’ devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry’.

[0147] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0148] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes amiddleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g.. ad hoc peer-to-peer networks).

[0149] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., forpurposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

[0150] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0151] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0152] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method, comprising: generating, by an artificial intelligence (Al) system, a ranking of a plurality of digital components based on performance data of the plurality of digital components; creating, by the Al system based on the ranking, a digital component pair that includes a first digital component and a second digital component; generating, by the Al system, first attribute scores of the first digital component and second attribute scores of the second digital component, the first attribute scores and the second attribute scores representing respective attributes of the first digital component or the second digital component; computing, by the Al system, differences between the first attribute scores and the second attribute scores, each difference representing a difference of a first attribute score corresponding to an attribute and a second attribute score corresponding to the attribute; identifying, by the Al system, one or more target attributes whose corresponding one or more differences are greater than other attributes of the first digital component and the second digital component; and generating, by the Al system and based on the one or more target attributes, a summary indicating at least one reason the first digital component and the second digital component have different performance data.

2. The computer-implemented method of claim 1, wherein performance data of a digital component indicates an acceptance level of the digital component, the plurality of digital components are ranked from the highest acceptance level to the lowest acceptance level, the first digital component has the highest acceptance level, and the second digital component has the lowest acceptance level.

3. The computer-implemented method of claim 2. wherein the performance data comprises at least one of clickthrough rate (CTR) or conversion rate (CVR), and the acceptance level is determined based on at least one of the CTR or CVR.

4. The computer-implemented method of claim 1, wherein generating, by the Al system, the first attribute scores of the first digital component and the second attribute scores of the second digital component comprises:identifying, by the Al system, a predetermined set of attributes; and for each attribute of the predetermined set of attributes, generating, by the Al system and using a quality model, a first attribute score corresponding to the attribute for the first digital component and a second attribute score corresponding to the attribute for the second digital component.

5. The computer-implemented method of claim 1, comprising: transmitting the first digital component, the second digital component, and the one or more target attributes to one or more computing devices associated with one or more evaluators; receiving feedback associated with the one or more target attributes from the one or more computing devices; and updating, based on the feedback, the one or more target attributes.

6. The computer-implemented method of claim 5, wherein a machine learning model is used to generate summaries, and wherein the computer-implemented method comprises refining the machine learning model using the feedback.

7. The computer-implemented method of claim 6, comprising: generating, by the Al system and based on the feedback, training data; and refining, by the Al system and using the training data, the machine learning model.

8. The computer-implemented method of claim 7, wherein: the machine learning model is a supervised machine learning model; and generating, by the Al system and based on the feedback, the training data comprises: including the one or more target attributes as a feature of the training data; and including, in a label of the training data, at least a part of the feedback.

9. The computer-implemented method of claim 7, wherein: the machine learning model is trained using a reinforcement learning (RL) algorithm; and generating, by the Al system and based on the feedback, the training data comprises:including, in the training data, at least one of the one or more target attributes, at least a part of the feedback, or a reward of the one or more target attributes.

10. The computer-implemented method of claim 1, comprising: updating, based on the one or more target attributes, the second digital component to generate an updated second digital component; inputting the updated second digital component into a performance prediction model to generate predicted performance data of the updated second digital component; and updating the one or more target attributes based on the predicted performance data of the updated second digital component.

11. The computer-implemented method of claim 1, comprising: classifying, by the Al system based on the performance data, the plurality of digital components into a first group of digital components and a second group of digital components; generating, by the Al system, attribute scores for each digital component of the first group of digital components and the second group of digital components; determining, by the Al system and based on the attribute scores, a first set of common attributes for the first group of digital components and a second set of common attributes for the second group of digital components; computing, by the Al system, additional differences between attribute scores of the first set of common attributes and attribute scores of the second set of common attributes; identifying, by the Al system, one or more additional target attributes whose corresponding one or more additional differences are greater than other attributes of the first set of common attributes and the second set of common attributes; and generating, by the Al system and based on the one or more additional target attributes, an additional summary indicating at least one reason the first group of digital components and the second group of digital components have different performance data.

12. The computer-implemented method of claim 11, comprising: generating, using a machine learning model, a prompt for each digital component of the first group of digital components and the second group of digital components, wherein the prompt indicates one or more attributes of the digital component, the first setof common attributes are identified based on the attribute scores and a first set of prompts for the first group of digital components, and the second set of common attributes are identified based on the attribute scores and a second set of prompts for the second group of digital components.

13. The computer-implemented method of claim 1, wherein the at least one reason indicates that the one or more target attributes cause performance disparity of the first digital component and the second digital component.

14. The computer-implemented method of claim 1, comprising: generating, by the Al system and based on the summary, training data; and refining, by the Al system and using the training data, a generative model that generated the plurality of digital components.

15. The computer-implemented method of claim 14, wherein generating, by the Al system and based on the summary, training data comprises: determining, by the Al system, a training digital component whose one or more attribute scores corresponding to the one or more target attributes satisfy one or more conditions; and generating, by the Al system and based on the training digital component, the training data.

16. A computer-implemented artificial intelligence (Al) system comprising: one or more processors; and one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: generating a ranking of a plurality of digital components based on performance data of the plurality of digital components; creating, based on the ranking, a digital component pair that includes a first digital component and a second digital component; generating first attribute scores of the first digital component and second attribute scores of the second digital component, the first attribute scores and the second attribute scores representing respective attributes of the first digital component or the second digital component;computing differences between the first attribute scores and the second attribute scores, each difference representing a difference of a first attribute score corresponding to an attribute and a second attribute score corresponding to the attribute; identifying one or more target attributes whose corresponding one or more differences are greater than other attributes of the first digital component and the second digital component; and generating, based on the one or more target attributes, a summary indicating at least one reason the first digital component and the second digital component have different performance data.

17. The computer-implemented Al system of claim 16, wherein performance data of a digital component indicates an acceptance level of the digital component, the plurality of digital components are ranked from the highest acceptance level to the lowest acceptance level, the first digital component has the highest acceptance level, and the second digital component has the lowest acceptance level.

18. The computer-implemented Al system of claim 17, wherein the performance data comprises at least one of clickthrough rate (CTR) or conversion rate (CVR), and the acceptance level is determined based on at least one of the CTR or CVR.

19. The computer-implemented Al system of claim 16, wherein generating the first attribute scores of the first digital component and the second attribute scores of the second digital component comprises: identifying, by the Al system, a predetermined set of attributes; and for each attribute of the predetermined set of attributes, generating, by the Al system and using a qualify model, a first attribute score corresponding to the attribute for the first digital component and a second attribute score corresponding to the attribute for the second digital component.

20. One or more non-transitory computer readable medium storing instructions, that when executed by a computer-implemented artificial intelligence (Al) system, causes the computer-implemented Al system to perform operations comprising: generating a ranking of a plurality of digital components based on performance data of the plurality of digital components;creating, based on the ranking, a digital component pair that includes a first digital component and a second digital component; generating first attribute scores of the first digital component and second attribute scores of the second digital component, the first attribute scores and the second attribute scores representing respective attributes of the first digital component or the second digital component; computing differences between the first attribute scores and the second attribute scores, each difference representing a difference of a first attribute score corresponding to an attribute and a second attribute score corresponding to the attribute; identifying one or more target attributes whose corresponding one or more differences are greater than other attributes of the first digital component and the second digital component; and generating, based on the one or more target attributes, a summary indicating at least one reason the first digital component and the second digital component have different performance data.

Images