Language model for predicting digital component selection data

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a large language model (LLM) to predict high performance keywords from a set of keywords. The predicted keywords are then used for a retrieval process of the digital component.

Description

Background Art

[0001] The present specification relates to data processing, and in particular, to predicting high-performance keywords from a set of keywords using a language model.

[0002] Digital components are selected as discrete units of digital content or digital information for inclusion in digital content served to a requesting user device. In order to select digital components to be served, the system needs to be able to evaluate which digital components are best suited for serving a particular service instance. One method of selecting digital components is to use keywords associated with the digital components.

[0003] However, finding keywords associated with digital components can be challenging. Keywords are often based on digital content, such as queries that have been received by the service system. There are often billions of queries to choose from. In addition, some systems summarize queries so that providers of digital components or recommendation processes that can identify keywords associated with digital components can search for queries. Other ways of determining keywords can also be used, such as evaluating the content of the digital component itself and determining keywords from the evaluated content.

[0004] Furthermore, when determining keywords, there are often hundreds or even thousands of resulting keywords to choose from. For example, for a single digital component, hundreds of keywords may be generated. Using all of these keywords for selection of digital components is inefficient. Processing such a large set of keywords at service time wastes processing resources, and from a user's perspective, manually maintaining a large set of keywords is impractical. Summary of the invention

[0005] Generally speaking, an innovative aspect of the subject matter described in this specification can be embodied in a method, the method comprising the following actions: obtaining a set of keywords, each keyword set being used in a selection process of one or more digital components, and each keyword in the keywords having associated performance data, the performance data indicating the performance of the keyword in the selection process of one or more digital components; for each keyword set: selecting a proper subset of keywords based on the associated performance data of the keywords, wherein the keywords in the proper subset of the keywords exceed the performance of keywords in the keyword set that are not in the proper subset of the keywords, and deleting the keywords in the proper subset of the keywords from the keyword set to generate a training set of keywords, wherein the training set of keywords is a keyword set without the deleted keywords; training a language model to generate a trained language model, wherein the trained language model predicts the deleted keywords from the training set for each keyword; processing a set of characters related to a target entity through the trained language model to predict a keyword recommendation set for the keyword; and providing the keyword recommendation set to an acceptance process; for each keyword in the keyword recommendation set accepted by the acceptance process, associating the keyword with the target entity for use in the selection process of selecting one or more digital components for the target entity.

[0006] These and other embodiments can each optionally include one or more of the following aspects. In one aspect, the associated performance data indicating the performance of the keyword includes click data specifying a click performance indicator for each query matched by the keyword and a landing page match metric for each query matched by the keyword.

[0007] In one aspect, processing the set of words related to the target entity through the trained language model includes: processing the set of keywords related to the target entity used in the selection process of one or more digital components through the trained language model to predict a keyword recommendation set of keywords.

[0008] In one aspect, the target entity is a web page; and the set of words associated with the target entity are keywords derived from the web page.

[0009] In one aspect, the target entity is a digital component used for presentation via a web page; and the word set associated with the target entity is a keyword determined for the digital component.

[0010] In one aspect, deleting keywords from the proper subset of keywords to generate a training set of keywords from the set of keywords includes deleting the proper subset of keywords from the proper subset of keywords.

[0011] In one aspect, keywords are deleted from a proper subset of keywords to generate a training set of keywords from the set of keywords, and all keywords are deleted from the proper subset of keywords.

[0012] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages. The technical implementations discussed in this specification identify high-performance keywords from an input set of keywords. High-performance keywords may be included in the input set, or may be keywords that are not included in the input set. This enables the identification of high-performance keywords that do not need to be in the input set, which is an improvement over the previous process of selecting a subset of keywords from the input set. Therefore, the machine learning model achieves tasks that cannot actually be accomplished by simply selecting a subset of keywords from the input set.

[0013] Additionally, by training on an actual set of keywords and associated performance data, the resulting model is less likely to exhibit performance bias based on the performance data of the predicted keywords themselves if the highest performing keywords are removed from the training set.

[0014] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of an example environment in which a language model predicts keywords for selecting a digit component.

[0016] Figure 2A is an illustration of the user interface for the keyword selection process.

[0017] Figure 2B It is a system flow chart of the keyword selection process.

[0018] Figure 3A is a system flow diagram of a system for training a language model to predict high-performance keywords.

[0019] Figure 3B is a flowchart of the process for training a language model to predict high-performance keywords.

[0020] Figure 4 is a flow chart of a process for predicting high-performance keywords from an input set of keywords using a language model.

[0021] Figure 5A and Figure 5B is a flow chart of an example process for determining a set of keywords for training a language model.

[0022] Figure 6 is a block diagram of an example computer.

[0023] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0024] The technology described in this written description involves a large language model (LLM) that is trained to recommend (predict) keywords that provide statistically better performance for digital components (e.g., impressions of ads, or clicks on ads, etc.), and technology that uses a log-based approach to provide advertisers with "keyword suggestions" for several keywords that can provide high-value returns.

[0025] A keyword is a term used to match a user query, typically one or more words, such as an n-gram. Matching can be based on semantic similarity, historical performance, stem matching, and other matching techniques. When a keyword indicates that a user query matches, a digital component associated with the keyword can be selected to be presented to the user who submitted the keyword.

[0026] In an implementation, a system obtains a set of keywords. Each keyword in the keywords is used in a selection process for one or more digital components. The keywords have associated performance data indicating the performance of the keyword during the selection process. For example, the performance data may be from a log storing performance data quantifying the performance of an existing set of keywords currently used for a digital component.

[0027] In some implementations, keywords in each set are selected by determining the user queries that the keywords match. Then, a performance metric is determined for each user query. An example performance metric is click-through rate, but other performance metrics may also be used. The performance metric of the query is then used to determine the performance metric of the keyword that the query matches. In variations of these implementations, for each user query, a landing page match metric is also determined. The landing page match metric is a measure of how well the user query "matches" the landing page, for example, a landing page that is semantically related to the landing page used to navigate to the landing page for the matching query, or a term of a form (root, synonym, etc.) is found in the landing page, etc. The performance metric of the keyword is then determined based on the performance metric of each query that the keyword matches and the landing page match metric.

[0028] In general, keywords are more suitable for training when a small number of queries with high performance metrics match landing pages. This is because some keywords match queries that perform well for landing pages but cannot be derived from landing pages. Since such keywords are difficult to determine in advance, this log-based analysis can be used to determine the set of training keywords that meet these criteria for training LLMs.

[0029] Keyword sets are used to generate training sets for LLM. First, for each keyword set, a proper subset of keywords is selected. The proper subset is based on the associated performance data of the keywords in the set, and the keywords in the proper subset of keywords are the best performing keywords in the set. In other words, the keywords in the proper subset of keywords exceed the performance of keywords in the keyword set that are not in the proper subset of keywords.

[0030] Then, for each keyword set, keywords are deleted from the proper subset of keywords to generate a training set of keywords from the keyword set. The training set of keywords is the keyword set without the deleted keywords. All the best performing keywords can be deleted, or different combinations of the best performing keywords can be deleted. In the latter case, multiple (or more) training sets can be generated from a single keyword set.

[0031] The LLM is then trained to predict the removed keywords from the training set for each keyword. Once the model is fully trained, the set of keywords related to the target entity is processed to predict a keyword recommendation set for the keywords. The target entity can be, for example, an advertising group, and the word set can be keywords that have been selected or recommended for that advertising group.

[0032] Therefore, by training the LLM on a set of high-performing keywords, the LLM can predict a set of keywords that are likely to be high-performing keywords that meet or exceed the performance of the input keywords for an input set of keywords, with the highest performing keyword subset removed. In addition, because the LLM is trained on keywords that match the landing page with fewer matching queries, the LLM predicts keywords that cannot be easily derived from the landing page. This enables the LLM to predict keywords that otherwise cannot be derived from semantic analysis or other traditional keyword expansion and derivation techniques.

[0033] The keyword recommendation set is then provided to an acceptance process, and for each keyword in the keyword recommendation set accepted by the acceptance process, the keyword is associated with a target entity for use in a selection process for selecting one or more digital components for the target entity. In some implementations, the target entity is a web page, and the keyword set associated with the target entity is keywords derived from the web page. In other implementations, the target entity is a digital component used for presentation via the web page, and the keyword set associated with the target entity is keywords previously determined for the digital component.

[0034] These and additional features are described in more detail below.

[0035] As used throughout this document, the phrase "digital component" refers to a discrete unit of digital content or digital information (e.g., a video clip, an audio clip, a multimedia clip, game content, an image, text, a gist, an artificial intelligence output, a language model output, or another unit of content). A digital component may be electronically stored in a physical memory device as a single file or in a collection of files, and a digital component may take the form of a video file, an audio file, a multimedia file, an image file, or a text file and include advertising information, such that an advertisement is a type of digital component.

[0036] Figure 1 1 is a block diagram of an example environment 100 in which a language model predicts keywords for selection of digital components. The example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects an electronic document server 104, a user device 106, a digital component server 108, and a service device 110. The example environment 100 may include many different electronic document servers 104, user devices 106, and digital component servers 108.

[0037] Client devices 106 are electronic devices capable of requesting and receiving online resources over network 102. Example client devices 106 include personal computers, gaming devices, mobile communication devices, digital assistant devices, augmented reality devices, virtual reality devices, and other devices that can send and receive data over network 102. Client devices 106 typically include user applications (such as web browsers) to facilitate sending and receiving data over network 102, but native applications (other than browsers) executed by client devices 106 may also facilitate sending and receiving data over network 102.

[0038] A game device is a device that enables a user to participate in a game application, for example, in which a user controls one or more characters, avatars, or other rendered content presented in a game application. A game device typically includes a computer processor, a memory device, and a controller interface (physical or visually rendered) that enables a user to control the content rendered by the game application. The game device can store and execute game applications locally, or execute game applications (e.g., online game applications) that are at least partially stored and / or served by a cloud server. Similarly, the game device can be docked with a game server that executes the game application and "streams" the game application to the game device. The game device can be a tablet device, a mobile telecommunications device, a computer, or another device that performs other functions in addition to executing the game application.

[0039] The digital assistant device includes a device with a microphone and a speaker. The digital assistant device is generally capable of receiving input by voice, and responding to the content using audible feedback, and can present other audible information. In some cases, the digital assistant device also includes a visual display or communicates with a visual display (e.g., by wireless or wired connection). When there is a visual display, feedback or other information can also be provided visually. In some cases, the digital assistant device can also control other devices, such as lights, locks, cameras, climate control devices, alarm systems, and other devices registered with the digital assistant device.

[0040] As shown, client device 106 is presenting electronic document 150. An electronic document is data that presents a set of content at client device 106. Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, search result pages, and feeds. Native applications (e.g., "apps" and / or game applications) (such as applications installed on mobile, tablet, or desktop computing devices) are also examples of electronic documents. Electronic documents can be provided to client devices 106 by electronic document servers 104 ("Electronic Doc Servers").

[0041] For example, electronic document server 104 may include a server hosting a publisher website. In this example, client device 106 may initiate a request for a given publisher web page, and electronic server 104 hosting the given publisher web page may respond to the request by sending a machine executable instruction that initiates presentation of the given web page at client device 106.

[0042] In another example, the electronic document server 104 may include an app server from which the client device 106 may download the app. In this example, the client device 106 may download the files required to install the app at the client device 106, and then execute the downloaded app locally (e.g., on the client device). Alternatively or in addition, the client device 106 may initiate a request to execute the app, which is transmitted to the cloud server. In response to receiving the request, the cloud server may execute the application and stream the user interface of the application to the client device 106, so that the client device 106 does not have to execute the app itself. Instead, the client device 106 may present the user interface generated by the cloud server executing the app, and transmit any user interaction with the user interface back to the cloud server for processing.

[0043] An electronic document may include a variety of content. For example, an electronic document 150 may include native content 152 that is located within the electronic document 150 itself and / or does not change over time. An electronic document may also include dynamic content that may change over time or on a per-request basis. For example, a publisher of a given electronic document (e.g., electronic document 150) may maintain data sources for populating various portions of the electronic document. In this example, a given electronic document may include a script, such as script 154, that causes a client device 106 to request content (e.g., a digital component) from a data source when the given electronic document is processed (e.g., rendered or executed) by a client device 106 (or a cloud server). The client device 106 (or cloud server) integrates the content (e.g., a digital component) obtained from the data source into a given electronic document to create a composite electronic document that includes the content obtained from the data source.

[0044] In some cases, a given electronic document (e.g., electronic document 150) may include a digital component script (e.g., script 154) that references a service device 110 or a particular service provided by the service device 110. In these cases, the digital component script is executed by the client device 106 when the given electronic document is processed by the client device 106. The execution of the digital component script configures the client device 106 to generate a request for a digital component 112 (referred to as a "component request"), which is transmitted to the service device 110 via the network 102. For example, the digital component script may enable the client device 106 to generate a packetized data request including header and payload data. The component request 112 may include event data specifying characteristics, such as the name (or network location) of the server for which the digital component is being requested, the name (or network location) of the requesting device (e.g., client device 106), and / or information that the service device 110 may use to select one or more digital components or other content to be provided in response to the request. The component request 112 is transmitted by the client device 106 to a server of the service device 110 via the network 102 (e.g., a telecommunications network).

[0045] The component request 112 may include event data specifying other event characteristics, such as characteristics of the electronic document being requested and the location of the electronic document at which the digital component may be presented. For example, event data specifying a reference (e.g., a URL) of an electronic document (e.g., a web page) in which the digital component is to be presented, an available location of the electronic document that can be used to present the digital component, the size of the available location, and / or a media type eligible for presentation at the location may be provided to the service device 110. Similarly, event data specifying keywords associated with the electronic document ("document keywords" or simply "keywords") or entities referenced by the electronic document (e.g., people, places, or things) may also be included in the component request 112 (e.g., as payload data) and provided to the service device 110 to facilitate identifying digital components eligible for presentation with the electronic document. The event data may also include a search query submitted from the client device 106 to obtain a search results page. For example, a query received by a user of the search system may be stored in the query storage device 111.

[0046] The component request 112 may also include event data related to other information, such as information that has been provided by a user of the client device, geographic information indicating the state or region in which the component request was submitted, or other information providing the context of the environment in which the digital component will be displayed (e.g., the time of day of the component request, the day of the week of the component request, the type of device (such as a mobile device or a tablet device) that will display the digital component. The component request 112 may be transmitted, for example, over a packetized network, and the component request 112 itself may be formatted as packetized data having a header and payload data. The header may specify a destination for the packet, and the payload data may include any of the information discussed above.

[0047] The service device 110 selects a digital component (e.g., third-party content such as video files, audio files, images, text, game content, augmented reality content, and combinations thereof, all of which may take the form of advertising content or non-advertising content) to be presented with a given electronic document (e.g., at a location specified by the script 154) in response to receiving the component request 112 and / or using information included in the component request 112.

[0048] In some implementations, the digital component is selected in less than one second to avoid errors that may result from delayed selection of the digital component. For example, a delay in providing the digital component in response to the component request 112 may cause page loading errors at the client device 106 or cause portions of the electronic document to remain unpopulated even after other portions of the electronic document are presented at the client device 106.

[0049] Moreover, as the delay in providing the digital component to the client device 106 increases, it becomes more likely that the electronic document will no longer be present at the client device 106 when the digital component is delivered to the client device 106, thereby negatively affecting the user's experience of the electronic document. Additionally, the delay in providing the digital component may cause the delivery of the digital component to fail, for example, if the electronic document is no longer present at the client device 106 when the digital component is provided.

[0050] In some implementations, the service device 110 is implemented in a distributed computing system that includes, for example, a server and a set of multiple computing devices 114 that are interconnected and identify and distribute digital components in response to the request 112. The set of multiple computing devices 114 operate together to select from millions of available digital components (DC 1-x ) to identify sets of digital components that are eligible for presentation in electronic documents. Millions of available digital components may be indexed, for example, in the digital component database 116. Each digital component index entry may reference a corresponding digital component and / or include distribution parameters (DP1 to DP2) that facilitate (e.g., trigger, constrain, or limit) the distribution / transmission of the corresponding digital component. x For example, the distribution parameters may facilitate (e.g., trigger) the transmission of the digital component by requiring the component request to include at least one criterion that matches (e.g., completely or with some pre-specified level of similarity) one of the distribution parameters of the digital component.

[0051] In some implementations, the distribution parameters for a particular digital component may include a distribution keyword that must be matched (e.g., to an electronic document, a document keyword, or a term specified in a component request 112) in order for the digital component to be eligible for presentation. Additionally or alternatively, the distribution parameters may include embeddings that may use various different data dimensions, such as website details and / or consumption details (e.g., page viewports, user scrolling speed, or other information about data consumption). The distribution parameters may also require that the component request 112 include information specifying a particular geographic region (e.g., a country or state) and / or information specifying that the component request 112 originated from a particular type of client device (e.g., a mobile device or a tablet device) in order for the digital component to be eligible for presentation. The distribution parameters may also specify an eligibility value (e.g., a ranking score, or some other specified value) that is used to evaluate the eligibility of the digital component for distribution / transmission (e.g., with other available digital components).

[0052] The identification of eligible digital components may be split into a plurality of tasks 117a to 117c, which are then assigned among the computing devices within the set 114 of the plurality of computing devices. For example, different computing devices in the set 114 may each analyze different portions of the digital component database 116 to identify various digital components having distribution parameters that match the information included in the component request 112. In some implementations, each given computing device in the set 114 may analyze different data dimensions (or groups of dimensions) and pass (e.g., transmit) the results of the analysis (Res 1 to Res 3) 118a to 118c back to the service device 110. For example, the results 118a to 118c provided by each of the computing devices in the set 114 may identify a subset of digital components that are eligible for distribution in response to the component request and / or a subset of digital components having certain distribution parameters. The identification of the subset of digital components may include, for example, comparing the event data to the distribution parameters, and identifying a subset of digital components having distribution parameters that match at least some features of the event data.

[0053] The service device 110 aggregates the results 118a-118c received from the set 114 of multiple computing devices and uses information associated with the aggregated results to select one or more digital components to be provided in response to the request 112. For example, the service device 110 can select the winning digital component set (one or more digital components) based on the results of one or more content evaluation processes, as described below. In turn, the service device 110 can generate and transmit reply data 120 (e.g., digital data representing the reply) over the network 102, which enables the client device 106 to integrate the winning digital component set into a given electronic document so that the winning digital component set (e.g., the winning third-party content) and the content of the electronic document are presented together at the display of the client device 106.

[0054] In some implementations, the client device 106 executes instructions included in the reply data 120 that configure the client device 106 and enable it to obtain the winning set of digital components from one or more digital component servers 108. For example, the instructions in the reply data 120 may include a network location (e.g., a uniform resource locator (URL)) and a script that causes the client device 106 to transmit a server request (SR) 121 to the digital component server 108 to obtain a given winning digital component from the digital component server 108. In response to the request, the digital component server 108 will identify the given winning digital component specified in the server request 121 (e.g., within a database storing a plurality of digital components) and transmit digital component data (DC data) 122 to the client device 106 that presents the given winning digital component in an electronic document at the client device 106.

[0055] When the client device 106 receives the digital component data 122, the client device will render the digital component (e.g., third-party content) and present the digital component at a location specified by or assigned to the script 154. For example, the script 154 may create a walled garden environment (such as a frame) that is presented within, e.g., next to, the native content 152 of the electronic document 150. In some implementations, the digital component is overlaid on (or adjacent to) a portion of the native content 152 of the electronic document 150, and the service device 110 may specify the presentation location within the electronic document 150 in the reply 120. For example, when the native content 152 includes video content, the service device 110 may specify a location or object within a scene depicted in the video content above which the digital component is to be presented.

[0056] The performance of keywords that resulted in the placement of digital components may be monitored and stored in a log within the service device 110. For example, data describing click-through rates, conversion rates, cost per click, and other data may be monitored for queries that match keywords associated with digital components. A particular keyword may have multiple performance data sets, each of which is associated with a particular digital component or group of digital components that were placed using that keyword. For example, a first digital component provider may have selected the keyword "hiking boots" to place a first set of digital components, while a second digital component provider may have selected the keyword "hiking boots" to place a second set of digital components. Thus, the keyword "hiking boots" will have a first performance data set for the first set of digital components, and a second performance data set for the second set of digital components.

[0057] The service device 110 may also include an artificial intelligence system 160 configured to generate data for identifying digital components. As described in more detail in this specification, the artificial intelligence ("AI") system 160 may generate digital component selection data (keywords) for the digital components. The digital component providers may then use these keywords as selection criteria for their digital components. Keywords may be single words or terms, or a combination of terms and numbers.

[0058] Large language models are models that are trained to generate and understand human language. LLMs are trained on large datasets of text and code, and they can be used for a variety of tasks. For example, LLMs can be trained to translate text from one language to another; summarize text, such as website content, search results, news articles, or research papers; answer questions about text, such as "What is the capital of Georgia?"; create chatbots that can have conversations with humans; and generate creative text, such as poetry, stories, and code.

[0059] The LLM can also be trained for a specific task. In the implementation described in this document, the LLM is trained to predict high-performance keywords from an input set 174 of keywords as output 172. The predicted high-performance keywords can be keywords included in the input set of keywords, or can be keywords that are not in the input set of keywords.

[0060] The language model 170 may be any suitable language model neural network that receives an input sequence consisting of text tokens selected from a vocabulary and autoregressively generates an output sequence consisting of text tokens from the vocabulary. For example, the language model 170 may be a Transformer-based language model neural network or a recurrent neural network-based language model.

[0061] In some cases, language model 170 may be referred to as an autoregressive neural network when the neural network used to implement language model 170 autoregressively generates an output sequence of tokens. More specifically, the autoregressively generated output is created by generating each particular token in the output sequence conditioned on the current input sequence, including any tokens that precede the particular text token in the output sequence (i.e., tokens that have been generated for any previous positions in the output sequence that precede the particular position of the particular token), and contextual input that provides context for the output sequence.

[0062] For example, when generating a word-gram at any given position in the output sequence, the current input sequence may include the word-gram at any previous position in the input sequence and the output sequence that precedes the given position. As a specific example, the current input sequence may include the input sequence followed by the word-gram at any previous position in the output sequence that precedes the given position. Optionally, the input and current output sequences may be separated by one or more predetermined word-grams within the current input sequence.

[0063] More specifically, to generate a particular word-gram at a particular position within the output sequence, the neural network of the language model 170 may process the current input sequence to generate a score distribution (e.g., a probability distribution) that assigns a corresponding score (e.g., a corresponding probability) to each word-gram in the word-gram vocabulary. The neural network of the language model 170 may then use the score distribution to select a word-gram from the vocabulary as the particular word-gram. For example, the neural network of the language model 170 may greedily select the word-gram with the highest score, or may sample the word-gram from the distribution, for example, using kernel sampling or another sampling technique.

[0064] As a specific example, the language model 170 can be an autoregressive Transformer-based neural network that includes (i) multiple attention blocks, each of which applies a self-attention operation; and (ii) an output subnetwork that processes the output of the last attention block to generate a score distribution.

[0065] The language model 170 may have any of a variety of Transformer-based neural network architectures. Examples of such architectures include those described in J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al., Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022; J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HF Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J.Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Elsen, SMJayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L.Sifre, L. Martens, XL Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya,D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C.Jones,J. Bradbury, M.Johnson, B.A. Hechtman, L. Weidinger, I. Gabriel, W.S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving, “Scaling language models: Methods, analysis & insights from training gopher,” CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” (Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer), arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, ApoorvKulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le, Towards a human-like open-domain chatbot, CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al., “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, 2020. Examples of such models include multi-task unified models, zero-shot models, domain-specific models, or language representation models. .

[0066] However, in general, a transformer-based neural network includes a sequence of attention blocks, and during processing of a given input sequence, each attention block in the sequence receives a corresponding input hidden state for each input token in the given input sequence. The attention block then updates each of the hidden states at least in part by applying self-attention to generate a corresponding output hidden state for each of the input tokens. The input hidden state of the first attention block is an embedding of the input token in the input sequence, and the input hidden state of each subsequent attention block is an output hidden state generated by the previous attention block.

[0067] In this example, the output subnetwork processes the output hidden state generated by the last attention block in the sequence for the last input token in the input sequence to generate a score distribution.

[0068] Typically, because language models are autoregressive, the service device 110 can use the same language model 170 to generate multiple different candidate output sequences in response to the same request, for example, by using beam search decoding from a score distribution generated by the language model 170, using a sampling and sorting decoding strategy, by using different random seeds for the pseudo-random number generator used when sampling different runs of the language model 170, or using another decoding strategy that exploits the autoregressive properties of the language model.

[0069] In some implementations, language model 170 is pre-trained, i.e., trained on language modeling tasks that do not require providing evidence in response to user questions, and service device 110 (e.g., using AI system 160) causes language model 170 to generate an output sequence according to a predetermined grammar through natural language prompts in the input sequence.

[0070] For example, service device 110 (e.g., AI system 160) or a separate training system pre-trains language model 170 (e.g., a neural network) on a language modeling task, such as a task that requires predicting the next token in the training data after the current sequence given a current text token sequence. As a specific example, language model 170 can be pre-trained on a large text dataset (e.g., text publicly available from the Internet or another text corpus) for a maximum likelihood objective.

[0071] In the implementation described in this specification, the language model 170 is trained to predict high-performance keywords from a set of keywords and the performance data of the keywords. Once trained, the language model 170 is used to predict keywords, which are then used in the retrieval process of the digital component.

[0072] refer to Figure 2A and Figure 2B The recommendation and selection of high-performance keywords are shown. Specifically, Figure 2A is a diagram of a user interface 200 for a keyword selection process, and Figure 2B is a system flow diagram 250 of the keyword selection process.

[0073] exist Figure 2A In , a keyword set 202 represented by {keyword set} is provided as input. The keyword set 202 can be keywords that have been manually curated, or keywords generated by an automated process, or a combination of both. Generally, the keywords describe a topic related to the digital component that the keywords are used to select. For example, in Figure 2A In the example embodiment, keyword set 202 may be related to hiking boots, and the digital component may describe hiking boots, or a provider of hiking boots.

[0074] The user interface 200 may be generated by the keyword recommendation front end 252. The keyword set 202 is provided to the language model 170. At inference time, the language model 170 processes the keyword set as input and generates a keyword recommendation set 204. The keyword recommendation set 204 is keywords that are predicted to be high-performance keywords, where the performance is measured according to one or more performance indicators, such as click-through rate data that specifies a click performance indicator for each keyword, or a combination of click-through rate and conversion rate, or some other combination that the model 170 is trained on.

[0075] like Figure 2A As shown, the keywords predicted to perform very well are leather hiking boots, men's hiking boots, hiking boots, and jungle boots. The user can select the corresponding keywords for selection of the digital component by clicking the corresponding selection boxes 204A, 204B, 204C, and 204D. Alternatively, the user can select all recommended keywords 204 by clicking the "Accept All" button 206.

[0076] refer to Figure 3A and Figure 3B To describe the training of language model 170. Specifically, Figure 3A is a system flow chart 300 for a system for training a language model 170 to predict high performance keywords, and Figure 3B is a flow chart of an example process 350 for training a language model to predict high performance keywords. System 300 and process 350 may be implemented in a computer system having one or more computers and storage devices in data communication with each other.

[0077] In operation, process 350 obtains a set of keywords (352). For example, Figure 3A As shown, keyword set 302 is a set of keywords used in the selection process of one or more digital components. Each keyword in the set has associated performance data indicating the performance of the keyword in the selection process of one or more digital components. Figure 3A As shown, there are Q keyword sets in the training set of keyword K and the associated performance data P of each corresponding keyword: [{K11...K1N}; {P11...P1N}], [{K21...K2M}; {P21...P2M}], ... [{KQ1...KQX}; {PQ1...PQX}]. For ease of description, the keywords are indexed by the relative performance indicated by their performance data, where index Kx1 is the highest performance keyword in each set. Figure 5A and Figure 5B An example of how to determine performance data for a keyword is described below.

[0078] Process 350 removes the best performing keyword from each set to form a training set (354). Figure 2A As shown, the deletion process 304 deletes the first N keywords from each set. For example, for N=3, the resulting training keyword set is [{K14...K1N}; {P14...P1N}], [{K24...K2M}; {P24...P2M}], ... [{KQ4...KQX}; PQ4...PQX}].

[0079] Although only one training set for each keyword set is shown in the figure, multiple training sets for each keyword set can be generated. For example, if N=4, the deletion process 304 can delete the true subsets of three keywords in the first four keywords until all combinations are exhausted, for example, [{K11, K15...KN}, {P11, P15...PN}], [{K12, K15...KN}, {P12, P15...PN}], [{K13, K15...KN}, {P13, P15...PN}], [{K14, K15...KN}, {P14, P15...PN}]. In addition, although the above example describes deleting three best performance keywords, fewer or more best performance keywords can be deleted to form a training set.

[0080] In some implementations, the deletion process 304 considers the performance data on which the model 170 is to be trained. Then, based on the performance data and for each keyword set, the deletion process 304 selects a proper subset of keywords based on the associated performance data of the keywords. The keywords in the proper subset of keywords are the best performing keywords in the set, i.e., keywords that exceed the performance of other keywords in the keyword set that are not in the proper subset of keywords. Then, the deletion process 304 deletes the keywords in the proper subset of keywords from the keyword set to generate a training set of keywords. The training set of keywords is the set of keywords without the deleted keywords.

[0081] The process 350 then trains the model 170 on the set of keywords to predict the deleted keywords from the set for each set (356). During training, the model 170 is provided with the keyword training set as input and generates predicted high-performance keywords. The training process adjusts the model weights and parameters based on the difference between the predicted high-performance keywords of the input set of training keywords and the deleted keywords in the training set. The difference can be determined by a loss function that processes the predicted keywords, the deleted keywords, the predicted performance, and optionally the embedding of the input keywords. This process is repeated until the model 170 reaches a performance level that is considered satisfactory. Any suitable training process that implements the above training can be used.

[0082] After the model is trained 170 , it is persistent and available for use.

[0083] Figure 4 is a flow chart of a process 400 for predicting high performance keywords from an input set of keywords using a language model. The process 400 may be implemented in a computer system having one or more computers and storage devices in data communication with each other.

[0084] Process 400 provides a set of keywords as input to a trained language model (402). For example, as shown in FIG. Figure 2A and Figure 2B As depicted, a keyword set 202 may be provided as input.

[0085] Process 400 predicts a set of keywords from an input and presents the keywords to a user (404). Figure 2A and Figure 2B As described, a high performing keyword set 204 is predicted and sent to a user device for display.

[0086] For each keyword accepted by the user, process 400 associates the keyword with a target entity for use in a selection process (404) of selecting one or more digital components for the target entity. For example, the target entity may be a set of digital components, such as advertisements, which are then associated with one or more of the predicted keywords. The keywords are then used to provide the digital components in response to the query or for placement on a web page.

[0087] Figure 5A and Figure 5B is a flow chart of an example process for determining a set of keywords for training a language model. Process 500 may be implemented in a computer system having one or more computers and storage devices in data communication with each other.

[0088] Process 500 is for selecting keywords from a set of keywords for a digital-oriented component to be included in a training set of keywords for training a language model.

[0089] Process 500 selects a keyword (502). The selected keyword may be a keyword that is already associated with one or more digital components.

[0090] Process 500 determines matching queries that match the keyword based on the selected keyword (502). The matching queries can be based on a semantic similarity process, historical performance, a stem matching process, and other matching techniques.

[0091] Process 500 determines a performance metric for a keyword (502). The performance metric is based on performance of matching a query. The performance can be based on click-through rate, based on conversion rate, based on impressions, or based on some other performance. The performance can be an average of the performance values ​​for the keyword, or some other central tendency or statistical data.

[0092] Process 500 determines whether the performance metric meets a performance threshold (508). The performance threshold may be a reference value selected by a system administrator, or may be an adjusted reference value.

[0093] If the process 500 determines that the performance metric meets the performance threshold, the process 500 adds the keyword to the training set (510). For example, the keyword can be added to Figure 3AA set in set 302 of .

[0094] Conversely, if the process 500 determines that the performance metric does not meet the performance threshold, then the process 500 does not add the keyword to the training set, and then the process 500 (510).

[0095] In some implementations, the keywords in the training set 302 are keywords that meet a threshold performance metric from a matching query that is not primarily determined from a landing page or other digital component associated with the keyword. This enables the language model to be trained on keywords that are not directly derived from the content of the landing page or digital component. Figure 5B One such process for selecting such keywords is process 520 of . Process 520 may be implemented in a computer system having one or more computers and storage devices in data communication with each other.

[0096] Process 520 determines a performance metric for each query matched by the keyword (522). The performance may be based on click-through rate, based on conversion rate, based on impressions, or based on some other performance. The performance metric may be stored in a log of device 110 or determined based on service and interaction data stored in the log.

[0097] Process 520 determines a landing page match metric for each query matched by the keyword (524). The landing page match metric is a measure of how well the user query "matches" the landing page, e.g., the landing page is semantically related to the matching query used to navigate to the landing page, or a term of the form (root, synonym, etc.) is found in the landing page, etc.

[0098] Process 520 determines a performance metric for the keyword based on the performance metric for each query matched by the keyword and the landing page match metric (526). In some implementations, the performance metric for the keyword is proportional to the performance metric for the matched query and inversely proportional to the landing page match metric for each query matched to the keyword, e.g., P KWD = f(P({Q}) / f(LPM({Q}), where f( ) is a function that generates a performance metric or a landing page match metric based on the constituent performance metrics of the matching query {Q} and the landing page match metric.

[0099] In other implementations, the landing page match metric may be a binary value, where each query is determined to "match" or "not match" the landing page. Queries determined to match have a value of 0, and queries determined to not match have a value of 1. A minimum number of N queries that do not match the landing page may be required before evaluating whether the keyword is included in the training set, e.g., N=10. If the keyword is evaluated, the performance metric is based on the performance metric for the unmatched queries.

[0100] Other methods of determining performance metrics for keywords may also be used.

[0101] In general, keywords are more suitable for training when a small number of queries with high performance metrics match the landing page. This is because some keywords match queries that perform well for the landing page but cannot be derived from the landing page. Since such keywords are difficult to determine in advance, log-based analysis can be used to determine the set of training keywords that meet these criteria for training the LLM.

[0102] Figure 6 6 is a block diagram of an example computer system 600 that can be used to perform the operations described above. System 600 includes a processor 610, a memory 620, a storage device 630, and an input / output device 640. Each of components 610, 620, 630, and 640 can be interconnected, for example, using a system bus 650. Processor 610 is capable of processing instructions for execution within system 600. In one implementation, processor 610 is a single-threaded processor. In another implementation, processor 610 is a multi-threaded processor. Processor 610 is capable of processing instructions stored in memory 620 or on storage device 630.

[0103] The memory 620 stores information within the system 600. In one implementation, the memory 620 is a computer readable medium. In one implementation, the memory 620 is a volatile memory unit. In another implementation, the memory 620 is a non-volatile memory unit.

[0104] The storage device 630 can provide mass storage for the system 600. In one implementation, the storage device 630 is a computer-readable medium. In various implementations, the storage device 630 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices over a network (e.g., a cloud storage device), or some other mass storage device.

[0105] The input / output device 640 provides input / output operations for the system 600. In one implementation, the input / output device 640 may include one or more of a network interface device (e.g., an Ethernet card), a serial communication device (e.g., an RS-232 port), and / or a wireless interface device (e.g., an 802.11 card). In another implementation, the input / output device may include a driver device configured to receive input data and send output data to other devices (e.g., a keyboard, a printer, a display, and other peripheral devices 660). However, other implementations may also be used, such as a mobile computing device, a mobile communication device, a set-top TV client device, etc.

[0106] Although in Figure 6 An example processing system is described in the specification, but the subject matter and implementation of the functional operations described in this specification may be implemented in other types of digital electronic circuit systems or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them.

[0107] An electronic document (referred to simply as a document for brevity) need not correspond to a file. A document may be stored in a portion of a file that holds other documents, in a single file dedicated to the document in question, or in multiple coordinated files.

[0108] With respect to situations in which the systems discussed herein collect and / or use personal information about a user, the user may be provided with the opportunity to enable / disable or otherwise control programs or features that may collect and / or use personal information (e.g., information about the user's social network, social actions or activities, the user's preferences, or the user's current location). Additionally, certain data may be disposed of in one or more ways prior to its storage or use such that personally identifiable information associated with the user is deleted. For example, the user's identity may be anonymized such that personally identifiable information about the user cannot be determined, or the user's geographic location may be generalized (such as to a city, zip code, or state level) if location information is available such that the user's specific location cannot be determined.

[0109] The subject matter and the embodiments of the operation described in this specification may be implemented in a digital electronic circuit system or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, that is, one or more modules of computer program instructions, which are encoded on a computer storage medium for data processing equipment to execute or control the operation of the data processing equipment. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, for example, a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for data processing equipment to execute. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them, or included in them. In addition, although the computer storage medium is not a propagation signal, the computer storage medium may be the source or destination of the computer program instructions encoded in the artificially generated propagation signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

[0110] The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0111] The term "data processing equipment" includes all types of equipment, devices and machines for processing data, for example, including programmable processors, computers, systems on chip, or a plurality of or a combination of the foregoing items. The equipment may include a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment may also include code for creating an execution environment for the computer program involved, for example, code constituting a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The equipment and execution environment may implement various different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0112] This document relates to service devices. As used herein, a service device is one or more data processing devices that perform operations to facilitate the distribution of content over a network. A service device is depicted as a single block in a block diagram. However, although a service device can be a single device or a single group of devices, the present disclosure contemplates that a service device can also be a group of devices, or even a plurality of different systems that communicate to provide various content to a client device. For example, a service device can encompass one or more of a search system, a video streaming service, an audio streaming service, an email service, a navigation service, an advertising service, a gaming service, or any other service.

[0113] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0114] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit system, and the device can also be implemented as a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0115] For example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to receive data from it or transfer data to it or both. However, a computer does not have to have such devices. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0116] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0117] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), interconnected networks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0118] A computing system may include a client and a server. The client and server are usually remote from each other and typically interact through a communication network. The relationship of client and server arises from computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML web page) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., the result of a user interaction) may be received from the client device at the server.

[0119] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but should be interpreted as descriptions of features that are peculiar to a particular embodiment of a particular invention. Certain features described in the context of a separate embodiment in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. In addition, although features may be described above as acting in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may involve a sub-combination or a variation of the sub-combination.

[0120] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0121] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions set forth in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method, the method comprising: Obtaining a set of keywords, each keyword set being used in a selection process of one or more digital components, and each keyword in the keywords having associated performance data, the performance data indicating a performance of the keyword in the selection process of one or more digital components; For each keyword set: selecting a proper subset of keywords based on the associated performance data of the keywords, wherein the keywords in the proper subset of keywords exceed the performance of the keywords in a set of keywords that are not in the proper subset of keywords; as well as Deleting the keywords in the proper subset of the keywords from the keyword set to generate a training set of keywords, wherein the training set of keywords is the keyword set without the deleted keywords; training a language model to generate a trained language model, wherein the trained language model predicts the deleted keyword from the training set for each keyword; Processing a word set related to a target entity through the trained language model to predict a keyword recommendation set of keywords; providing the keyword recommendation set to an accepting process; as well as For each keyword in the keyword recommendation set that is accepted by the acceptance process, the keyword is associated with the target entity for use in the selection process of selecting one or more digital components for the target entity.

2. A computer-implemented method as described in claim 1, wherein the associated performance data indicating the performance of the keyword includes click data, the click data specifying a click performance indicator for each query matched by the keyword and a landing page match metric for each query matched by the keyword.

3. The computer-implemented method of claim 2, wherein the performance of a keyword is directly proportional to a performance metric of matching queries that match the keyword and inversely proportional to a landing page matching metric of the matching queries that match the keyword.

4. The computer-implemented method of claim 1 , wherein processing the set of words associated with a target entity through the trained language model comprises: A set of keywords related to the target entity used in the selection process of one or more digital components is processed by the trained language model to predict a keyword recommendation set of keywords.

5. The computer-implemented method of claim 4, wherein: The target entity is a web page; and The word set related to the target entity is keywords derived from the web page.

6. The computer-implemented method of claim 5, wherein: The target entity is a digital component that is used for presentation via a web page; and The word set associated with the target entity is a keyword determined for the digital component.

7. The computer-implemented method of claim 1 , wherein removing keywords from the proper subset of the keywords to generate a training set of keywords from the set of keywords: A proper subset of keywords is deleted from the proper subset of keywords.

8. The computer-implemented method of claim 1 , wherein removing keywords from the proper subset of the keywords to generate a training set of keywords from the set of keywords: All of the keywords are deleted from the proper subset of the keywords.

9. The computer-implemented method of claim 1, wherein the language model is one of a multi-task unified model, a zero-shot model, a domain-specific model, or a language representation model.

10. A system, comprising: One or more computers for data communications; as well as One or more non-transitory computer-readable media storing instructions executable by the one or more computers, the instructions, when executed by the one or more computers, causing the one or more computers to perform operations comprising: Obtaining a set of keywords, each keyword set being used in a selection process of one or more digital components, and each keyword in the keywords having associated performance data, the performance data indicating a performance of the keyword in the selection process of one or more digital components; For each keyword set: selecting a proper subset of keywords based on the associated performance data of the keywords, wherein the keywords in the proper subset of keywords exceed the performance of the keywords in a set of keywords that are not in the proper subset of keywords; and Deleting the keywords in the proper subset of the keywords from the keyword set to generate a training set of keywords, wherein the training set of keywords is the keyword set without the deleted keywords; training a language model to generate a trained language model, wherein the trained language model predicts the deleted keyword from the training set for each keyword; Processing a word set related to a target entity through the trained language model to predict a keyword recommendation set of keywords; providing the keyword recommendation set to a receiving process; and For each keyword in the keyword recommendation set that is accepted by the acceptance process, the keyword is associated with the target entity for use in the selection process of selecting one or more digital components for the target entity.

11. The system of claim 10, wherein the associated performance data indicative of the performance of the keyword includes click data specifying a click performance metric for each query matched by the keyword and a landing page match metric for each query matched by the keyword.

12. The system of claim 11, wherein the performance of a keyword is directly proportional to a performance metric of matching queries matching the keyword and inversely proportional to a landing page matching metric of the matching queries matching the keyword.

13. The system of claim 10, wherein processing the set of characters associated with a target entity through the trained language model comprises: A set of keywords related to the target entity used in the selection process of one or more digital components is processed by the trained language model to predict a keyword recommendation set of keywords.

14. The system of claim 13, wherein: The target entity is a web page; and The word set related to the target entity is keywords derived from the web page.

15. The system of claim 13, wherein: The target entity is a digital component that is used for presentation via a web page; and The word set associated with the target entity is a keyword determined for the digital component.

16. The system of claim 10, wherein removing keywords from the proper subset of the keywords to generate a training set of keywords from the set of keywords: A proper subset of keywords is deleted from the proper subset of keywords.

17. The system of claim 10, wherein removing keywords from the proper subset of the keywords to generate a training set of keywords from the set of keywords: All of the keywords are deleted from the proper subset of the keywords.

18. One or more non-transitory computer-readable media storing instructions that, when executed by an artificial intelligence system, cause the artificial intelligence system to perform operations comprising: Obtaining a set of keywords, each keyword set being used in a selection process of one or more digital components, and each keyword in the keywords having associated performance data, the performance data indicating a performance of the keyword in the selection process of one or more digital components; For each keyword set: selecting a proper subset of keywords based on the associated performance data of the keywords, wherein the keywords in the proper subset of keywords exceed the performance of the keywords in a set of keywords that are not in the proper subset of keywords; as well as Deleting the keywords in the proper subset of the keywords from the keyword set to generate a training set of keywords, wherein the training set of keywords is the keyword set without the deleted keywords; training a language model to generate a trained language model, wherein the trained language model predicts the deleted keyword from the training set for each keyword; Processing a word set related to a target entity through the trained language model to predict a keyword recommendation set of keywords; providing the keyword recommendation set to an accepting process; as well as For each keyword in the keyword recommendation set accepted by the acceptance process, the keyword is associated with the target entity for use in the selection process of selecting one or more digital components for the target entity.

19. The non-transitory computer-readable medium of claim 18, wherein the associated performance data indicative of the performance of the keyword comprises click data specifying a click performance metric for each query matched by the keyword and a landing page match metric for each query matched by the keyword.

20. The non-transitory computer-readable medium of claim 18, wherein the performance of a keyword is directly proportional to a performance metric of matching queries that match the keyword and inversely proportional to a landing page matching metric of the matching queries that match the keyword.

Images