Training neural networks to predict action values based on non-user-specific feature data

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network on a training dataset. In one example, the training dataset includes training inputs in a first domain that are generated from log data of user activities on a service platform collected during a signed-in mode for users signed into the service platform. The training dataset also includes training inputs in a second domain that are generated from log data of user activities on the service platform collected during a signed-out mode for users that are not signed into the service platform.

Description

TRAINING NEURAL NETWORKS TO PREDICT ACTION VALUES BASED ON NON-USER-SPECIFIC FEATURE DATABACKGROUND

[0001] This specification relates to training machine learning models, e.g., neural networks.

[0002] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.SUMMARY

[0003] This specification relates to techniques for training neural networks and using the trained neural networks to select content, e g., digital components, to distribute to users based on user interactions with a service platform, e.g., an Internet-based service platform.

[0004] Users connected to the Internet are exposed to a variety of digital content (e.g., search results, web pages, digital components, news articles, social media posts, audio information output by a digital assistant device). Some of these exposures to content may contribute to the users performing a target action. For example, a user that is exposed to a web page about an endangered species may sign up for a newsletter directed to helping save that endangered species, where signing up for the newsletter can be considered the target action. Similarly, a user that is exposed to a digital component in a webpage about a particular type of mobile device can ultimately acquire that particular type of mobile device, where acquisition of the mobile device can be considered the target action. Examples of target actions can also include registering with a website / service, adding items to an online cart, downloading a whitepaper, acquiring a product or even clicking (or otherwise interacting with) a digital component, or more generally any user action linked to the digital content or part of it. When a user performs a target action, performance of the target action can be referred to as a conversion.

[0005] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of obtaining, from a training dataset, a batchof training inputs including training inputs in a first domain that are generated from log data of user activities on a service platform collected during a signed-in mode for users signed into the service platform, and training inputs in a second domain that are generated from log data of user activities on the service platform collected during a signed-out mode for users that are not signed into the service platform; for each training input in the batch: processing, by a backbone of the neural network, the training input to generate a feature embedding; processing, by one or more action value prediction heads of the neural network, the feature embedding to generate one or more predicted action values for a target action; and processing, by a domain classifier head of the neural network, the feature embedding to generate a predicted domain score that represents a likelihood that the training input is in the first domain; determining a loss function that includes at least (i) one or more action value loss terms that measure, for each training input in the batch, a difference between the one or more predicted action values and one or more ground truth action values associated with the training input and (ii) a domain score loss term that measures, for each training input in the batch, a difference between the predicted domain score and a ground truth domain score of the training input; an determining one or more updates to current values of parameters of the neural network based on optimizing the loss function. Other implementations of this aspect include corresponding apparatus, systems, and computer programs, configured to perform the aspects of the methods, encoded on computer storage devices.

[0006] These and other embodiments can each optionally include one or more of the following features. In some aspects, determining the one or more updates to current values of parameters of the neural network includes computing gradients of the loss function with respect to parameters of the domain classifier head; multiplying the gradients by a negative constant to determine reversed gradients; and computing, based on the reversed gradients and by backpropagation, gradients of the loss function with respect to parameters of the backbone.

[0007] In some aspects, the neural network includes a gradient reversal layer arranged between the backbone of the neural network and the domain classifier head.

[0008] In some aspects, a training input in the first domain includes at least one feature that is not included in a training input in the second domain.

[0009] In some aspects, the at least one feature is generated from a portion of log data of user actions on the service platform that is only collected during the signed-in mode and not during the signed-out mode.

[0010] In some aspects, the target action includes a user click on a digital component presented to the user.

[0011] In some aspects, the one or more predicted action values include a value that indicates a likelihood of the target action being performed.

[0012] In some aspects, the one or more predicted action values includes a value that indicates a time length of the target action being performed.

[0013] In general, another innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of obtaining a batch of training inputs generated from log data characterizing actions of users on a service platform, each training input in the batch including (i) user-specific features that are generated from a portion of the log data that is only collected during a signed-in mode and (ii) non-user-specific features that are generated from another portion of the log data that is collected during both the signed-in mode and a signed-out mode; for each training input included in the batch: processing, by a first tower of the neural network, the user-specific features included in the training input to generate a user feature embedding; and processing, by a second tower of the neural network, the non-user- specific features included in the training input to generate a surrogate user feature embedding; and determining one or more updates to parameter values of the second tower of the neural network based on minimizing a distance between the user feature embeddings and the surrogate user feature embeddings. Other implementations of this aspect include corresponding apparatus, systems, and computer programs, configured to perform the aspects of the methods, encoded on computer storage devices.

[0014] Some aspects include, for each training input included in the batch: processing, by one or more action value prediction heads of the neural network, the user feature embedding included in the training input to generate one or more predicted action values and determining one or more updates to parameter values of the prediction head and to parameter values of the first tower based on minimizing a difference between the one or more predicted action values and one or more ground truth action values associated with the training input.

[0015] In some aspects, the action is a search action that uses a user query, and the non-user- specific features include features related to the search action.

[0016] In general, another innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of receiving session data that describes actions performed by a signed-out user when interacting with a service platform processing the session data using the neural network trained using the respective method of any preceding claim to generate one or more predicted action values for a target action that can be performed by the signed output user on the service platform; and determining whether to provide a digital component for presentation to the user based on the one or more predicted action values for the target action. Other implementations of this aspect include corresponding apparatus, systems, and computer programs, configured to perform the aspects of the methods, encoded on computer storage devices.

[0017] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. The specification describes techniques for training one or more neural networks on training datasets generated from session data that was generated as a result of user interactions with a service platform. The session data can be collected during both a signed-in mode and a signed-out mode. In the signed- out mode, no user credentials (e.g., user identifier or password) are provided by a user to the service platform, and hence less information, and in particular user-specific information, is made available to the service platform.

[0018] By incorporating the domain classification training to improve the training of a backbone of a neural network as described in this specification, the neural network can effectively and accurately predict action values for a target action that can be performed by a user during a user session. Additionally, a neural network can choose between two separate towers, including a first tower trained on user-specific training data and a second tower trained on non-user-specific training data, to generate user feature embeddings that encode enough information, even for user sessions in the signed-out mode where less information is available, allowing the neural network to nonetheless make an accurate prediction of the action values for the target action even in the absence of certain user-specific information. Thus, the techniques described in this document provide for similar accuracy in predictions even when less information is available to make the predictions. Additionally, by having a domain-basedtraining approach and relying on user-specific training data, e.g., data about the demographics of the users, the ages of the users, etc., and non-user specific data, e.g., session data, the techniques described here improve the efficiency of the training process. Thus, the described techniques provide a particular way to use artificial intelligence to supplement information to generate accurate results when more robust information is not available, e.g., when a user is not signed in to the service platform.

[0019] The increased accuracy in the predicted action values makes possible a reduction in computing resource consumption and / or power consumption when selecting digital components, displaying the selected digital components, or both. For example, a service apparatus can refrain from presenting a larger number of digital components throughout a user session, and instead present a relatively smaller number of digital components only when the predicted action values satisfy, e.g., meets or exceeds, some predetermined threshold values. The savings in computing resource and / or power consumption are especially important for edge devices, e.g., mobile phones, tablet computers, and smartwatches, on which the digital components are displayed, because the savings in turn improve the operational time of the edge devices and the duration between battery charges.

[0020] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG. l is a block diagram illustrating an example environment that includes a service platform.

[0022] FIG. 2 is a block diagram of an example training system for training a neural network.

[0023] FIG. 3 is a flow diagram of an example process for training a neural network.

[0024] FIG. 4 is a flow diagram of sub-operations of one of the operations of the process ofFIG. 3.

[0025] FIG. 5 is a block diagram of another example training system for training a neural network.

[0026] FIG. 6 is a flow diagram of another example process for training a neural network.

[0027] FIG. 7 is a flow diagram of an example process for using a neural network to perform inference.

[0028] FIG. 8 shows a block diagram of an example computer.

[0029] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0030] FIG. 1 is a block diagram illustrating an example environment 100 that includes client devices 105A-105N, a network 115, a service platform 120, and data stores 122, 124. In some implementations, network 115 can include a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. In some implementations, the data store 122, 124 can be a memory (e.g., random access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or another type of component or device capable of storing data.

[0031] The data stores 122, 124 can each include multiple storage components, e.g., multiple drives or multiple databases, that can span multiple computing devices, e.g., multiple server computers. In some implementations, the data store 122 is independent from the data store 124. For example, they can be implemented on distinct sets of computing devices, potentially at distinct geographical locations. In some implementations, the data stores 122, 124 are implemented on the same computing devices, e.g., but in different data structures.

[0032] The client devices 105A-105N can each include computing devices such as personal computers, gaming devices, mobile communication devices, digital assistant devices, augmented reality devices, virtual reality devices, and other devices that can send and receive data over the network 115. Each client device includes at least one application 110, such as a web browser, to facilitate the sending and receiving of data over the network 115, but native applications (other than browsers) executed by the client device can also facilitate the sending and receiving of data over the network 115. While only client devices 105A-105N are shown in FIG. 1, the environment 100 can support a large number of concurrent sessions with many client devices 105.

[0033] A gaming device is a device that enables a user to engage in gaming applications, for example, in which the user has control over one or more characters, avatars, or other rendered content presented in the gaming application. A gaming device typically includes a computer processor, a memory device, and a controller interface (either physical or visually rendered) thatenables user control over content rendered by the gaming application. The gaming device can store and execute the gaming application locally, or execute a gaming application that is at least partly stored and / or served by a cloud server (e.g., online gaming applications). Similarly, the gaming device can interface with a gaming server that executes the gaming application and “streams” the gaming application to the gaming device. The gaming device may be a tablet device, mobile telecommunications device, a computer, or another device that performs other functions beyond executing the gaming application.

[0034] Digital assistant devices include devices that include a microphone and a speaker. Digital assistant devices are generally capable of receiving input by way of voice, and respond with content using audible feedback, and can present other audible information. In some situations, digital assistant devices also include a visual display or are in communication with a visual display (e.g., by way of a wireless or wired connection). Feedback or other information can also be provided visually when a visual display is present. In some situations, digital assistant devices can also control other devices, such as lights, locks, cameras, climate control devices, alarm systems, and other devices that are registered with the digital assistant device.

[0035] The service platform 120 can include one or more computing devices, such as a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc., data stores (e.g., hard disks, memories, databases), networks, software components, and / or hardware components. These components can be implemented together in a single device or networked in various combinations of multiple different devices that operate together to support the functionalities of the service platform 120.

[0036] In some examples, the service platform 120 can be an online intermediation service platform, a search engine platform, a social networking service platform, a content-sharing service platform, a number-independent interpersonal communications service platform, an operating system platform, a web browser platform, a virtual assistant platform, a cloud computing service platform, an online digital component distribution platform, or another Internet-based service platform. The service platform 120 can also be an internal or private service platform, e.g., an internal content-sharing platform of an organization.

[0037] The service platform 120 can interact with the client devices 105A-105N, such as using the application 110, to communicate data between the service platform 120 and users ofthe service platform 120. For example, the service platform 120 can be a search engine platform that identifies resources, e.g., web pages, images, text documents, multimedia content, e.g., videos, that are relevant to a user’s information needs and to present information about the resources in a manner that is most useful to the user. In some implementations, the service platform 120 is a multi-service platform that provides multiple services, e.g., search engine services as well as content-sharing services. A multi-service platform can provide any number of services, e.g., any combination of the services described herein.

[0038] In examples in which the service platform 120 is a search engine platform, users interact with the search engine platform through the use of one or more of the client devices 105A-105N. For example, the client device 105A executes an application 110, such as a web browser, or another standalone application (e.g., a mobile application or mobile app), which provides a search field that allows a user to formulate search queries and submit them to the search engine platform.

[0039] The search engine platform receives a submitted query from the client device 105 A, and executes the query against a content database (e.g., index) of available documents such as web pages, images, text documents, and multimedia content. The search engine platform identifies content in the content database which matches or is otherwise responsive to the query, and responds by generating search results which are transmitted to the client device 105 A in a form that can be presented to the user. For example, in response to a query from the client device 105 A, the search engine platform can transmit a search results web page to be displayed in the web browser executing on the client device 105 A.

[0040] As another example, the service platform 120 can be a virtual assistant platform (also known as “automated assistant platform” or “mobile assistant platform”) that can be interacted with by users through an application installed on each of one or more of the client devices 105A- 105N. During operation, the virtual assistant platform receives input from the user (e.g., typed or spoken natural language input) and responds with responsive content (e.g., visual and / or audible natural language output). The virtual assistant platform can provide a broad range of functionalities through interactions with various local and / or third-party applications, websites, or other platforms.

[0041] As another example, the service platform 120 can be a content-sharing service platform that provides digital content, e.g., digital video, streaming video, images, digital music,streaming music, social media information, etc., to one or more of the client devices 105A-105N and can allow users to share content among multiple users. To provide digital content, for example, the content-sharing service platform can allow a user to subscribe to a paid or free channel through an application 110 installed on the client device 105 A. The content- shari ng service platform can present different types of access or different subscriptions levels to the user. The content-sharing service platform can also allow the user to consume, upload, search for, curate, share, approve of (e.g., “like”), dislike, and / or comment on content items of various channels.

[0042] In these examples and many other examples, the service platform 120 can interact with a user that is using an application 110 installed on each of the client device 105A-105N that is either in a signed-in mode for the user, or alternatively in a signed-out mode for the user.

[0043] In some cases, the user has credentials, e g., a username and password, associated with a user account maintained by the service platform 120. If the user wants to sign in to the user account on the client device 105 A, then the user can open the application 110 installed on the client device 105 A and enters the credentials, e.g., in a user interface of the application. Alternatively, if the user wants to access the service platform 120 in the signed-out mode, then the user enters no such credentials, or the user explicitly indicates, e.g., by selecting a sign out button, in the user interface of the application 110. The service platform 120 can also allow guest users that have not registered with the service platform 120 to interact with the service platform 120 in the signed-out mode. Generally, the signed-in mode refers to a mode in which the user has provided credentials and the service platform 120 has verified the credentials as belonging to a particular user. In contrast, the signed-out mode refers to a mode in which the user has not provided credentials such that the service platform 120 does not know what user is interacting with the service platform 120.

[0044] In a signed-in mode, credentials of the user can be utilized by a client device to enable the service platform 120 to at least selectively provide various services that are specific to the user, or selectively access various data that is specific to the user, or both. Utilizing the service platform 120 in a signed-in mode provides various technical benefits, such as generating responses that are more relevant to the queries and to the user’s informational needs, in a manner that is more readily understood by the user, and more quickly.

[0045] However, not all users have user accounts with the service platform 120. Nor does a user that has a user account choose to enter the credentials every time they interact with the service platform 120. Moreover, data security concerns could arise when a user operates in signed-in mode for a client device that is only being transiently utilized by the user, e.g., only a limited quantity of interactions and / or for a limited period of time. Generally, functionalities of the service platform 120 may be limited in the signed-out mode, various benefits of a signed-in mode may be unavailable in the signed-out mode, or both.

[0046] The service platform 120 has access to the data stores 122, 124 for storing log data generated, e.g., collected, during interaction between the users and the service platform 120. The log data includes session data of user sessions. Each user session can include one or more actions performed by the user at the client device during the session. In some cases, a user session is defined using a period of time, e.g., 10 minutes, 30 minutes, or 60 minutes, for sequential actions performed by the user. In some cases, a user session is defined to begin from the time a user provides their credentials and end at the time the user logs out or otherwise becomes inactive on the service platform 120. In some cases, a user session can start when the user opens the application 110 and end when the user closes the application 110. If the application is a web browser and the service platform 120 hosts a website, the user session can start when the user navigates to the website and end when the user navigates away from the website or closes the browser. The session data for each user session can be stored in either one of the data stores 122, 124 using any suitable data structure. For example, the log data can be stored using database, index, list, linked list, table, or any combination thereof.

[0047] Depending on whether a user interacts with the service platform 120 in a signed-in mode or in a signed-out mode, however, different types and / or different amounts of session data are stored for that user session. In FIG. 1, the data stores 122, 124 correspond respectively to these two different modes of operation of the service platform 120. In particular, the session data of user sessions in a signed-in mode is stored in the data store 122, and session data of user sessions in a signed-out mode is stored in the data store 124.

[0048] Generally, a narrower range of types, a smaller amount, or both of session data can be stored for a user session that is in the signed-out mode, than another user session that lasts (approximately) the same time period of time, but is in the signed-in mode. This is because lessinformation, e.g., less information that is specific to the user, is made available to service platform 120 when the user interacts with the service platform 120 in the signed-out mode.

[0049] When operating in a signed-in mode, the session data for each user session can additionally include a username, or another unique identifier, of a user associated with the user session. The session data for each session can further include user profile data for the user that can be determined from the username. For example, the user profile data can include information about the user that the user provided when the user registered with the service platform 120. The user profile information can include, for example, user attributes, e.g., demographic information, of the user, the geographic location of the user, the age of the user, the spoken language of the user, etc. When operating in a signed-out mode, no username or profile information is included in the session data.

[0050] For example, when the service platform 120 is a search engine platform, the session data stored in either data store 122 or data store 124 for each user session can include one or more search queries, one or more search results responsive to the queries, one or more digital components provided for presentation to a user, one or more actions, e.g., a selection, browse, copy, or print action, performed by the user responsive to the search results, including one or more target actions performed by the user on the digital components, and, in some cases, metadata, e.g., geographic location and timestamp metadata, associated with the search queries and / or the search results. In this example, the session data stored in data store 122 will differ from the session data stored in data store 124 in that the session data stored in data store 122 additionally includes at least a username of a user who submitted the search queries that is not included in the session data stored in data store 124. The session data stored in data store 122 can further include user-specific data, e.g., the demographics of the user (who is identified by the username), the age of the user, etc., both of which is not included in the session data stored in data store 124. For signed-in users, the session data can also include data that spans multiple user sessions, e.g., a count of the number of times a query has been submitted by the user over multiple sessions of that user. However, multi-session data would not be available for a signed- out user.

[0051] As another example, when the service platform 120 is a content-sharing service platform, the session data stored in either data store 122 or data store 124 for each user session can include data defining what content items were viewed by a user during the user session, oneor more digital components provided for presentation to the user, and one or more target actions performed by the user on the digital components. In this example, the session data stored in data store 122 can differ from the session data stored in data store 124 in that the session data stored in data store 122 additionally includes at least a username of a user who viewed the content items that is not included in the session data stored in data store 124. The session data stored in data store 122 can further include user-specific data, e.g., data defining content items that have marked by the user (who is identified by the username) for later consumption, channels that the user has subscribed to, content items that the user has liked, topics the user is interested in, etc., all of which is not included in the session data stored in data store 124.

[0052] As another example, when the service platform 120 is an online intermediation service platform, the session data stored in either data store 122 or data store 124 for each user session can include data defining what products (e g., goods or service) were viewed, selected to add to a wish list, selected to add to a cart, purchased by a user during the user session, one or more digital components provided for presentation to the user, and one or more target actions performed by the user on the digital components. In this example, the session data stored in data store 122 can differ from the session data stored in data store 124 in that the session data stored in data store 122 additionally includes at least a username of a user who viewed the products that is not included in the session data stored in data store 124. The session data stored in data store 122 can further include user-specific data, e.g., payment information and contact information entered by the user during a session, which is not included in the session data stored in data store 124.

[0053] When configured as any one of the platforms mentioned above or other service platforms, the service platform 120 includes a service apparatus 130. The service apparatus 130 chooses digital components that will be presented to the client devices 105A-105N while they are interacting with the service platform 120. For example, when configured as a search engine platform, the service platform 120 can integrate a digital component into a search results web page to be displayed in a web browser executing on a client device. As another example, when configured as a content-sharing service platform, the service platform 120 can integrate a digital component into, or use the digital component as, the content item to be displayed in an application executing on the client device.

[0054] As illustrated above, session data can include various types of data that is observed during a user session, including actions of the user such as content requested by the user, content viewed by the user, content interacted with by the user, etc. Additionally, the types of session data can vary based on the type of content or type of service(s) provided by the service platform 120.

[0055] As used throughout this document, the phrase “digital component” refers to a discrete unit of digital content or digital information (e.g., a video clip, audio clip, multimedia clip, gaming content, image, text, bullet point, artificial intelligence output, language model output, or another unit of content). A digital component can electronically be stored in a physical memory device as a single file or in a collection of files, and digital components can take the form of video files, audio files, multimedia files, image files, or text files and include advertising information, such that an advertisement is a type of digital component.

[0056] In some implementations, the service apparatus 130 identifies one or more digital components 152 that are eligible to be presented on the client devices from among a corpus of thousands or millions of available digital components. The thousands or millions of available digital components can be indexed in a digital component database 150. Each digital component index entry can reference the corresponding digital component and / or include distribution parameters that contribute to (e.g., trigger, condition, or limit) the distribution / transmission of the corresponding digital component.

[0057] In particular, the service apparatus 130 uses an artificial intelligence (“Al”) system 140 to facilitate the selection of the one or more digital components 152 from the digital component database 150 that will be provided for presentation on the client devices 105A-105N. The Al system 140 includes one or more neural networks, and determines whether a digital component from the digital component database 150 should be selected as the selected digital component 152, based on the outputs of the one or more neural networks included in the Al system 140. In turn, the service apparatus 130 can generate and transmit, over the network 115, data that enables a client device to present the selected digital component 152 at a display of the client device. This data can include the digital component itself or a link to a networked location from which the client device can download the digital component.

[0058] As described in more detail below, the Al system 140 can request, e.g., query, the one or more neural networks to generate one or more predicted action values for a target action, e.g.,a target action corresponding to a digital component that is available for presentation on any one of the client devices 105A-105N. The Al system 140 can use at least the predicted action values to determine whether to present a selected digital component 152 on the client device.

[0059] Each digital component can include one or more target actions. A target action can be an action performed with respect to the digital component (e.g., a selection or other interaction with the digital component) or an action performed by the user after viewing or interacting with the digital component. Such an action can include adding an item that is the subject of the digital component to a virtual shopping cart, purchasing the item, registering for a newsletter related to the subject of the digital component, downloading an application that is the subject of the application, downloading a security patch to a client device, etc.

[0060] FIG. 2 is a block diagram of an example training system 200 for training a neural network 240. Although described in terms of a neural network 240, the training system 200 can be used to train other types of machine learning models. The training system 200 is an example of a system implemented as computer programs on one or more computers in one or more locations that trains the neural network 240 on a training dataset 210 to determine trained values of the parameters of the neural network 240.

[0061] The training dataset 210 includes multiple first domain training inputs 212 and multiple second domain training inputs 214 that correspond to a signed-in mode and a signed-out mode, respectively. Each first domain training input 212 corresponds to one or more user sessions that occurred in the signed-in mode, and includes features generated from the session data of the user session(s) that occurred in the signed-in mode. In contrast, each second domain training inputs 214 corresponds to one or more user sessions that occurred in the signed-out mode, and includes features generated from the session data of the user session(s) that occurred in the signed-out mode.

[0062] The first domain training inputs 212 generally each include a larger number of features than the second domain training inputs 214. This makes sense, because a wider range of types, a greater amount, or both of session data can be store during a user session that is in the signed-in mode, than another user session that lasts, e.g., approximately, the same time period of time, but is in the signed-out mode.

[0063] The neural network 240 includes a neural network backbone 241 (or “backbone” for short). The backbone 241 is configured to extract features from an input, that is, configured toprocess a training input, e.g., a first domain training input 212 or a second domain training input 214, to generate a feature embedding 243 of the training input. An embedding refers to an ordered collection of numerical values, e.g., a vector, matrix, or other tensor of numerical values.

[0064] The neural network 240 includes one or more action value prediction heads 244A-N and a domain classifier head 246. The action value prediction heads 244A-N are configured to process the feature embedding 243 generated by the backbone 241 from the training input to generate predicted action values 254A-N, respectively, for the training input.

[0065] The predicted action values are typically defined with reference to a target action. Users connected to the Internet are exposed to a variety of digital content (e.g., search results, web pages, digital components, news articles, social media posts, audio information output by a digital assistant device). Some of these exposures to content may contribute to the users performing a target action. For example, a user that is exposed to a web page about an endangered species may sign up for a newsletter directed to helping save that endangered species, where signing up for the newsletter can be considered the target action. Similarly, a user that is exposed to a digital component in a webpage about a particular type of mobile device can ultimately acquire that particular type of mobile device, where acquisition of the mobile device can be considered the target action. Examples of target actions can also include registering with a website / service, adding items to an online cart, downloading a whitepaper, acquiring a product or even clicking (or otherwise interacting with) a digital component. When a user performs a target action, performance of the target action can be referred to as a conversion.

[0066] For example, the predicted action values 254A-N can include a value that indicates a likelihood of a target action being performed by the user during the user session, following the user session, and / or resulting from the user viewing the digital component. As another example, the predicted action values 254A-N can include a value that indicates a time duration of the target action being performed. As yet another example, the predicted action values 254A-N can include a predicted conversion rate for a digital component. In this example, the predicted action value can be a predicted likelihood of a user clicking or otherwise interacting with a digital component upon viewing the digital component or a predicted likelihood of the user completing another action, e.g., acquiring a product that is the subject of the digital component, after viewing or interacting with the digital component.

[0067] The domain classifier head 246 distinguishes between first domain training inputs 212 that correspond to user sessions in the signed-in mode from second domain training inputs 214 that correspond to user sessions in the signed-out mode. The domain classifier head 246 is configured to process the feature embedding 243 generated by the backbone 241 from the training input to generate a predicted domain score 256 that represents a likelihood that the training input is in the first domain (or, analogously, in the second domain).

[0068] A “backbone” or a “head” of a neural network refers to a group of one or more neural network layers in the neural network. The backbone 241 can be implemented with any appropriate neural network architecture that enables it to perform its described function. For example, the backbone 241 can include any appropriate types of neural network layers, e.g., embedding layers, fully connected layers, attention layers, and so forth, in any appropriate number, e.g., 10 layer, or 50 layers, or 100 layers, and connected in any appropriate configuration, e.g., as a directed graph of layers. As a particular example, the backbone 241 can have a wide and deep architecture, that is, can include a large number of layers, e.g., 50, 100, or more layers, each having a large number of units, e.g., 2048, 4096, or more units.

[0069] The action value prediction heads 244A-N and the domain classifier head 246 can each be implemented with any appropriate neural network architecture that enables them to perform their described functions. Generally, however, each head 244A-N, 246 has fewer layers than the backbone 241. For example, they can each include any appropriate types of neural network layers, e.g., fully connected layers, attention layers, activation layers, and so forth, in any appropriate number, e.g., 1 layer, or 5 layers, or 10 layers, and connected in any appropriate configuration, e.g., as a directed graph of layers.

[0070] The training system 200 performs the training over multiple training iterations. At each training iteration, the training system 200 updates the parameters of the neural network 240 by performing a forward pass through the neural network using the training inputs obtained from the training dataset 210 and then perform a backward pass through the neural network 240 to compute the respective gradients through backpropagation.

[0071] During training, the training system 200 trains the backbone 241 to generate semantically meaningful feature embeddings 243 of the training inputs. The training system 200 also trains the action value prediction heads 244A-N to generate more accurate predicted action values 254A-N, e g., to generate predicted action values that are close to the ground truth actionvalues associated with the training inputs. The training system 200 also trains the domain classifier head 246 to attempt to distinguish the first domain training inputs from the second domain training inputs, e.g., to generate higher predicted domain scores for first domain training inputs while generating lower predicted domain scores for second domain training inputs, e.g., when the predicted domain score represents a likelihood that the training input is in the first domain.

[0072] In this way, the training system 200 encourages the feature embeddings 243 generated by the backbone 241 to encode meaningful knowledge for training inputs from each domain. In particular, the training system 200 trains the backbone 241 to encode enough information in the feature embeddings 243 even for second domain training inputs 214, which generally include a smaller number of features than the first domain training inputs 212, for the action value prediction heads 244A-N to generate more accurate predicted action values 254A- N.

[0073] In some implementations, during training, the training system 200 maximizes a loss with respect to the parameters of the domain classifier head 246 to improve the accuracy of the domain classifier head 246 and minimizes the loss with respect to the parameters of the backbone 241 to decrease the accuracy of the domain classifier head 246. This maximizationminimization scheme can be realized by including a gradient reversal layer in the neural network 240 and minimizing the overall loss. In these implementations, the gradient reversal layer 242 can be inserted between the backbone 241 and the domain classifier head 246, as illustrated in FIG. 2. During the forward propagation, the gradient reversal layer 242 acts as an identity transform such that the output is same as the input. During the backpropagation, the gradient reversal layer 242 takes the gradient from the subsequent layer, multiplies it by a negative constant, and passes it to the preceding layer.

[0074] After the training, the training system 200 outputs at least a portion of the trained neural network 240 to the Al system 140 of FIG. 1 for deployment to perform inference. For example, the training system 200 outputs the backbone 241 and one or more of the action value prediction heads 244A-N to the Al system 140 for inclusion in a neural network 250 that is included in the Al system 140 to perform inference. An example way of deploying and using the trained neural network 240 to perform inference will be described in FIG. 7. 1

[0075] Optionally, the training system 200, the Al system 140, or a different system can further fine-tune some or all of the parameters of the neural network 240 before using it to perform inference. When included in the neural network 250, the parameters of the trained parameter values of the backbone 241 and the action value prediction heads 244 A-N may be frozen, or they may be further trained, e.g., on different training datasets, using a different loss function, or both, e.g., in conjunction with the parameters of the other layers of the neural network 250.

[0076] FIG. 3 is a flow diagram of an example process 300 for training a neural network. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 200 of FIG. 2, appropriately programmed in accordance with this specification, can perform the process 300. Operations of the process 300 can also be implemented as instructions stored on one or more computer-readable media, which may be non-transitory, and execution of the instructions by one or more data processing apparatus can cause the one or more data processing apparatus to perform the operations of the process 300.

[0077] The system can repeatedly perform iterations of the process 300 on different batches of training inputs to update the parameters of the neural network, i.e., of the backbone, the action value prediction heads, and the domain classifier head.

[0078] The system can continue performing iterations of the process 300 until termination criteria for the training of the neural network have been satisfied, e.g., until the parameters have converged, until a threshold amount of wall clock time has elapsed, or until a threshold number of iterations of the process 300 have been performed.

[0079] The system obtains a batch of training inputs from a training dataset (operation 302). The training dataset includes training inputs in a first domain that are generated from log data of user activities on a service platform collected during a signed-in mode for users signed into the service platform. Each first domain training input corresponds to one or more user sessions that occurred in the signed-in mode, and includes features generated from the session data of the user session(s) that occurred in the signed-in mode.

[0080] The training dataset also includes training inputs in a second domain that are generated from log data of user activities on the service platform collected during a signed-out mode for users that are not signed into the service platform. Each second domain training inputscorresponds to one or more user sessions that occurred in the signed-out mode, and includes features generated from the session data of the user session(s) that occurred in the signed-out mode. The first domain training inputs generally each include a larger number of features than the second domain training inputs. For example, a first domain training input can include one or more features, e.g., features that are determined from a username of a signed-in user when interacting with the service platform, that are not included in a second domain training input.

[0081] The system will generally obtain different training inputs at different iterations, e.g., by sampling a fixed number of inputs from a larger number of training inputs at each iteration. The batch of training inputs will generally include training inputs from both domains, e.g., include one or more training inputs in the first domain and one or more training inputs in the second domain. Each training input is associated with one or more ground truth action values for a target action that can be performed by a user during the corresponding user session(s), either in the signed-in mode or in the signed-out mode.

[0082] The system repeatedly performs following operations 304-308 for each training input included in the batch of training inputs obtained from the training dataset.

[0083] The system processes, by the backbone of the neural network, the training input to generate a feature embedding (operation 304).

[0084] The system processes, by the one or more action value prediction heads of the neural network, the feature embedding to generate one or more predicted action values for the target action (operation 306). Different action value prediction heads can be configured to generate different predicted action values for the same target action based on processing the feature embedding. For example, a first action value prediction head can generate a first predicted action value that indicates a likelihood of the target action being performed, a second action value prediction head can generate a second predicted action value that indicates a time length of the target action being performed, and so on.

[0085] The system processes, by the domain classifier head of the neural network, the feature embedding to generate a predicted domain score that represents a likelihood that the training input is in the first domain (or, analogously, in the second domain) (operation 308).

[0086] The system determines a loss function (operation 310). The loss function can include one or more action value loss terms that are defined with respect to the outputs of the action value prediction heads of the neural network. The action value loss terms can be used to updateparameter values of the one or more action value prediction heads of the neural network and, by virtue of backpropagation, parameter values of the backbone of the neural network.

[0087] For example, the loss function can include an action value loss term that corresponds to a respective predicted action value. In this example, each action value loss term measures, for each training input in the batch, a difference between a predicted action value and a corresponding ground truth action value associated with the training input.

[0088] As another example, the loss function can include one action value loss term that corresponds to all of the predicted action values, and measures, for each training input in the batch, a combined difference between the one or more predicted action values and their corresponding ground truth action values associated with the training input.

[0089] The loss function can also include a domain score loss term that is defined with respect to the outputs of the domain classifier head of the neural network. The domain score loss term measures, for each training input in the batch, a difference between the predicted domain score and a ground truth domain score of the training input. In the example where the predicted domain score represents a likelihood that the training input is in the first domain, the ground truth domain score for a first domain training input can be 1, and the ground truth domain score for a second domain training input can be 0. The domain score loss term can be used to update parameter values of the domain classifier head of the neural network and, by virtue of backpropagation, parameter values of the backbone of the neural network.

[0090] In one example, the loss function can be a weighted sum of all of the loss terms mentioned above, where each loss term is weighted by a corresponding weight. Such weights generally correspond to tunable hyperparameters of the system.

[0091] The system determines one or more updates to current values of parameters of the neural network based on optimizing the loss function (operation 312). The system can do this by computing, for each training input in the batch, respective gradients of the loss function with respect to the parameters of the neural network by backpropagation through the appropriate parameters of the neural network. The system can then determine the updates by applying an update rule, e.g., an Adam update rule, an Rmsprop update rule, or a stochastic gradient descent (SGD) update rule, to the respective gradients.

[0092] In some implementations where a gradient reversal layer is included in the neural network between the backbone and the domain classifier head, as illustrated in FIG. 2,computing the respective gradients of the loss function through b ackpropagation will also involve reversing the gradients backpropagated to (the last neural network layers included in) backbone from (the first neural network layers included in) the domain classifier head. This is explained in more detail with reference to FIG. 4, which shows sub-operations 402-406 corresponding to operation 312 that can be performed by some implementations of the system.

[0093] The system computes gradients of the loss function with respect to parameters of the domain classifier head (operation 402).

[0094] The system multiplies the gradients by a negative constant to determine reversed gradients (operation 404). For example, the negative constant can be a negative integer, e.g., -1, or another negative number.

[0095] The system computes, based on the reversed gradients and by backpropagation, gradients of the loss function with respect to parameters of the backbone (operation 406). That is, the gradients of the loss function with respect to parameters of the backbone are determined using the reversed gradients of the of the loss function with respect to parameters of the domain classifier head.

[0096] FIG. 5 is a block diagram of an example training system 500 for training a neural network 540. The training system 500 is an example of a system implemented as computer programs on one or more computers in one or more locations that trains the neural network 540 on a training dataset 510 to determine trained values of the parameters of the neural network 540.

[0097] The training dataset 510 can be generated from log data collected during interaction between the users and a service platform. The log data can include session data of user sessions. Each user session can include one or more actions performed by a user on a client device during the interaction. Each user session is either in a signed-in mode or in a signed-out mode.

[0098] More specifically, the training dataset 510 includes multiple training inputs 512. Each training input 512 includes multiple features. The multiple features include user-specific features 513 that are generated from a portion of the log data that is only collected during a signed-in mode, e.g., session data of user sessions that are in the signed-in mode. For example, the userspecific features 513 can include features determined from user-specific information that is identifiable from the usernames of signed-in users when interacting with the service platform.

[0099] The multiple features also include non-user-specific features 514 that are generated from another portion of the log data that is collected during a signed-out mode, e.g., session dataof user sessions that are in the signed-out mode, or during both the signed-in mode and the signed-out mode. No personally identifiable information is included in non-user-specific features. For example, the non-user-specific features 514 can include features related to the history actions performed by users when interacting with the service platform.

[0100] As a particular example for illustration, when service platform is a search engine platform, the non-user-specific features can include features that describe one or more search queries, one or more search results responsive to the queries, one or more actions (e.g., a selection, browse, copy, or print action) performed by a user responsive to the search results, and, in some cases, metadata (e.g., geographic location and timestamp metadata) associated with the search queries and / or the search results. In contrast, the user-specific features can include features that relates to the username of the user, that describes information identifiable from the username of the user (e.g., the gender and age of the user), or both.

[0101] The neural network 540 includes a first neural network tower 542 (or “a first tower” for short). The first tower 542 is configured to extract features from user-specific features included in a training input 512, that is, configured to process the user-specific features included in the training input 512, to generate a user feature embedding 543 of the training input 512.

[0102] The neural network 540 includes a second neural network tower 544 (or “a second tower” for short). The second tower 544 is configured to extract features from non-user-specific features included in the training input 512, that is, configured to process the non-user-specific features included in the training input 512, to generate a surrogate user feature embedding 545 of the training input 512. The surrogate user feature embedding 545 can have the same dimensionality as the user feature embedding 543.

[0103] The neural network 540 also includes one or more action value prediction heads 546A-N. The action value prediction heads 546A -N are configured to process the user feature embedding 543 generated by the first tower 542 from the user-specific features included in the training input 512 to generate predicted action values 547A-N, respectively, for the training input 512.

[0104] The predicted action values are typically defined with reference to a target action (examples of a target action are discussed above in the Summary section). For example, the predicted action values 547A-N can include a value that indicates a likelihood of a target action being performed by the user during a user session, following the user session, and / or after theuser views or interactions with a digital component. As another example, the predicted action values 547 A-N can include a value that indicates a time length of the target action being performed. As yet another example, the predicted action values 547A-N can include a predicted conversion rate for a digital component (a predicted likelihood of a user clicking (or otherwise selecting) a digital component upon viewing the digital component).

[0105] A “tower” of a neural network refers to a group of one or more neural network layers in the neural network. The first and second towers 542, 544 can each be implemented with any appropriate neural network architecture that enables them to perform their described functions. For example, the first tower 542 or the second tower 544 can include any appropriate types of neural network layers, e.g., embedding layers, fully connected layers, attention layers, and so forth, in any appropriate number, e g., 10 layer, or 50 layers, or 100 layers, and connected in any appropriate configuration, e.g., as a directed graph of layers. As a particular example, the first tower 542 can have an architecture that is deeper than the second tower 544; that is, the first tower 542 can include a larger number of layers than the second tower 544.

[0106] The training system 500 performs the training over multiple training iterations. At each training iteration, the training system 500 updates the parameters of the neural network 540 by performing a forward pass through the neural network using the training inputs 512 obtained from the training dataset 510 and then perform a backward pass through the neural network 540 to compute the respective gradients through backpropagation.

[0107] During training, the training system 500 trains the first tower 542 to generate semantically meaningful user feature embeddings 543 of the training inputs 512; trains the second tower 544 to generate surrogate user feature embedding 545 that are proximate in distance to the user feature embeddings 543 generated by the first tower 542 in an embedding space; and trains the action value prediction heads 546A-N to generate more accurate predicted action values 547A-N, i.e., to generate predicted action values that are close to the ground truth action values associated with the training inputs. These distances, or “similarities,” in the embedding space can be computed in various ways, such as with cosine similarity, dot products, etc.

[0108] After the training, the training system 500 outputs at least a portion of the trained neural network 540 to the Al system 140 of FIG. 1 for deployment to perform inference. For example, the training system 500 outputs the first tower 542, the second tower 544, and one ormore of the action value prediction heads 546A-N to the Al system 140 for inclusion in a neural network 550 that is included in the Al system 140 to perform inference. An example way of deploying and using the trained neural network 540 to perform inference will be described in FIG. 7.

[0109] Optionally, the training system 500, the Al system 140, or a different system can further fine-tune some or all of the parameters of the neural network 540 before using it to perform inference. When included in the neural network 550, the parameters of the trained parameter values of the first tower 542, the second tower 544, and the one or more action value prediction heads 546A-N may be frozen, or they may be further trained, e.g., on different training datasets, using a different loss function, or both, e.g., in conjunction with the parameters of the other layers of the neural network 550.

[0110] FIG. 6 is a flow diagram of an example process 600 for training a neural network. For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 500 of FIG. 5, appropriately programmed in accordance with this specification, can perform the process 600. Operations of the process 600 can also be implemented as instructions stored on one or more computer-readable media, which may be non-transitory, and execution of the instructions by one or more data processing apparatus can cause the one or more data processing apparatus to perform the operations of the process 600.

[0111] The system can repeatedly perform iterations of the process 600 on different batches of training inputs to update the parameters of the neural network, i.e., of the first tower, the second tower, and the one or more action value prediction heads.

[0112] The system can continue performing iterations of the process 600 until termination criteria for the training of the neural network have been satisfied, e.g., until the parameters have converged, until a threshold amount of wall clock time has elapsed, or until a threshold number of iterations of the process 600 have been performed.

[0113] The system obtains a batch of training inputs from a training dataset (operation 602). The system will generally obtain different training inputs at different iterations, e.g., by sampling a fixed number of inputs from a larger number of training inputs at each iteration. Each training input is associated with one or more ground truth action values for a target action that can beperformed by a user during the corresponding user session, either in the signed-in mode or in the signed-out mode.

[0114] The training dataset can be generated from log data collected during interaction between the users and a service platform. The log data can include session data of user sessions. Each user session can include one or more actions performed by a user on a client device during the interaction. Each user session is either in a signed-in mode or in a signed-out mode.

[0115] The training dataset includes multiple training inputs. Each training input includes multiple features. The multiple features include user-specific features that are generated from a portion of the log data that is only collected during the signed-in mode. The multiple features also include non-user-specific features that are generated from another portion of the log data that is collected during the signed-out mode, or during both the signed-in mode and the signed- out mode.

[0116] The system repeatedly performs following operations 604-608 for each training input included in the batch of training inputs obtained from the training dataset.

[0117] The system processes, by the first tower of the neural network, the user-specific features included in the training input to generate a user feature embedding (operation 604).

[0118] The system processes, by the second tower of the neural network, the non-user- specific features included in the training input to generate a surrogate user feature embedding (operation 606).

[0119] The system processes, by the one or more action value prediction heads of the neural network, the user feature embedding generated by the first tower from the user-specific features included in the training input to generate one or more predicted action values (operation 608). Different action value prediction heads can be configured to generate different predicted action values for the same target action based on processing the user feature embedding. For example, a first action value prediction head can generate a first predicted action value that indicates a likelihood of the target action being performed, a second action value prediction head can generate a second predicted action value that indicates a time length of the target action being performed, and so on.

[0120] The system determines one or more updates to parameter values of the neural network based on optimizing a loss function (operation 610). The system can do this by computing, for each training input in the batch, respective gradients of the loss function withrespect to the parameters of the neural network by backpropagation through the appropriate parameters of the neural network. The system can then determine the updates by applying an update rule, e.g., an Adam update rule, an Rmsprop update rule, or a stochastic gradient descent (SGD) update rule, to the respective gradients.

[0121] The loss function can include a surrogate loss term that is defined with respect to the outputs of the second tower of the neural network. The surrogate loss term measures, for each training input in the batch, a distance between the between the user feature embedding generated by the first tower and the surrogate user feature embedding generated by the second tower. The surrogate loss term can be used to update parameter values of the second tower of the neural network.

[0122] The loss function can include one or more action value loss terms that are defined with respect to the outputs of the action value prediction heads of the neural network. The action value loss terms can be used to update parameter values of the one or more action value prediction heads of the neural network and, by virtue of backpropagation, parameter values of the first tower of the neural network.

[0123] For example, the loss function can include an action value loss term that corresponds to a respective predicted action value. In this example, each action value loss term measures, for each training input in the batch, a difference between a predicted action value and a corresponding ground truth action value associated with the training input.

[0124] As another example, the loss function can include one action value loss term that corresponds to all of the predicted action values, and measures, for each training input in the batch, a combined difference between the one or more predicted action values and their corresponding ground truth action values associated with the training input.

[0125] In one example, the loss function can be a weighted sum of all of the loss terms mentioned above, where each loss term is weighted by a corresponding weight. Such weights generally correspond to tunable hyperparameters of the system.

[0126] Merely as an example, FIG. 7 depicts how the service apparatus 130 that includes the Al system 140 can generate one or more predicted action values for a target action, and then use the predicted action values to determine whether to select a digital component for presentation to the user on one of the client devices 105 A through 105N. Other ways of using the Al system 140to select digital components for presentation to users on the client devices 105 A through 105N are possible.

[0127] FIG. 7 is a flow diagram of an example process for using a neural network to perform inference. For convenience, the process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, a system, e.g., the Al system 140 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 700. Operations of the process 700 can also be implemented as instructions stored on one or more computer-readable media, which may be non-transitory, and execution of the instructions by one or more data processing apparatus can cause the one or more data processing apparatus to perform the operations of the process 700.

[0128] The system receives session data of a current user session that describes actions performed by a user during the current user session to interact with a service platform (operation 702). The user can be either a signed-in user or a signed-out user. If the user is a signed-in user, the system has access to additional user information that would not be available for a signed-out user. For example, the system can store user profile information for users that are registered with the system. This information can include the user’s name, geographic location, historical actions performed by the user at the system (e.g., resources and / or digital components requested by and / or selected by the user), and / or other information about the user.

[0129] The system processes the session data using a neural network to generate one or more predicted action values for a target action that can be performed by the user on the service platform (operation 704).

[0130] For example, the neural network can be the neural network 250 of FIG. 2. The neural network 250 includes the backbone 241 and the one or more action value prediction heads 244A- N that have been trained by the training system 200. The neural network receives, as input, session data of the current user session between the user and the service platform, and processes the input to generate, as output, the one or more predicted action values for the target action that can be performed by the user.

[0131] For example, the predicted action values can include a value that indicates a likelihood of the target action being performed by the user during the current user session or following the user session. As another example, the predicted action values can include a value that indicates a time length of the target action being performed by the user (e.g., an amount ofT1time the user will spend at a website after navigating to the website). As yet another example, the predicted action values can include a predicted conversion rate for a given digital component, e.g., a predicted likelihood of the user clicking or otherwise selecting or interacting with the given digital component upon viewing the digital component or performing a different target action after viewing or interacting with the digital component.

[0132] In this example, by virtue of how it is trained, the neural network 250 can generate accurate predicted action values because the backbone 241 is capable of encoding enough information in the feature embeddings even for user sessions that are in the signed-out mode, where the session data that is made available to the system generally includes a narrower range of types, a smaller amount, or both than the session data for those user sessions in the signed-in mode.

[0133] As another example, the neural network can be the neural network 550 of FIG. 5. The neural network 550 includes the first tower 542, the second tower 544, and the one or more action value prediction heads 546A-N that have been trained by the training system 500. The neural network receives, as input, session data of the current user session between the user and the service platform, and processes the input to generate, as output, the one or more predicted action values for the target action that can be performed by the user.

[0134] In this example, the neural network chooses between the first tower 542 and the second tower 544 to generate a feature embedding, and subsequently processes the feature embedding using the action value prediction heads to generate the predicted action values. This choice is made based on whether the current user session is in the signed-in mode or in the signed-out mode: if in signed-in mode, then the neural network chooses the first tower 542 to process the session data to generate a user feature embedding; alternatively, in signed-out mode, then the neural network chooses the second tower 544 to process the session data to generate a surrogate user feature embedding.

[0135] Likewise, in this example, by virtue of how it is trained, the neural network 550 can generate accurate predicted action values because the second tower 544 is capable of encoding enough information in the surrogate user feature embeddings even for user sessions that are in the signed-out mode, where the session data that is made available to the system generally includes a narrower range of types, a smaller amount, or both than the session data for those user sessions in the signed-in mode.

[0136] The system determines whether to provide a given digital component for presentation to the user on a client device based on the one or more predicted action values for the target action (operation 706). The system can select a given digital component from among a set of candidate digital components based on one or more predicted action values for each candidate digital component. For example, the system can use the prediction neural network to generate one or more predicted action values for each candidate digital component and each digital component can have one or more corresponding target actions, the target actions for the digital components can differ, e.g., one can be adding an item to a virtual shopping cart while another is registering for a newsletter.

[0137] In this example, the system can select, as the given digital component, the candidate digital component having the highest predicted action value or a highest combination (e.g., sum, average, or product of), multiple predicted action values generated for each candidate digital component. In another example, the system can select the given digital component based on a score that is based on a combination of the predicted action value(s) for the given digital component and a selection parameter, which can be an amount, e.g., a monetary amount, that the publisher of the given digital component is willing to provide to a publisher in exchange for the given digital component being presented with a resource, e.g., webpage, of the publisher.

[0138] In another example, the system can determine that the given digital component should be selected as the selected digital component for presentation to the user if each predicted action value satisfies a corresponding predetermined threshold action value. Alternatively, the system can determine that the given digital component should not be selected as the selected digital component if at least one predicted action value fails to satisfy a corresponding predetermined threshold action value.

[0139] For example, the given digital component can be selected by the Al system 140. As another example, the given digital component can be selected by another component of the service apparatus 130. The other component can be, but need not be, a machine learning model. For example, it can be a rule-based engine that analyzes the digital component database 150 to identify various digital components having distribution parameters that match a current user session of the interaction between a user and the service platform 120.

[0140] In various implementations, the process 700 can be performed prior to, in conjunction with, or subsequent to the actual selection of the given digital component. Advantageously, whenperformed prior to the actual selection process, assuming the one or more predicted action values do not satisfy the corresponding predetermined threshold action values, then no digital component needs to be selected from the digital component database 150. Because a selection process is not performed, the computing resource and / or power consumption of the system can be reduced.

[0141] FIG. 8 is a block diagram of an example computer system 800 that can be used to perform operations described above. The system 800 includes a processor 810, a memory 820, a storage device 830, and an input / output device 840. Each of the components 810, 820, 830, and 840 can be interconnected, for example, using a system bus 850. The processor 810 is capable of processing instructions for execution within the system 800. In one implementation, the processor 810 is a single-threaded processor. In another implementation, the processor 810 is a multi -threaded processor. The processor 810 is capable of processing instructions stored in the memory 820 or on the storage device 830.

[0142] The memory 820 stores information within the system 800. In one implementation, the memory 820 is a computer-readable medium. In one implementation, the memory 820 is a volatile memory unit. In another implementation, the memory 820 is a non-volatile memory unit.

[0143] The storage device 830 is capable of providing mass storage for the system 800. In one implementation, the storage device 830 is a computer-readable medium. In various different implementations, the storage device 830 can include, for example, a hard disk device, an optical disk device, a storage device that is shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large capacity storage device.

[0144] The input / output device 840 provides input / output operations for the system 800. In one implementation, the input / output device 840 can include one or more of a network interface devices, e.g., an Ethernet card, a serial communication device, e.g., and RS-232 port, and / or a wireless interface device, e.g., and 802.11 card. In another implementation, the input / output device can include driver devices configured to receive input data and send output data to other devices, e.g., keyboard, printer, display, and other peripheral devices 860. Other implementations, however, can also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0145] Although an example processing system has been described in FIG. 8, implementations of the subject matter and the functional operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0146] An electronic document (which for brevity will simply be referred to as a document) does not necessarily correspond to a file. A document may be stored in a portion of a file that holds other documents, in a single file dedicated to the document in question, or in multiple coordinated files.

[0147] For situations in which the systems discussed here collect and / or use personal information about users, the users may be provided with an opportunity to enable / disable or control programs or features that may collect and / or use personal information (e.g., information about a user’s social network, social actions or activities, a user’s preferences, or a user’s current location). In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information associated with the user is removed. For example, a user’s identity may be anonymized so that the no personally identifiable information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined.

[0148] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively, or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machinegenerated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or acombination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0149] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0150] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing The apparatus can include special purpose logic circuitry, e g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0151] This document refers to a service apparatus. As used herein, a service apparatus is one or more data processing apparatus that perform operations to facilitate the distribution of content over a network. The service apparatus is depicted as a single block in block diagrams. However, while the service apparatus could be a single device or single set of devices, this disclosure contemplates that the service apparatus could also be a group of devices, or even multiple different systems that communicate in order to provide various content to client devices. For example, the service apparatus could encompass one or more of a search system, a video streaming service, an audio streaming service, an email service, a navigation service, an advertising service, a gaming service, or any other service.

[0152] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form,including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0153] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0154] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0155] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0156] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0157] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

[0158] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, butrather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0159] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0160] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0161] What is claimed is:

Claims

CLAIMS1. A computer-implemented method for training a neural network, wherein the method comprises: obtaining, from a training dataset, a batch of training inputs comprising training inputs in a first domain that are generated from log data of user activities on a service platform collected during a signed-in mode for users signed into the service platform, and training inputs in a second domain that are generated from log data of user activities on the service platform collected during a signed-out mode for users that are not signed into the service platform; for each training input in the batch: processing, by a backbone of the neural network, the training input to generate a feature embedding; processing, by one or more action value prediction heads of the neural network, the feature embedding to generate one or more predicted action values for a target action; and processing, by a domain classifier head of the neural network, the feature embedding to generate a predicted domain score that represents a likelihood that the training input is in the first domain; determining a loss function that includes at least (i) one or more action value loss terms that measure, for each training input in the batch, a difference between the one or more predicted action values and one or more ground truth action values associated with the training input and (ii) a domain score loss term that measures, for each training input in the batch, a difference between the predicted domain score and a ground truth domain score of the training input; and determining one or more updates to current values of parameters of the neural network based on optimizing the loss function.

2. The method of claim 1, wherein determining the one or more updates to current values of parameters of the neural network comprises: computing gradients of the loss function with respect to parameters of the domain classifier head; multiplying the gradients by a negative constant to determine reversed gradients; and computing, based on the reversed gradients and by backpropagation, gradients of the loss function with respect to parameters of the backbone.

3. The method of any one of claims 1-2, wherein the neural network comprises a gradient reversal layer arranged between the backbone of the neural network and the domain classifier head.

4. The method of any one of claims 1-3, wherein a training input in the first domain includes at least one feature that is not included in a training input in the second domain.

5. The method of claim 4, wherein the at least one feature is generated from a portion of log data of user actions on the service platform that is only collected during the signed-in mode and not during the signed-out mode.

6. The method of any one of claims 1-5, wherein the target action comprises a user click on a digital component presented to the user.

7. The method of any one of claims 1-6, wherein the one or more predicted action values comprise a value that indicates a likelihood of the target action being performed.

8. The method of any one of claims 1-6, wherein the one or more predicted action values comprise a value that indicates a time length of the target action being performed.

9. A computer-implemented method for training a neural network, wherein the method comprises: obtaining a batch of training inputs generated from log data characterizing actions of users on a service platform, each training input in the batch comprising (i) user-specific features that are generated from a portion of the log data that is only collected during a signed-in mode and (ii) non-user-specific features that are generated from another portion of the log data that is collected during both the signed-in mode and a signed-out mode; for each training input included in the batch: processing, by a first tower of the neural network, the user-specific features included in the training input to generate a user feature embedding; and processing, by a second tower of the neural network, the non-user-specific features included in the training input to generate a surrogate user feature embedding; and determining one or more updates to parameter values of the second tower of the neural network based on minimizing a distance between the user feature embeddings and the surrogate user feature embeddings.

10. The method of claim 9, further comprising: for each training input included in the batch: processing, by one or more action value prediction heads of the neural network, the user feature embedding included in the training input to generate one or more predicted action values; and determining one or more updates to parameter values of the prediction head and to parameter values of the first tower based on minimizing a difference between the one or more predicted action values and one or more ground truth action values associated with the training input.

11. The method of any one of claims 9-10, wherein the action is a search action that uses a user query, and the non-user-specific features comprise features related to the search action.

12. A computer-implemented method comprising: receiving session data that describes actions performed by a signed-out user when interacting with a service platform; processing the session data using the neural network trained using the respective method of any preceding claim to generate one or more predicted action values for a target action that can be performed by the signed output user on the service platform; and determining whether to provide a digital component for presentation to the user based on the one or more predicted action values for the target action.

13. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-12.

14. One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-12.

Images