Reinforced learning for computational graphics

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, receive data representative of a computational graph including a plurality of nodes and directed edges unidirectionally connecting two adjacent nodes, and receive training data including a reward function for a reinforcement learning model. A value function of the reinforcement learning model is initialized. For each node that is not an end node of the plurality of nodes, a value function of the reinforcement learning model is determined. The determining comprises: updating the value function based on (i) respective directed edges starting from the node and connecting subsequent nodes and (ii) the reward function; it is determined that the updated value function converges, and in response, the updated value function is provided as a value function. A value function for processing a user query is stored and used to process the user query.

Description

Background Art

[0001] The present specification relates to processing user queries for information, for example, on an exchange platform, and in particular to processing user queries along a computational graph using a machine learning model.

[0002] The example exchange platform enables the exchange of goods, content and services between end users and providers. Providers can list or provide their goods, content and services on the exchange platform, and end users obtain goods, content and services from providers via the exchange platform.

[0003] Reinforcement learning systems can be deployed in such platforms to facilitate various operations of the platform, including, for example, searching and retrieving information, such as information related to items provided on the platform. In a reinforcement learning system, an agent typically interacts with the environment by performing actions selected by the reinforcement learning system in response to receiving an observation that characterizes the current state of the environment. Some reinforcement learning systems select an action to be performed by the agent in response to receiving a given observation based on the output of a neural network. Summary of the invention

[0004] This specification relates to processing user queries on a computing platform using a machine learning model. In more detail, some implementations described herein relate to user query processing in a computational graph framework assisted by reinforcement learning techniques. This includes training and using a reinforcement model to generate output data, which includes one or more predictions of a user input (e.g., a user query). For example, in the context of an exchange platform, a prediction may include a list of items corresponding to a user query. In addition, one of the embodiments described herein deploys a random walk sampling technique formulated based on implicit user feedback information, and performs random walk sampling on a computational graph representing a query-list relationship to generate multiple candidate trajectories connecting a query node (or initial node) and a list node (or end node).

[0005] Specific embodiments of the subject matter described in this specification can be implemented to obtain one or more of the following advantages. For example, the innovation described in this specification improves the accuracy for processing user queries. Using a reinforcement learning model, the system can inject historical data as context when processing user queries using a computational graph. In addition, the system can bias the weight values ​​for directed edges based on negative interactions, which further improves accuracy compared to techniques that only consider positive interactions. In this document, positive interactions include clicking, adding to a shopping cart, purchasing items in a list, or any other interaction that expresses interest in an item, while negative interactions include scrolling through or indicating a lack of interest in a list of items. The system also includes a scoring function for sampling random walks, wherein the scoring function is determined based on a similarity measure between adjacent nodes. Within the context of reinforcement learning techniques, the scores obtained from a more accurate scoring function can improve the efficiency and accuracy of training the corresponding reinforcement learning model. In this way, the system can reduce or avoid inaccurate predictions of different node groups in the computational graph.

[0006] In addition, the techniques described in this specification can improve the efficiency of processing user queries. Executing the operations of the trained reinforcement learning model is generally computationally efficient. Using a transition matrix to predict node-to-node transitions along corresponding edges can significantly reduce computational costs compared to training policy functions for reinforcement learning models with large number of model parameters and / or nonlinear operations for predicting actions during inference operations.

[0007] In some implementations, the techniques described herein can utilize a Monte Carlo value function to generate a trajectory from an initial node to an end node that is relatively shorter than a trajectory generated by prior art techniques. In this way, the techniques described herein facilitate providing data associated with an end node in response to a user query by performing fewer computational steps.

[0008] In addition, as described below in conjunction with one embodiment, by deploying a specific random walk sampling algorithm using implicit user feedback (e.g., a user's reaction to a previously generated output list corresponding to a previous query), the described technology further enhances the efficiency of training and using the trained model to process user queries. In general, the system described herein can alleviate the cold start problem common in general sampling techniques by associating a specific list of items with formative attributes such as stores and tags in a directed computational graph. In addition, the system can further help the cold start problem by incorporating implicit user feedback into the initial edge weights of the directed computational graph (and the initial probability distribution of the random walk sampling from multiple potential edges in the end). In addition, the initial edge weights can be updated using reinforcement training techniques during the training process, and the updated edge weights can be stored offline in a memory. The trained random sampling weights are used to generate random walk samples for processing user queries. In this way, the system can use the described technology in a part of traditional technologies such as graph neural networks to train or perform reasoning operations for input user queries, and is scalable and efficient for computational graphs with billions of nodes and tens of billions of edges.

[0009] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram of an example environment in which an exchange platform facilitates the exchange of goods, services, or content between providers and users.

[0011] Figure 2 is a block diagram illustrating an example system for processing user queries.

[0012] Figure 3 A computational graph is illustrated that represents a relationship between a user query and a list of items that are responsive to the user query.

[0013] Figure 4 Illustration of two clusters in a computational graph.

[0014] Figure 5 is a flowchart of an example process for training a reinforcement learning model.

[0015] Figure 6 is a flowchart of an example process of using the trained reinforcement learning model to process user queries.

[0016] Figure 7 is a flow chart of an example process for determining a state-value function when training a reinforcement learning model.

[0017] Figure 8 is a block diagram of a computing device that may be used to implement the systems and methods described herein.

[0018] Similar reference numbers and designations in different drawings indicate similar elements. DETAILED DESCRIPTION

[0019] The described techniques generally involve processing user queries to provide output predictions (e.g., product lists) in response to user queries on a computing platform (e.g., an exchange platform). In general, an exchange platform is a platform configured to support item exchanges and provide information or item lists in response to user queries. The techniques described herein are also applicable to other applications, including but not limited to search engine applications, recommendation systems, etc. For brevity and ease of description, the following description is described in the context of an exchange platform.

[0020] In an example implementation in the context of an exchange platform, a provider may provide items (e.g., goods, services, and content) available to users of the exchange platform (e.g., registered members of the platform or visitors of the platform) on the exchange platform. Within a web page provided by the exchange platform, a user may enter a search query into a search bar, and in response, the exchange platform may identify listings responsive to the query and provide the listings within a first web page that lists items matching the search query. The user may interact with the items, for example, browse the items without further action, click on one or more listings to obtain more information related to the corresponding items, add one or more items to a shopping cart, or purchase one or more items.

[0021] The relationship between a user query and a list of items on an exchange platform can generally be represented using a computational graph, and the described technology can use the computational graph to determine the output (e.g., a list of items) to be provided in response to the user query. The computational graph can be represented as a directed graph or an undirected graph. A directed graph can include multiple nodes and directed edges connecting corresponding nodes. In such a graph, the user query can be represented by an initial node, and a list of specific products is represented as an end node. By searching for a trajectory including multiple connected directed edges that link nodes starting from the initial node (i.e., the user query) to an end node (e.g., a list of specific items), an end node can be provided as an output in response to processing the initial node. The following is in conjunction with the attached Figure 3 and 4 Describes computational graphs in more detail.

[0022] In order to search for trajectories in response to specific user queries, heuristic methods can be used to bias directed edges, but such methods may suffer from multiple disadvantages. First, such methods generally assume an unbiased prior probability distribution of directed edges (i.e., the unconditional probabilities of all directed edges are equal to each other). The unbiased prior probability assumption does not take into account the context or cumulative information of the nodes connected by the directed edges, the information associated with the specific user account requesting the user query, or other statistical data obtained or stored in the exchange platform. Second, such bias-based methods also do not implement a mechanism to consider user or system feedback when selecting one of the directed edges, let alone distinguish between positive and negative interactions of users. Third, these methods also do not optimize or control the biased directed edges across different clusters in the directed graph. In general, each cluster in a directed graph generally includes multiple nodes that are semantically similar to each other, and nodes from different clusters are generally semantically far away from each other. Due to the lack of full consideration of the dissimilarity across different clusters, traditional techniques may predict a list of items in a cluster different from the cluster to which the corresponding user query belongs, reducing the accuracy and efficiency of processing user queries.

[0023] The techniques described in this document can at least address the above limitations. By training and using reinforcement learning models and taking into account user feedback information, the described techniques can effectively process user queries in computational graphs.

[0024] Figure 1 1 is a block diagram of an example environment 100 in which an exchange platform 110 facilitates the exchange of items (such as goods, services, or content) between providers and users. The example environment 100 includes a network 104, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 104 connects one or more user devices 102, one or more provider devices 106, the exchange platform 110, and one or more external sources 108.

[0025] User device 102 and provider device 106 are electronic devices capable of requesting and receiving data over network 104. Examples of such devices include personal computers, mobile communication devices, digital assistant devices, and other devices that can send and receive data over network 108.

[0026] The exchange platform 110 is a computing platform that can be operated and maintained by an exchange service provider. The exchange platform 110 enables providers to list their items on the exchange platform 110 and enables users to obtain items listed on the exchange platform 110. Figure 1As depicted in the block diagram in , the exchange platform 110 is depicted as a single block with various sub-blocks. However, while the exchange platform 110 may be a single device or a single group of devices, the present specification contemplates that the exchange platform 110 may also be a group of device devices, or even a plurality of different systems that communicate with each other to enable the exchange of goods, services, and / or content. The exchange platform 110 may also be a provider of items, i.e., the provider device 106 and the exchange platform 110 may be integrated with each other. Alternatively, the exchange platform 110 may be an entity different from the provider, i.e., the provider device 106 and the exchange platform 110 may be separate entities, such as Figure 1 as shown in .

[0027] The provider uses an application 106-A executed on the provider device 106 to communicate with the exchange platform 110, for example, to create or manage the provider's item list on the exchange platform 110 and / or perform other appropriate tasks related to the exchange platform 110 (e.g., transferring an amount to the provider based on the item obtained by the user). The application 106-A can transmit data to the exchange platform 110 through the network 104, and receive data from the exchange platform 110. The application 106-A can be implemented as a local application developed for a specific platform or a specific device, a web browser providing a web interface, or another appropriate type of application. The application 106-A can present and detect user interactions (e.g., a user's touch, a mouse click, etc.) using various interfaces that enable, for example, the provider to create and manage the provider's item list on the exchange platform 110.

[0028] A user of the user device 102 may use the application 102-A to communicate with the exchange platform 110, for example, to view a list of items, search for items, obtain items, and / or perform other appropriate tasks related to the exchange platform 110. The application 102-A may transmit data to the exchange platform 110 through the network 104, and receive data from the exchange platform 110. The application 102-A may be implemented as a native application developed for a specific platform or a specific device, a web browser providing a web interface, or another appropriate type of application. The application 102-A may present and detect user interactions (e.g., a user's touch, a mouse click, etc.) using various interfaces that enable, for example, the user to view a list of items, search for items, obtain items, and / or perform other appropriate tasks related to the exchange platform 110.

[0029] The exchange platform 110 includes one or more front-end servers 112 and one or more back-end servers 114. The front-end servers 112 can transmit data to the user device 102 and the provider device 106 through the network 104, and receive data from the user device 102 and the provider device 106. For example, the front-end server 112 can provide interfaces and / or data to the applications 102-A and 106-A executed on the user device 102 and the provider device 106, respectively, for presentation with the interfaces. The front-end server 112 can also receive data specifying the user's interaction with the interface provided by the front-end server 112 to the user device 102 and the provider device 106. The front-end server 112 can update the interface, provide a new interface, and / or update the data presented by the interface presented in the application 102-A and 106-A, respectively, based on the user / provider's interaction with the user device 102 and the provider device 106.

[0030] The front-end server 112 may also communicate with the back-end server 114. For example, the front-end server 112 may identify data to be processed by the back-end server 114, such as data specifying information required to create a list requested by the provider 106, data specifying the quantity of a given item that the user of the user device 102 is requesting to obtain. The front-end server 112 may also receive data for a particular user of the user device 102 or data for a particular provider of the provider device 106 from the back-end server 114 and transmit the data to the appropriate user device 102 or provider device 106 via the network 104.

[0031] Backend server 114 includes item engine 116, search engine 118, and ranking engine 120. As used in this specification, the term "engine" refers to hardware, such as one or more data processing devices (e.g., one or more processors), that executes a set of programmed instructions that result in the performance of a set of tasks. Figure 1 These three engines are depicted, but the operations of these engines as described in this specification may be performed in whole or in part by one or more other engines. In other words, some implementations may include more than one Figure 1 , to perform the operations described in this specification. Alternatively, some implementations may include fewer engines that are used to perform the operations described in this specification. Furthermore, even if an implementation includes Figure 1 The same three engines depicted in , as depicted in this specification, operations performed by one of these engines can also be performed by one or more other engines.

[0032] The item engine 116 manages the creation and modification of item lists, as requested by a provider via an application 106-A on a provider device 106. The item engine 116 may receive data specifying an item description for a list initiated by a provider from a front-end server 112. Based on the description, the item engine 116 may create a list within the exchange platform 110. The description of the item may include, for example, the name of the item, a brief description of the item, the number of items, the amount required to obtain a particular item, the amount required to deliver the item to a destination, the completion time of the item to arrive at the destination, and one or more images of the item. The item engine 116 may use some or all of this information to create a list of items on the exchange platform 110. The item engine 116 may store the data for the list, including the received information, in an item data storage device 124. The item data storage device 124 may include one or more databases (or other appropriate data storage structures) stored in one or more non-transitory data storage media (e.g., (one or more) hard drives, flash memory, etc.).

[0033] The item engine 116 may also receive data from the front-end server 112 specifying characteristics of a list of items that the provider 106 may want to modify. For example, the provider 106, through the application 106-A, may attempt to modify one or more characteristics of the provider's items listed on the exchange platform 110. The modified characteristics are transmitted from the application 106-A to the front-end server 112 via the network 104. The item engine 116, in turn, receives data from the front-end server 112 specifying characteristics of a list of items that the provider 106 wants to modify. The characteristics to be modified may include, for example, the number of available items and the amount of money required to obtain the items. The item engine 116 may use the data about the modified characteristics to modify the list of items on the exchange platform 110. The item engine 116 may then use the modified characteristics to update the item characteristics stored in the item data storage device 124.

[0034] The search engine 118 manages the retrieval of item lists (as requested by a user via the application 102-A on the user device 102). The search engine 118 may receive data specifying a user's request to search for items from the front-end server 112, and retrieve items according to the user's request. If a user searches for an item or item type on the exchange platform, the user enters a search query into the search bar of the application 102-A. The user's search query is received by the front-end server 112, which in turn sends the search query to the search engine 118. The search engine 118 uses the data specified in the search query to identify (one or more) lists stored in the item data storage device 124. The search engine 118 transmits the (one or more) identified lists to the front-end server 112, which in turn provides a specific list or list summary for presentation on the application 102-A. If the list summary is presented to the user in the application 102-A, the user may select a link to a list from among the list summaries. The user's selection of the link is received by the front-end server 112, which interprets the user's selection as a request for data about a specific list. The front-end server 112 requests the search engine 118 to provide data about a particular list, which the search engine 118 obtains from the item data storage device 124. The search engine 118 responds to the front-end server 112 with the obtained data, which is then provided by the front-end server 112 to the application 102-A in the form of a page showing the list of items. The page can be a content page in a native application or a web page in a web browser.

[0035] The search engine 118 may also include a reinforcement learning engine 122 configured to process user queries within the context of the computational graph. Figure 2 Further details of this process and the reinforcement learning engine 122 are described.

[0036] If a summary of a list, i.e., a plurality of items, is presented to the user in application 102-A, the ranking engine 120 is used to rank the plurality of items in descending order of relevance to the user. The ranking engine extracts a set of descriptive attributes from each item (e.g., its color, texture, material, and shape) and learns which attributes the user likes or dislikes, forming an interpretable user preference profile for ranking the plurality of items in real time. An affinity score (e.g., a Beta distribution) is calculated for each attribute. Parameters corresponding to the affinity score for each attribute are stored in an attribute data storage device 128. The attribute data storage device 128 may include one or more databases (or other appropriate data storage structures) stored in one or more non-transitory data storage media (e.g., (one or more) hard drives, flash memory, etc.).

[0037] When a user views a list of items displayed on the exchange platform on application 102-A, the user may decide to obtain the item. The user may select a button (or other appropriate user interface element) on an interface presented on application 102-A, which may cause front-end server 112 to present a different user interface to the user, wherein the user may enter relevant details (e.g., quantity of items, destination address, payment information) to begin the completion process for obtaining the item. Upon submitting this information (e.g., by clicking a submit button on the user interface), the details entered by the user and the characteristics of the item desired by the user are received by front-end server 112 and passed to item engine 116. Item engine 116 evaluates whether the received data is valid (e.g., whether the quantity of items requested by the user is equal to or less than the available quantity of the item, whether the shipping address is correct, whether the payment information is correct).

[0038] If the data received from the user is invalid, item engine 116 sends a message to the front-end server indicating that the request was rejected, and optionally, may also include an explanation of why the request was rejected (e.g., the payment instrument was not approved or the shipping address entered was invalid). Front-end server 112 may provide a new user interface for presentation in application 102-A, in which the user is notified that the user's request was unsuccessful.

[0039] However, if the data received from the user is valid, the item engine 116 processes the payment using the received payment information and sends a message including the received user data to the appropriate provider to begin the fulfillment process. The item engine 116 may store purchase information about the item (e.g., an identifier of the user who purchased the item, the number of items purchased, the amount provided for the item, the date of purchase) in a purchase data storage device 126. The purchase data storage device 126 may include one or more databases (or other appropriate data storage structures) stored in one or more non-transitory data storage media (e.g., (one or more) hard drives, flash memory, etc.). Subsequently, the item engine 116 may send a message to the front-end server 112 indicating that the fulfillment process has begun. Upon receiving the message from the item engine 116, the front-end server 112 may provide a new user interface for presentation in the application 102-A, in which the user is notified that the user's request was successful and that order processing has begun.

[0040] Figure 2 2 is a block diagram illustrating an example system (e.g., reinforcement learning engine 210) for processing user queries. Reinforcement learning engine 210 can be equivalent to reinforcement learning engine 122, which is Figure 1 A portion of the search engine 118 is depicted in FIG.

[0041] like Figure 2As shown in , reinforcement learning engine 210 can process input including one or more user queries 205 and graph data 213 to generate output data 225 in response to one or more user queries 205.

[0042] Can be exchanged by a platform (e.g. Figure 1 exchange platform 110) from a user device (e.g., Figure 1 The user query 205 is received by the user device 102 of the exchange platform. The user device may include a smart phone, a smart tablet or a smart wearable device (e.g., a smart watch), a computer, or other suitable user device. The user account may be a registered account of the exchange platform, or a guest account or identifier of an unregistered user (e.g., not registered) affiliated with the platform. The user query may include a request for data representing a list of items (e.g., goods, services, and content) on the exchange platform 110 that are available to users of the exchange platform. The exchange platform may transmit data corresponding to the user query for presentation (e.g., display) on the user device.

[0043] The user query typically includes information that can be appropriately represented by a node in a computational graph. For example, the user query may include a text portion, a graph portion, or a combination of both. The data corresponding to the user query may include a list of items that are semantically related to the user query. For example, the user query may include the text "ring" and be represented as the initial node of a directed graph. The data corresponding to and responding to the user query may be represented by another node of the directed graph (referred to herein as an end node) and include a list of goods to be displayed on the user device, such as "wedding rings", "engagement rings", "diamond rings" or other related goods. In some implementations, the user query may also include information such as a user identifier, a user's previous social activities on the exchange platform, historical information related to the user's previous purchases, sales, or both, and tags of genres (genres) associated with the user account involving at least one of preferences, items, services, content, or activities.

[0044] In addition to the description throughout this document, controls may be provided to the user that allow the user to choose whether and when the systems, programs, or features described herein may enable the collection of user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current location), as well as whether to send content or communications from a server to the user. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that personally identifiable information about the user cannot be determined, or the user's geographic location (e.g., city, zip code, or state level) may be summarized where location information is obtained so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how that information is used, and what information is provided to the user.

[0045] Graph data 213 can be generated from historical data representing the relationship between user queries and corresponding item lists. For example, historical data can include historical user queries and corresponding item lists provided to historical user queries. Historical data can also include user interactions with the provided item lists, such as clicks, adding to a shopping cart, purchases, or indicating lack of interest. Graph data 213 can be represented as a computational graph. A computational graph typically includes multiple nodes interconnected by unidirectional edges. Within the context of an exchange platform, and assuming that the computational graph is directed, user queries can be grouped into initial nodes, and list items can be grouped into end nodes. The initial node in the computational graph does not have any preceding nodes, i.e., the initial node is the starting point of a unidirectional edge. The end node in a directed graph does not have any subsequent nodes, i.e., the end node is the end point of a directed edge, and no further (one or more) unidirectional edges extend from the end node to (one or more) other nodes (except the node before the end node).

[0046] The computational graph may form a trajectory, each trajectory starting from an initial node and ending at an end node. The computational graph may also include intermediate nodes located along the trajectory connecting the initial node and the end node. The intermediate nodes may include, for example, a store node (S) representing a store on the exchange platform, a tag node (T) representing a seller-specific tag or category of a corresponding product, and a user node (U) representing a user associated with a user query. In some cases, the store node and the user node may be represented by a digital array (e.g., "user: 5928" and "store: 1529"), while the tag node and the user query node may be represented by a string or series of characters or letters (e.g., "tag: clothing" and "query: wedding dress").

[0047] A directed graph also includes a plurality of directed edges, each directed edge unidirectionally connecting two adjacent nodes. More specifically, a directed edge traverses from a first node of a pair of adjacent nodes to a second node of the pair of adjacent nodes. The first node is also referred to as a previous node relative to the second node, and the second node is referred to as a subsequent node relative to the first node. In addition, there may be two opposite directed edges between a pair of adjacent nodes. For example, for two adjacent nodes A and B that are not end nodes, a first directed edge may go from node A to node B, and a second directed edge may go from node B to node A. In addition, each directed edge may be associated with a weight or score indicating a probability or reward of transferring from the first node to the second node along the corresponding directed edge. In combination Figure 3 Describes computational graphs in more detail.

[0048] The graph data 213 representing the above-mentioned computation graph may be stored in the exchange platform 110 , such as a memory device of the exchange platform 110 . The reinforcement learning engine 210 may receive the graph data 213 from the memory device of the exchange platform 110 .

[0049] To generate output data 225 for a user query 205, the reinforcement learning engine 210 may use the trained model weight values ​​to predict one or more trajectories that have the user query. Each trajectory includes one or more directed edges selected from an interconnected computational graph and corresponding nodes connected by these directed edges. Each trajectory starts from an initial node and ends at an end node. The initial node corresponds to a specific user query, and the end node corresponds to a list of items in response to the specific user query 205. As described above, the output data 225 corresponding to the user query may include representations (e.g., graphics, text, or both, such as a list) of one or more items. The list of items may include products, such as rings, books, mobile phones, laptops, or other suitable products.

[0050] The reinforcement learning engine 210 may include a training engine 220, which is used to train a reinforcement learning model through training samples. The training samples may include historical data, such as historical user queries and historical output data (e.g., one or more item lists) in response to the historical user queries. In some cases, the training samples may also include user feedback information. For example, the user feedback information may include user interactions with the historical output data in response to the historical user queries. User feedback may include: positive feedback or interactions, such as clicking, adding to a shopping cart, or purchasing an item list; and negative feedback or interactions, such as indicating a lack of interest in the item list. The training engine 220 may update the model weight values ​​of the reinforcement learning model by optimizing an objective function. In the context of the reinforcement learning model, the objective function may include the total reward (e.g., the value function of a state, also referred to as a state value function) in each period (e.g., one or more time steps). The reinforcement learning engine 210 may store the trained reinforcement learning model in the memory 215 and deploy the trained model to process the user query 105 and the graph data 213 to generate the output data 225. In some implementations, the reinforcement learning engine 210 may store a converged state value function or a policy function (e.g., a transition matrix) of the trained reinforcement learning model in the storage device 215 , and use the converged state value function or the policy function to process the user query to generate output data 225 .

[0051] The reinforcement learning engine 210 can "implicitly" (e.g., the occurrence of different types of user interactions) use user feedback information to generate initial weights assigned to corresponding directed edges, and update the initial weights via training a specific reinforcement learning model, rather than using heuristically determined weights to bias directed edges to process user queries. The user feedback information represents positive and negative user feedback in historical data, so the initial weights can be more accurate than the weights generated without such user feedback, thereby further improving training efficiency and reducing training costs. Positive user feedback can include, for example, clicking, adding to a shopping cart, or purchasing a list of items in response to a previous user query, while negative user feedback can include, for example, scrolling through or closing a user interface that displays a list of items provided in response to a user query. Typically, the initial weights determined for directed edges connecting nodes with more cumulative positive reactions typically have higher values ​​than other directed edges. That is, these directed edges with higher weight values ​​are "biased" relative to other directed edges. In addition, the initial weights can also be used to sample random walks (e.g., a strategy of moving from one node to another via directed edges) for training and using reinforcement learning models. The following is combined with Figure 7 Describes the details of the random walk sampling technique used in reinforcement learning models. In addition, Figure 5-7 Two different reinforcement learning models and their corresponding training and inference operations are described in more detail.

[0052] Figure 3 A computational graph 300 is illustrated that represents the relationship between a user query and a list of items responsive to the user query.

[0053] Generally speaking, exchange platforms (e.g. Figure 1 The exchange platform 110 may include historical data representing the relationship or interaction between user queries and corresponding output data for the user queries. For example, the historical data may include previous user queries related to "wedding", "wedding party", "wedding speech", or "wedding guests", and corresponding output data in response to the previous user queries. For example, the output data represents a list of items related to "wedding", "wedding party", "wedding speech", or "wedding guests". These item lists are associated with items provided by providers on the exchange platform. The relationship and interaction between user queries and corresponding output data can be used to generate a computational graph (e.g., Figure 3 Calculation graph 300).

[0054] like Figure 3 As shown in , the computation graph 300 may include one or more initial nodes (Q) representing user queries. The initial nodes may include a first user query 320A related to "wedding party", a second user query 320B related to "wedding speech", a third user query 320C related to "wedding", and a fourth user query 320D related to "wedding guests".

[0055] The computational graph 300 may include one or more end nodes (L) representing output data (e.g., one or more item lists) for corresponding user queries. In some cases, for an exchange platform having a list catalog from a provider that operates a store or shop on the exchange platform, the lists of items may be further grouped by the store or shop to which they belong, and each list may be further labeled with different tags by the provider to better describe the item. In view of this, the end nodes (L) may be further divided into store nodes (S) and tag nodes (T). The store node (S) typically represents a unique store identifier for the corresponding store, and the tag node (T) typically represents a corresponding tag for the item list. For example, and as Figure 3As shown in , the initial node 320A is linked to one or more store nodes 330A, each of which has a unique identifier, such as represented by a series of numbers. The initial node 320A is also linked to one or more tag nodes 310A, each of which has a corresponding tag, such as "n_wedding_shower_bingo", "n_shower_bingo", "n_wedding_shower". It should be appreciated that the tags used for different item lists do not have to be unique, as long as the tags can provide additional searchable metadata. In some cases, two or more item lists may share a common tag. For example, the system (or a seller or store owner of an exchange platform) may associate a list representing a piece of clothing with a tag (e.g., "tag: clothing"). Based on this tag and other tags, the system can generate a directed computational graph with multiple nodes, so that the node corresponding to "tag: clothing" has one or more outgoing edges pointing to all list nodes associated with "tag: clothing".

[0056] Each end node (L), (S) or (T) is connected to the initial node (Q) by a corresponding edge in the computational graph 300, and each corresponding edge has a weight value associated therewith. An edge can have one or two directions, and each direction can have a corresponding weight value. More specifically, the weight values ​​of two opposite directed edges between two adjacent nodes can be different. For example, a directed edge from node A to node B can be determined with a weight value of 0.2, while another directional edge from node B to node A can be determined with a weight value of 1.4. It should be noted that these weight values ​​are provided as examples and should not be used to infer the range or scale of weight values. In some cases, the weight values ​​can be normalized so that the range of weight values ​​is from 0 to 1.

[0057] The weight value can represent the possibility or immediate reward of transferring between nodes along that direction. The weight value or edge weight can be determined using different techniques (e.g., heuristic techniques, incorporating user feedback information, reinforcement learning, or other suitable techniques). Figure 7 Describe the process of determining weight values.

[0058] like Figure 3 As shown in , a first edge 350A from node 320A to one of the store nodes 330A (e.g., "592365045") may have a weight value of 0.971, while a second edge 340A from node 320A to one of the label nodes 310A (e.g., "n_shower_bingo") may have a weight value of 1.

[0059] Similarly, other user query nodes (e.g., nodes 320B, 320C, and 320D) can be connected to their corresponding store nodes (e.g., nodes 330B, 330C, and 330D) and label nodes (e.g., nodes 310B, 310C, and 310D) using corresponding edges (340B, 340C, 340D, 350B, 350C, and 350D) with corresponding weight values.

[0060] In addition, the computation graph can have one or more different clusters, each of which can have one or more user query nodes and corresponding list nodes (store nodes and label nodes). In general, since different clusters can represent different semantic information determined from different user queries, user queries in different clusters are usually less similar than user queries in the same cluster. The described techniques can apply a specific algorithm to "bias" edge weight values ​​to reduce the likelihood of transitions across different clusters. Figure 7 Describes more details on determining weight values.

[0061] Figure 4 Two clusters 430 and 460 in a computation graph 400 are illustrated. As an example, the computation graph 400 includes a first cluster 401 and a second cluster 403. Although the two clusters 401 and 403 share different characteristics, in some cases where a soft boundary is applied, a node 440 or 450 or both may be assigned to the two clusters 401 and 403. Therefore, the two clusters 401 and 403 may be connected to each other by two directed edges 443 and 445 connecting the two nodes 440 and 450. The first cluster 401 may include a plurality of nodes, such as nodes 410, 420, 430, and 440, and one or more of these nodes in the first cluster 401 may be a query node(s), while the remaining nodes may be list, store, or label nodes(s). Similarly, the second cluster 403 may also include multiple nodes, for example, nodes 450, 460, 470, and 480, and one or more of these nodes in the second cluster 403 may be (one or more) query nodes, while the remaining nodes may be list, store, or (one or more) tag nodes. Nodes in the same cluster are connected to each other by directed edges associated with weight values. As described above, the weight value generally refers to the probability or reward of transferring from a first node to a second node along the edge. In addition, in order to efficiently and accurately determine the list nodes for user queries, the weight values ​​assigned to the edges connecting nodes in the same cluster should be "biased" by the algorithm so that edges connecting nodes in the same cluster can have larger weight values ​​than edges connecting nodes across different clusters. For example, and as Figure 4As shown in , the weight values ​​of edges 443 and 445 connecting nodes 440 and 450 from two clusters are less than 0.2, while the weight values ​​of other edges (e.g., edges 413 and 415) connecting nodes in the same cluster (e.g., nodes 410 and 420) are at least greater than 0.23. The weight values ​​of the edges are determined by training a reinforcement learning model, and in some cases, the exchange platform can determine and assign initial weight values ​​of different edges based on user feedback information from stored historical data. Combined Figure 7 More details on determining weight values ​​are described.

[0062] Figure 5 is a flow chart of an example process 500 for training a reinforcement learning model. For convenience, process 500 is described as being performed by a system of one or more computers located in one or more locations. For example, a system (e.g., Figure 1 Exchange platform 110, Figure 1 Search engine 118 or Figure 1 The reinforcement learning engine 122) can perform process 500 when appropriately programmed.

[0063] In order to train a reinforcement learning model, a system implementing the technology first receives data representing a computational graph (502). As described above, the computational graph may include a directed graph having a plurality of nodes and directed edges. Each directed edge in the plurality of directed edges connects two adjacent nodes in the plurality of nodes. The directed edge starts from the previous node of the two adjacent nodes and ends at the subsequent node of the two adjacent nodes. The plurality of nodes include: one or more initial nodes, each initial node corresponding to a user query; and one or more end nodes, each end node associated with data corresponding to the corresponding user query. Typically, the initial node does not have any preceding nodes connected by directed edges. Similarly, the end node does not have any subsequent nodes connected by directed edges. However, in some cases, if nodes are selected as starting nodes for random walk sampling, they are referred to as initial nodes. Similarly, if the nodes happen to be the end nodes where the random walk stops, they are referred to as end nodes. In this context, the initial node and the end node may have inbound and outbound edges.

[0064] As an example, the described technology can construct a weighted bipartite graph G = (Q, L, E, W), where Q represents the query node (or initial node), L represents the list node (or terminal node), and E represents the edge connecting different nodes, for example, E = {e i，j =(q i , l j )|q i ∈Q&l j ∈L} and W represents the weight value for the corresponding edge E.

[0065] In some cases, the system can extend the computational graph to include additional types of nodes for enhanced connectivity, which in turn can improve the efficiency and accuracy of processing user queries. For example, the system can extend the above-described weighted bipartite graph G by including store nodes (S) and label nodes (T) into the graph G. In this way, the system can generate extended connectivity for historical query nodes and list nodes. In addition, the extended graph is still bipartite: {Q, S, T}, which is a separate partition from the list node L. Therefore, the above-described expansion techniques do not make the extended graph incompatible with the reinforcement learning framework assigned to the original weighted bipartite graph G. On the contrary, the algorithms and techniques applied to the original bipartite graph remain applicable to the extended bipartite graph. In some cases, the weight value of the edge between a store or label node and a list node can be set to 1, for example,

[0066] The system receives training data (504) for training a reinforcement learning model. The training data may include multiple training examples. For example, the training data may include multiple query list pairs. Each query-list pair includes a specific user query received by the system and a corresponding list of ground truth items corresponding to and responsive to the received user query. The ground truth items can be automatically annotated or manually annotated using a preprocessing model. In some implementations, the query-list pair may include historical data, wherein each query-list pair in the historical data includes a user query and a corresponding list provided to the user query that has reached a threshold effect. For example, the provided list of items has reached a threshold percentage of conversion rate, for example, 80% of the user accounts requesting the user query have a positive interaction with the provided list. As described above, a positive interaction may include clicking or tapping a representation of a list, completing a transaction with the list, saving the list on a user device for later use, or other suitable positive interactions.

[0067] In general, historical data may implicitly carry user feedback information. Such feedback information may include positive user feedback and negative user feedback. Positive feedback may include, for example, when a list of items is provided in response to a specific user query, a user clicking on a list of items, adding a list of items to a shopping cart, or purchasing a list of items. Negative feedback may include a user scrolling over a list of items displayed on a user interface, closing the user interface, indicating a lack of interest in a display, turning off a display, or other suitable negative feedback. For example, historical data may be stored in a memory as a specific data structure (e.g., a query log).

[0068] The described framework for training and using reinforcement learning models to process user queries in a computational graph setting takes into account implicit user feedback information. More specifically, given a record for each query q i The query log, the set of lists Respectively represent list nodes (or terminal nodes) provided in response to user click, add to cart and purchase queries. Note that the above types of user feedback information are listed as examples to help describe the corresponding algorithms described below, and other suitable classifications for classifying or defining implicit user feedback information can be used.

[0069] To generate the bipartite graph G from historical data, the system adds each unique user query to the query log and query the log The system adds each unique listing to the list log. and list the logs The system generates a set of query-list pairs. to collate the query log and listing log and count the occurrences of clicks, add to carts, and purchases for each query-listing pair. The system will connect the two nodes in the pair with edge e i，j Add it to the edge set E and associate the edge with the corresponding weight value w of the edge i，j Typically, a higher weight value relates to a higher likelihood that a user is interested in a list node provided in response to a particular user query.

[0070] The weight value W between two nodes i and j can be calculated based on implicit user feedback information i，j For example, the weight value w i，j Can be based on query-list pairs The weighted sum of the total occurrences of clicks, add to carts, and purchases. Used to determine the weight value w i，j An example equation for is as follows:

[0071] w i，j =C 1 |click i，j |+C 2 |cart i，j |+C 3 |purchases i，j | Equation (1).

[0072] In equation (1), coefficients C1, C2, and C3 are predefined weights for different user interactions. To bias listings where purchases outnumber clicks or shopping carts, the system may typically have a C2 coefficient that satisfies C3. 1 <C 2 <C 3 Any suitable value of .

[0073] In some cases, the weight values ​​w obtained from historical data are i，j Can be used as initial values ​​for edges in computational graphs. These weight values ​​can be updated by training various reinforcement learning models on the training data.

[0074] The system can generate training data based on historical data, or receive historical data as training samples to train a reinforcement learning model. For example, the system can collect historical data within a previous time period (e.g., a month, a year, or other suitable time period). The historical data may include records of user queries and lists of items that have been clicked, added to a shopping cart, or purchased by a user. The user query is represented by the query text, and the list is represented by a unique ID (e.g., a store ID) and an item title (e.g., a tag) belonging to the unique ID. For example, the record may include more than 100 million unique lists, more than 100 million unique user queries, 60 million tags, and 3 million unique store IDs. The record may also include more than 1 billion query-list interactions, of which 3.4% were purchased, 6.2% were added to the cart, and 90.2% were clicked. The corresponding computational graph representing the historical data and the record may include more than 1 billion edges.

[0075] Before training the reinforcement learning model, the system further obtains data defining a reward function. The report function r(s, a) is generally a function that receives (i) the current state of the agent in the environment (i.e., s in the reward function) and (ii) the action determined by the policy to be executed by the agent in the environment (i.e., a in the reward function), and generates a reward for the agent that executes the determined action in the current state. The reward function can be specifically designed based on different requirements for training reinforcement learning models. For example, in the context of a directed graph, the current node in the directed graph is considered to be the current state, and the directed edges from the multiple directed edges of the current node are considered to be actions. The system can select an action to be performed by selecting a directed edge from the current node to traverse in the directed graph. When the selected edge is taken at the current node, the system obtains a reward based on the reward function, and after traversing along the selected directed edge, the system reaches the next state at the subsequent node.

[0076] The system also initializes a value function (506) of the reinforcement learning model. The value function may include an action-value function Q(s, a) that receives (i) the current state s of the agent and (ii) an action a generated by the policy and to be performed by the agent, and determines the expected reward of the agent after taking the action in the current state. In other words, the action-value function is configured to generate a corresponding value for each pair of a current node and a corresponding directed edge starting from the current node. In some implementations, the value function may include a state-value function V(s) that receives the current state of the agent to determine the expected reward after the agent takes the action following the policy. Typically, the system may initialize the state-value function and / or the action-value function to zero.

[0077] The expected reward may include an expectation of a weighted reward for each state in a sequence of states following the current state. For example, in the context of a directed graph, the sequence of states may include a sequence of nodes following the current node and connected by corresponding directed edges that form a trajectory. The trajectory ends at an end node in the directed graph. The expected reward may include a weighted sum of (i) a direct reward to the node that adopts the selected directed edge and (ii) a discounted reward for the reward obtained at the subsequent node by selecting the corresponding directed edge (e.g., the state value of the node immediately following the current node). The discounted reward is generated using a discount factor, which can be specifically designed according to different training requirements. For simplicity, the value function in the following description refers to the state value function by default. In conjunction with Figure 7 Describes an example of an initialization value function.

[0078] Referring back to training the reinforcement learning model, after receiving the computational graph, the system determines a value function (508) of the reinforcement learning model for the computational graph for each node that is not an end node of the plurality of nodes. More specifically, in order to train the reinforcement learning model, the system typically attempts to repeatedly maximize the action-value function Q(s, a) for all states until the value function V(s) converges. The system may generate a transfer function based on the action-value function Q(s, a). Within the framework of the computational graph, states refer to different nodes (e.g., user query nodes), and actions refer to different edges connecting two adjacent nodes. Each edge may be assigned a weight value that may be related to an immediate reward or action value for transferring along the edge at the node. As described above, each edge in the computational graph may be assigned a corresponding initial weight value determined from historical data. The initial weight value may represent the initial possibility of transferring from the corresponding node along the corresponding edge. In combination Figure 7 Describes the details of determining the value function when training a reinforcement learning model.

[0079] Once the value function is determined, the system stores the value function for use in processing user queries associated with the computation graph (510). In some cases, the system generates a policy model (e.g., a transfer function from a node to another node along an edge starting from a node) from the determined value function (or action value function). The system can store the policy function in a memory unit of the switching platform. The stored value function or policy function can be used later to process user queries associated with the computation graph.

[0080] In some implementations, the system may store the value function in a memory device of the system. The system may provide the stored value function as an offline reinforcement learning model for later processing of user queries. In addition, the training process of the reinforcement learning model may be performed offline, and the determined value function or policy function may be deployed for online processing of user queries.

[0081] Figure 6 is a flow chart of an example process 600 for processing a user query using a trained reinforcement learning model. For convenience, process 600 is described as being performed by a system of one or more computers located in one or more locations. For example, a system (e.g., Figure 1 Exchange platform 110, Figure 1 Search engine 118 or Figure 1 The reinforcement learning engine 122) can perform process 600 when appropriately programmed.

[0082] To process the user query, the system performs reasoning operations of the trained reinforcement learning model. For example, the system can retrieve data representing the determined value function or policy function from the memory, and use the computational graph to process the user query based on the determined value function or policy function.

[0083] The system receives a user query from a user device (602). The user query corresponds to an initial node in a plurality of initial nodes in a directed graph stored in an exchange platform. The user query may include text information, graphic information, or other suitable information indicating the interests of the corresponding user. The system may set the user query as an initial node in a computational graph. In some cases, the system may select an existing user query node as a node for the user query. Alternatively, the system may generate a new user query node for the user query after determining a location for the new query node. To determine the location, the system may generate an embedding of information included in the user query and determine the location based on the embedding. Once the new user query node is added to the computational graph, the system may use various techniques to determine the connectivity of the user query node to other existing nodes. For example, the system may connect the new user query node to all possible existing nodes or nodes in a cluster containing the location of the new user query node. (Alternatively, the system may determine a similarity level between the user query and the existing user query node and determine the location based on the similarity level. And if the new query and the existing user query node share a similarity level above a threshold level, the system may assign the new query to the existing user query node.

[0084] After receiving the user query, the system generates a specific trajectory in the computational graph using the value function (or policy function) of the trained reinforcement learning model (604). The specific trajectory includes multiple nodes and multiple edges connecting these multiple nodes. The specific trajectory starts from the user query and ends at an end node in the computational graph. For example, the system can generate a transfer that identifies edges between adjacent nodes based on the value function (or policy function). In some cases, the value function can specify multiple candidate trajectories starting from a new user query node, each candidate trajectory having a corresponding cumulative reward. The system can determine the trajectory by selecting one of the multiple candidate trajectories as the trajectory based on the cumulative reward. For example, the system can select a trajectory with the maximum cumulative reward among all candidate trajectories. In some cases, the system can select more than one trajectory for a single user query.

[0085] In some cases, the system may determine the trajectory at the node level. For example, for the first step, the system selects an edge for the first node from multiple edges at the first node, and transfers from the first node to the second node using the selected edge. Then, for the second step, the system selects another edge for the second node from multiple edges at the second node, and transfers from the second node to the third node using the selected other edge. The system may repeatedly select edges and transfer from the node using the selected edges until the end node is reached. At each step, the system may select edges for the node from a global perspective. For example, the system may determine the value function for the node and select the edge that maximizes the value function. Alternatively or additionally, in some cases, the system may select edges based on local immediate rewards (e.g., r(s, a) of a reinforcement learning model). For example, the system may determine the immediate rewards r(s, a) transferred from different edges at the node, and select the edge with the maximum immediate reward.

[0086] In some cases, the trajectory length (i.e., the number of nodes and directed edges included in the trajectory) can be determined using different methods. In some implementations, the trajectory length can be preset based on the number of directed edges. For example, the system can preset the number of directed edges to 2, 3, 5, 10 or more to generate multiple directed edges. In the case where a random walk algorithm is used, the system can perform multiple random walks with a preset trajectory length. For the case where the end node is not known to the system, the trajectory length is not fixed in advance, but is dynamically determined after reaching the end node. By counting the number of edges from the initial node to the end node, the trajectory length can be determined in real time.

[0087] The system provides data associated with the track to the user device in response to the user query (606). For example, the system can provide data associated with end nodes in the track to the user device. The end nodes can include a list of items or information related to a store and labels of items belonging to the store. In some cases, the system can provide data associated with additional nodes in the track to the user device. For situations where the system selects more than one track, the system can provide multiple end nodes from these tracks to the user device. Once the data is received, the user device can display the data (e.g., one or more lists of items) on a display.

[0088] Figure 7 is a flow chart of an example process 700 for determining a value function when training a reinforcement learning model. For convenience, process 700 is described as being performed by a system of one or more computers located in one or more locations. For example, a system (e.g., Figure 1 Exchange platform 110, Figure 1 Search engine 118 or Figure 1The reinforcement learning engine 122) can perform process 700 when appropriately programmed.

[0089] The techniques described in this specification implement two different reinforcement learning models: a value iteration method and a Monte Carlo value function estimation method for processing user queries in an exchange platform. Therefore, before entering into the details of the process 700 for determining a value function when training a machine learning model, the two different reinforcement learning models are described immediately below.

[0090] Value iteration methods typically optimize an objective function (e.g., a reward function) by iteratively updating the action-value function of all possible actions in each state, and iteratively updating the state-value function of that state based on the updated action-value function. The system determines the value function after determining that the state-value function converges.

[0091] On the other hand, the Monte Carlo value function method typically uses random walks to optimize the objective function. The system samples a random walk (e.g., a directed edge randomly selected based on a probability distribution) of the current state (e.g., the current node) and scores the random walk using a scoring function. Based on the scoring function, the Monte Carlo value function method can define contextual information based on the historical interactions of the nodes in the directed graph. For example, historical interactions may include previous positive interactions associated with the end node for the user query. Historical interactions implicitly include user feedback information, which can be integrated into the random walk sampling technique implemented with the Monte Carlo value function method. Compared with other random walks for user queries, the scores of random walks along the trajectory that terminates at the end node with previous positive interactions are "biased" (i.e., have a higher score). The score is typically related to the reward associated with the sampled random walk, and ultimately to the weight value associated with the corresponding edge in the computational graph.

[0092] To determine the value function when training a reinforcement learning model, the system first initializes the value of the value function 702. In some cases, the system can initialize the value function with a predetermined value, a value determined using a probability distribution (uniform, Gaussian, or other suitable distribution), or a value determined using historical data.

[0093] For the case where the Monte Carlo value function method is implemented, the system can initialize the value function based on the initial edge weight values. As described above, the initial edge weight value w i，j The determination may be based on a weighted sum of occurrences of user interactions with the output listing in response to the user query. The user interactions may include clicks, add to cart, and purchases of the output listing.

[0094] After initializing the value function, for each state of the epoch (e.g., the entire sequence from the initial node to the end node), the system updates the corresponding action-value function based on (i) the corresponding directed edges starting from the node and connecting the subsequent nodes and (ii) the value associated with the reward function. The corresponding directed edges include all possible directed edges starting from the node. The value associated with the reward function can be, for example, an immediate reward for taking the corresponding directed edge in the state, or as another example, the value associated with the reward function can be the state value for the state, for example, the expected return from the current state.

[0095] For the value iteration method, the system first updates the value function (i.e., the action-value function) by updating the expectation of the return value of the action-value function for each node and each possible directed edge starting from the node. The return value is a weighted sum of (i) the immediate reward after obtaining a specific directed edge at the node and (ii) the state value obtained by the corresponding subsequent node connected in the specific directed edge. The state value is generated by the state-value function and can be discounted by multiplying the state value by a discount factor. The discount factor can be a suitable positive real number less than 1. In order to update the value function, the system first generates an intermediate value by summing the immediate reward and the discounted state value, and generates a weighted sum of all intermediate values ​​of all possible directed edges from the current node. The weight used to sum the intermediate values ​​is associated with a probability distribution that indicates the probability of the current state transferring to the next node along a specific directional edge. After updating the action-value function using all possible directed edges in the state, the system updates the state-value function based on the updated value function.

[0096] For the Monte Carlo value function method, the system determines a plurality of candidate random walks for each node in the directional graph. The system also determines a walk length (e.g., the number of edges in the computation graph or the above-mentioned trajectory length) for each of the plurality of candidate random walks. For example, the plurality of candidate random walks may include one thousand to one million random walks, and the walk length of each of these candidate random walks may include three steps, each step corresponding to an edge in the computation graph. In each step of the candidate random walk, the corresponding node is transferred to a corresponding subsequent node along a different directed edge in the directed graph.

[0097] The system samples random walks from K candidate random walks based on a probability distribution. The probability distribution indicates the likelihood of each random walk transferring from one node to a subsequent node along the corresponding edge. The probability distribution is determined based on the value function of the current node. The sampled random walks in the context of a directed graph are equivalent to the selected directed edges starting from the current node. An example random walk sampling technique is based on the Metropolis-Hastings algorithm, which uses implicit user feedback information. More details of the example random walk sampling technique are described below.

[0098] The system scores the sampled random walks using a scoring function of a reinforcement learning model. The scoring function can be specifically designed to meet different training requirements. For example, the system can determine or obtain data in historical data representing the occurrence of visits to nodes that are traversed to process user queries (e.g., the number of visits to peer nodes in response to user queries), and determine the score of the sampled walk based on the occurrence of the nodes it wanders to and the visits associated with the nodes. In some cases, the scoring function can be determined based on a similarity measure between subsequent nodes and nodes associated with the sampled random walk along a directed edge. For example, the scoring function can be a nonlinear function of the measured similarity, such as a power function.

[0099] In the Monte Carlo value function method, for each state (e.g., node) in the observed random walk, the system updates the action value function by the weighted sum of the state value function of the state and the score determined for the random walk. The weights sum to 1 and range from 0 to 1. The system also updates the score by multiplying by a learning factor. The learning score can range from zero to one, including 0.1, 0.3, 0.5, or other suitable values. The Monte Carlo value function method is related to the per-visit Monte Carlo algorithm.

[0100] For both reinforcement learning algorithms, the system determines whether the state value function for the current state has converged after repeatedly updating the value functions of all nodes in the directed graph. As described above, the state value function can be determined by taking the maximum value from the action value function for the current state.

[0101] In response to determining that the state value function has converged, the system provides an updated value function as the value function. As described above, the system can use the stored value function to process the user query during the inference phase.

[0102] Referring to the Monte Carlo value function method, the system can implement different techniques to sample random walks. One advantageous method is to consider implicit user feedback information in the sampling algorithm to obtain better efficiency and accuracy. An example of a random walk sampling technique can use implicit user feedback information to determine a probability distribution indicating the likelihood of transitioning along a particular edge. As described above, the system can determine the initial weight values ​​w for the edges in the computational graph. i，j .

[0103] The cumulative distribution function can be used to determine the initial edge weight value. For each node n and classification edge E n，* An example cumulative distribution function (with corresponding weights W n，* Arrange in descending order so that W n，* >W n+1，* ) is as follows:

[0104]

[0105] In order to reduce the computational cost for sampling, the system implements the Metropolis-Hastings algorithm to improve the inverse transform sampling. The Metropolis-Hastings algorithm is a Markov chain Monte Carlo method for obtaining a sequence of random samples from a probability distribution that is difficult to sample directly. The system can be based on conditional probabilities. Further sampling edge transfer.

[0106] Additionally, as a faster method, the system implements a breadth-first search (BFS) algorithm instead of a depth-first search (DFS) algorithm for sampling random walks. Although the BFS algorithm requires slightly more memory, the BFS algorithm has better cache consistency than the DFS algorithm because it has fewer cache misses. An example sampling algorithm is described below.

[0107] The example algorithm receives as input a starting node n, a predetermined number of walks c, a predetermined walk length k, a set of edges E, a weight value W in a computation graph determined based on historical data, and a multiplier m. The example algorithm also receives global variables such as variance, a node dictionary, and a set of count values.

[0108] The system implementing the example algorithm first selects the i-th node in a sequence of nodes sorted in a predetermined order. To implement it, the system selects a specific edge E from the edge set E according to a uniform distribution n，* , and use binary search to search for nodes.

[0109] For each walk in the total number of walks, the system performs the Metropolis-Hastings algorithm to sample the i-th node in the sequence of nodes sorted in a predetermined order based on the i-th node and the global variance. The edge connecting the i-th node to the j-th node is the edge selected for the random walk. The system then repeatedly performs the Metropolis-Hastings algorithm on the j-th node. In addition, the system can recursively perform the algorithm on the same node sampled with different edges. In this way, the system can efficiently sample random walks and store all possible trajectories starting from the i-th node in a data structure. In some implementations, the system can rearrange the stored random walks according to a predetermined order (e.g., non-increasing order).

[0110] For the Monte Carlo value function method that implements the Metropolis-Hastings algorithm, the system repeatedly updates the weight value w when training the reinforcement learning model. i，j The updated weight values ​​are then used to update the probability distribution of random walk sampling using the Metropolis–Hastings algorithm.

[0111] Once the reinforcement learning model is trained using the Monte Carlo value function method, the system can process user queries by generating random walks on query nodes using the trained Metropolis-Hastings algorithm, and scoring the random walks using a specific scoring algorithm. Since the trained random walk sampling algorithm implicitly reflects the value function of the Monte Carlo value function, the system can use random walks to process user queries. More specifically, the system samples random walks using the above algorithm to predict trajectories from an initial node corresponding to the user query to a terminal node associated with data corresponding to the user query.

[0112] Figure 8 800, 850, which may be used as a client or as a server or multiple servers or in a cloud computing environment to implement the systems and methods described in this document. Computing device 800 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 850 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, smart watches, head-mounted devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions are meant to be exemplary only and are not meant to limit the implementations described and / or claimed in this document.

[0113] The computing device 800 includes a processor 802, a memory 804, a storage device 806, a high-speed interface 808 connected to the memory 804 and a high-speed expansion port 810, and a low-speed interface 812 connected to a low-speed bus 814 and the storage device 806. Each of the components 802, 804, 806, 808, 810 and 812 is interconnected using various buses and can be installed on a common motherboard or in other appropriate ways. The processor 802 can process instructions for execution within the computing device 800, including instructions stored in the memory 804 or on the storage device 806, to display graphical information of a GUI on an external input / output device (e.g., a display 816 coupled to the high-speed interface 808). In other implementations, multiple processors and / or multiple buses, as well as multiple memories and multiple types of memories, can be used appropriately. In addition, multiple computing devices 800 can be connected, each of which provides a portion of the necessary operations (e.g., as a server group, a group of blade servers, or a multi-processor system).

[0114] The memory 804 stores information within the computing device 800. In one implementation, the memory 804 is a computer-readable medium. In one implementation, the memory 804 is one or more volatile memory units. In another implementation, the memory 804 is one or more non-volatile memory units.

[0115] The storage device 806 can provide mass storage for the computing device 800. In one implementation, the storage device 806 is a computer-readable medium. In various implementations, the storage device 806 can be a hard disk device, an optical disk device, a magnetic tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 804, the storage device 806, or a memory on the processor 802.

[0116] The high-speed controller 808 manages bandwidth-intensive operations for the computing device 800, while the low-speed controller 812 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In one implementation, the high-speed controller 808 is coupled to the memory 804, the display 816 (e.g., through a graphics processor or accelerator), and is coupled to the high-speed expansion port 810, which can accept various expansion cards (not shown). In this implementation, the low-speed controller 812 is coupled to the storage device 806 and the low-speed expansion port 814. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device such as a switch or a router, through, for example, a network adapter.

[0117] As shown, computing device 800 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 820, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 824. In addition, it can be implemented in a personal computer such as a laptop computer 822. Alternatively, components from computing device 800 can be combined with other components in a mobile device (not shown) such as device 850. Each of such devices can contain one or more computing devices 800, 850, and the entire system can be composed of multiple computing devices 800, 850 that communicate with each other.

[0118] The computing device 850 includes a processor 852, a memory 864, an input / output device such as a display 854, a communication interface 866, and a transceiver 868, and other components. The device 850 may also be provided with a storage device (e.g., a microdrive or other device) to provide additional storage. Each of the components 850, 852, 864, 854, 866, and 868 are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other suitable ways.

[0119] The processor 852 can process instructions for execution within the computing device 850, including instructions stored in the memory 864. The processor can also include independent analog and digital processors. For example, the processor can provide coordination of other components of the device 850, such as control of a user interface, applications executed by the device 850, and wireless communications performed by the device 850.

[0120] The processor 852 can communicate with the user through a control interface 858 and a display interface 856 coupled to the display 854. The display 854 can be, for example, a TFT LCD display or an OLED display, or other suitable display technology. The display interface 856 may include appropriate circuit modules for driving the display 854 to present graphics and other information to the user. The control interface 858 can receive commands from the user and convert them to submit to the processor 852. In addition, an external interface 862 that communicates with the processor 852 can be provided to enable close range communication of the device 850 with other devices. The external interface 862 can provide, for example, wired communication (e.g., via a docking process) or wireless communication (e.g., via Bluetooth or other such technologies).

[0121] The memory 864 stores information within the computing device 850. In one implementation, the memory 864 is a computer-readable medium. In one implementation, the memory 864 is one or more volatile storage units. In another implementation, the memory 864 is one or more non-volatile storage units. An expansion memory 874 may also be provided and connected to the device 850 via an expansion interface 872, which may include, for example, a SIMM card interface. Such an expansion memory 874 may provide additional storage space for the device 850, or may also store applications or other information for the device 850. Specifically, the expansion memory 874 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Therefore, for example, the expansion memory 874 may be provided as a security module for the device 850, and may be programmed with instructions that allow the secure use of the device 850. In addition, security applications and additional information may be provided via a SIMM card, such as placing identification information on the SIMM card in an unbreakable manner.

[0122] As described below, the memory may include, for example, flash memory and / or MRAM memory. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as the memory 864, the expansion memory 874, or a memory on the processor 852.

[0123] The device 850 can communicate wirelessly via a communication interface 866, which may include digital signal processing circuitry, if necessary. The communication interface 866 may provide communications in various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messages, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS. Such communications may be performed, for example, via a radio frequency transceiver 868. In addition, short-range communications may be performed, for example, using Bluetooth, WiFi, or other such transceivers (not shown). In addition, a GPS receiver module 870 may provide additional wireless data to the device 850, which may be used appropriately by applications running on the device 850.

[0124] Device 850 may also communicate audibly using audio codec 860, which may receive verbal information from a user and convert it into usable digital information. Audio codec 860 may also generate audible sounds for the user, such as through a speaker, such as in a handset of device 850. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on device 850.

[0125] As shown, computing device 850 can be implemented in a variety of different forms. For example, it can be implemented as a cellular phone 880. It can also be implemented as a part of a smart phone 882, a personal digital assistant, or another similar mobile device.

[0126] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit modules, integrated circuit modules, specially designed ASICs, computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system comprising at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system, and to transmit data and instructions to the storage system.

[0127] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any computer program product, device and / or apparatus, such as a disk, optical disk, memory, programmable logic device (PLD) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, voice, or tactile input.

[0129] The systems and techniques described herein can be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser, through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication of any form or medium, such as a communication network. Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0130] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between a client and a server arises from computer programs running on the respective computers and having a client-server relationship to each other.

[0131] As used in this specification, the term "module" is intended to include, but is not limited to, one or more computers configured to execute one or more software programs that include program code that causes the processing unit(s) / device(s) of the computer to perform one or more functions. The term "computer" is intended to include any data processing or computing device / system, such as a desktop computer, laptop computer, mainframe computer, personal digital assistant, server, handheld device, smart phone, tablet computer, e-reader, or any other electronic device capable of processing data.

[0132] Many embodiments have been described. However, it will be appreciated that various modifications may be made without departing from the spirit and scope of the invention. Therefore, other embodiments are within the scope of the following claims. Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of what is claimed, but rather as descriptions of features that are unique to particular embodiments. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment.

[0133] In addition to the above embodiments, the following embodiments are also innovative:

[0134] Embodiment 1 is a method for training a reinforcement learning model, comprising: receiving data representing a computational graph, wherein the computational graph comprises a plurality of nodes and a plurality of directed edges, wherein each directed edge connects two adjacent nodes, wherein the plurality of nodes comprises (i) one or more initial nodes, each initial node corresponding to a user query and having no preceding node, and (i) one or more terminal nodes, each terminal node being associated with data corresponding to the user query and having no subsequent node; receiving training data comprising a reward function for a reinforcement learning model; initializing a value function of the reinforcement learning model, wherein the value function is configured to be specific to the node and generating a corresponding value for each pair of corresponding directed edges starting from the node and terminating at corresponding end nodes in the computational graph; for each node that is not an end node among the plurality of nodes, determining a value function of the reinforcement learning model for the computational graph (including: updating the value function of the reinforcement learning model based on (i) corresponding directed edges starting from the node and connecting subsequent nodes and (ii) the reward function; determining that the updated value function converges; and in response to determining that the updated value function converges, providing the updated value function as the value function); and using the computational graph to process the user query based on the determined value function.

[0135] Embodiment 2 is a method of embodiment 1, wherein processing a user query based on the determined value function includes: receiving a user query corresponding to an initial node in the computational graph from a user device; in response to receiving the user query, using the value function of the trained reinforcement learning model, generating a specific trajectory in the computational graph for the user query, the trajectory including multiple nodes and starting from the initial node in the computational graph to an end node; and providing data associated with one or more nodes including the end node in the specific trajectory corresponding to the user query to the user device.

[0136] Embodiment 3 is the method of embodiment 1 or 2, wherein updating the value function of the reinforcement learning model for the node further includes: using the reward function to generate (1) an immediate reward for the node and the corresponding directed edge of the node, and (2) a discounted reward for the subsequent node in the corresponding directed edge; and updating the value function based on the immediate reward and the discounted reward.

[0137] Embodiment 4 is a method of any one of embodiments 1-3, further comprising determining the corresponding directed edge starting from the node and connecting subsequent nodes, wherein the determination comprises: determining multiple candidate random walks for the node, wherein each candidate random walk goes from the node to the corresponding subsequent node along a different directed edge in the computational graph; sampling a random walk from the multiple candidate random walks for the node based on the value function; and determining the edge as the corresponding directed edge based on the sampled random walk.

[0138] Embodiment 5 is the method of embodiment 4, wherein updating the value function of the reinforcement learning model further comprises: generating a score for the sampled random walk using the scoring function of the reinforcement learning model; generating a discounted reward for the node in the corresponding directed edge using the reward function; and updating the value function based on a weighted sum of the score and the discounted reward.

[0139] Embodiment 6 is the method of embodiment 5, wherein the scoring function is based on a measure of similarity between the corresponding subsequent node associated with the sampled random walk and the node.

[0140] Embodiment 7 is a method of any one of embodiments 2-6, wherein determining the specific trajectory includes determining a sequence of directed edges in the specific trajectory, wherein each directed edge in the sequence is determined by an operation, the operation including: for each node that is not the end node in the specific trajectory: determining the corresponding maximum value generated by the value function for the node; and based on the maximum value, selecting an edge from multiple directed edges starting from the node as the directed edge.

[0141] Embodiment 8 is a system comprising one or more computers and one or more storage devices storing instructions, wherein the instructions, when executed by the one or more computers, are operable to cause the one or more computers to perform the method of any one of embodiments 1-7.

[0142] Embodiment 9 is a computer storage medium encoded with a computer program, wherein the program includes instructions that, when executed by a data processing device, are operable to cause the data processing device to perform the method of any one of embodiments 1 to 7.

[0143] Embodiment 10 is a method for processing user queries, comprising: receiving data representing a computational graph, wherein the computational graph comprises a plurality of nodes and a plurality of directed edges, wherein each directed edge connects two adjacent nodes, wherein the plurality of nodes comprises (i) one or more initial nodes, each initial node corresponding to a user query, and (i) one or more end nodes, each end node being associated with data corresponding to the user query; receiving data representing a specific user query; based on weights assigned to the plurality of directed edges, processing the specific user query using a specific Markov chain Monte Carlo algorithm to generate output data in response to the specific user query, wherein the weights are determined by training the specific Markov chain Monte Carlo algorithm using reinforcement learning; and providing the output data in response to the specific user query so as to be displayed on a user device.

[0144] Embodiment 11 is the method of embodiment 10, wherein processing the specific user query includes: generating a trajectory including a sequence of edges in the multiple edges by repeatedly sampling a random walk from a current node to a subsequent node, wherein the sampling is performed according to a probability distribution generated based on at least a subset of the weights assigned to the multiple directed edges.

[0145] Embodiment 12 is the method of embodiment 10 or 11, wherein using reinforcement learning to train the specific Markov chain Monte Carlo algorithm includes: determining initial weights for the multiple directed edges based on user feedback information for historical output data provided for historical user queries, and updating the weights based on the initial weights by updating the value function.

[0146] Embodiment 13 is the method of embodiment 12, wherein determining the initial weights for the multiple directed edges based on the user feedback information includes: for each of the multiple edges, determining a weighted sum of the user feedback information, wherein the user feedback information includes the occurrence of a user interaction type with the historical output data provided for historical user queries, and wherein the user interaction type includes clicking, adding to a shopping cart, or purchasing a list of items represented in the historical output data.

[0147] Embodiment 14 is the method of embodiment 12 or 13, wherein updating the value function comprises: receiving training data including a reward function; initializing the value function, wherein the value function is configured to generate a corresponding value for each pair of a node and a corresponding directed edge starting from the node and terminating at a corresponding end node in the computational graph; and for each node that is not an end node among the multiple nodes, repeatedly updating the value function for the computational graph based on (i) the corresponding directed edge starting from the node and connecting subsequent nodes and (ii) the reward function until it is determined that the value function converges.

[0148] Embodiment 15 is the method of embodiment 14, wherein updating the value function for the node further comprises: generating a score for the edge sampled by the Markov Chain Monte Carlo algorithm from a scoring function; generating a discounted reward for the node in the sampled edge using the reward function; and updating the value function based on a weighted sum of the score and the discounted reward.

[0149] Embodiment 16 is the method of any one of embodiments 10-15, wherein the Markov Chain Monte Carlo algorithm is the Metropolis-Hastings algorithm.

[0150] Embodiment 17 is a system comprising one or more computers and one or more storage devices storing instructions, wherein the instructions, when executed by the one or more computers, are operable to cause the one or more computers to perform the method of any one of embodiments 10-16.

[0151] Embodiment 18 is a computer storage medium encoded with a computer program, wherein the program includes instructions that, when executed by a data processing device, are operable to cause the data processing device to perform the method of any one of embodiments 10 to 16.

[0152] Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as functioning in certain combinations, and even initially claimed as such, one or more features from a claimed combination may in some cases be deleted from that combination, and a claimed combination may be directed to subcombinations or variations of subcombinations.

[0153] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all illustrated operations be performed to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0154] Certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, some of the processes depicted in the accompanying drawings do not necessarily require the particular order shown, or sequential order, to achieve the desired results.

Claims

1. A method for training a reinforcement learning model, comprising: receiving data representing a computational graph, wherein the computational graph comprises a plurality of nodes and a plurality of directed edges, wherein each directed edge connects two adjacent nodes, wherein the plurality of nodes comprises (i) one or more initial nodes, each initial node corresponding to a user query and not having any preceding node, and (i) one or more terminal nodes, each terminal node being associated with data corresponding to the user query and not having any subsequent node; receiving training data including a reward function for a reinforcement learning model; Initializing a value function of the reinforcement learning model, wherein the value function is configured to generate a respective value for each pair of the node and a respective directed edge starting from the node and terminating at a corresponding end node in the computational graph; For each node that is not an end node of the plurality of nodes, determining a value function of the reinforcement learning model for the computational graph includes: updating the value function of the reinforcement learning model based on (i) corresponding directed edges starting from the node and connecting subsequent nodes and (ii) the reward function; Determining that the updated value function converges; and In response to determining that the updated value function converges, providing the updated value function as the value function; and The user query is processed using the computational graph based on the determined cost function.

2. The method of claim 1, wherein: Processing the user query based on the determined value function includes: receiving, from a user device, a user query corresponding to an initial node in the computational graph; In response to receiving the user query, using the value function of the trained reinforcement learning model, generating a specific trajectory in the computation graph for the user query, the trajectory comprising a plurality of nodes and starting from an initial node in the computation graph to an end node; and Data associated with one or more nodes including the end node in the particular trace corresponding to the user query is provided to the user device.

3. The method of claim 1, wherein: Updating the value function of the reinforcement learning model for the node further includes: generating, using the reward function, (1) an immediate reward for the node and a corresponding directed edge of the node, and (2) a discounted reward for the subsequent node in the corresponding directed edge; and The value function is updated based on the immediate reward and the discounted reward.

4. The method of claim 1, further comprising determining the corresponding directed edge starting from the node and connecting subsequent nodes, wherein the determining comprises: Determine a plurality of candidate random walks for the node, wherein each candidate random walk travels from the node to a corresponding subsequent node along a different directed edge in the computation graph; sampling a random walk from a plurality of candidate random walks for the node based on the value function; as well as The corresponding directed edges are determined based on the sampled random walks.

5. The method of claim 4, wherein: Updating the value function of the reinforcement learning model for the node further includes: generating a score for the sampled random walk using a scoring function of the reinforcement learning model; generating a discounted reward for the node in the corresponding directed edge using the reward function; and The value function is updated based on a weighted sum of the score and the discounted reward.

6. The method of claim 5, wherein: The scoring function is based on a similarity measure between the corresponding subsequent node associated with the sampled random walk and the node.

7. The method of claim 2, wherein: Determining the specific trajectory includes determining a sequence of directed edges in the specific trajectory, wherein each directed edge in the sequence is determined by an operation comprising: for each node that is not the end node in the specific trajectory: determining a corresponding maximum value generated by the cost function for the node; and Based on the maximum value, an edge is selected from a plurality of directed edges starting from the node as the directed edge.

8. A system comprising: one or more memory devices storing instructions; as well as One or more data processing devices configured to interact with the one or more memory devices and, when executing the instructions, perform operations comprising: receiving data representing a computational graph, wherein the computational graph comprises a plurality of nodes and a plurality of directed edges, wherein each directed edge connects two adjacent nodes, wherein the plurality of nodes comprises (i) one or more initial nodes, each initial node corresponding to a user query and not having any preceding node, and (i) one or more terminal nodes, each terminal node being associated with data corresponding to the user query and not having any subsequent node; receiving training data including a reward function for a reinforcement learning model; Initializing a value function of the reinforcement learning model, wherein the value function is configured to generate a respective value for each pair of the node and a respective directed edge starting from the node and terminating at a corresponding end node in the computational graph; For each node that is not an end node of the plurality of nodes, determining a value function of the reinforcement learning model for the computational graph includes: updating the value function of the reinforcement learning model based on (i) corresponding directed edges starting from the node and connecting subsequent nodes and (ii) the reward function; Determining that the updated value function converges; and In response to determining that the updated value function converges, providing the updated value function as the value function; and The user query is processed using the computational graph based on the determined cost function.

9. The system of claim 8, wherein: Processing the user query based on the determined value function includes: receiving, from a user device, a user query corresponding to an initial node in the computational graph; In response to receiving the user query, using the value function of the trained reinforcement learning model, generating a specific trajectory in the computation graph for the user query, the trajectory comprising a plurality of nodes and starting from an initial node in the computation graph to an end node; and Data associated with one or more nodes including the end node in the particular trace corresponding to the user query is provided to the user device.

10. The system of claim 8, wherein: Updating the value function of the reinforcement learning model for the node further includes: generating, using the reward function, (1) an immediate reward for the node and a corresponding directed edge of the node, and (2) a discounted reward for the subsequent node in the corresponding directed edge; and The value function is updated based on the immediate reward and the discounted reward.

11. The system of claim 8, wherein: The operation further includes: determining the corresponding directed edge starting from the node and connecting the subsequent nodes, wherein the determining includes: Determine a plurality of candidate random walks for the node, wherein each candidate random walk travels from the node to a corresponding subsequent node along a different directed edge in the computation graph; sampling a random walk from a plurality of candidate random walks for the node based on the cost function; and The corresponding directed edges are determined based on the sampled random walks.

12. The system of claim 11, wherein: Updating the value function of the reinforcement learning model for the node further includes: generating a score for the sampled random walk using a scoring function of the reinforcement learning model; generating a discounted reward for the node in the corresponding directed edge using the reward function; and The value function is updated based on a weighted sum of the score and the discounted reward.

13. The system of claim 12, wherein: The scoring function is based on a similarity measure between the corresponding subsequent node associated with the sampled random walk and the node.

14. The system of claim 9, wherein: Determining the specific trajectory includes determining a sequence of directed edges in the specific trajectory, wherein each directed edge in the sequence is determined by an operation comprising: for each node that is not the end node in the specific trajectory: determining a corresponding maximum value generated by the cost function for the node; and Based on the maximum value, an edge is selected from a plurality of directed edges starting from the node as the directed edge.

15. A non-transitory computer-readable medium storing instructions that, when executed by one or more data processing devices, cause the one or more data processing devices to perform operations comprising: receiving data representing a computational graph, wherein the computational graph comprises a plurality of nodes and a plurality of directed edges, wherein each directed edge connects two adjacent nodes, wherein the plurality of nodes comprises (i) one or more initial nodes, each initial node corresponding to a user query and not having any preceding node, and (i) one or more terminal nodes, each terminal node being associated with data corresponding to the user query and not having any subsequent node; receiving training data including a reward function for a reinforcement learning model; Initializing a value function of the reinforcement learning model, wherein the value function is configured to generate a respective value for each pair of the node and a respective directed edge starting from the node and terminating at a corresponding end node in the computational graph; For each node that is not an end node of the plurality of nodes, determining a value function of the reinforcement learning model for the computational graph includes: updating the value function of the reinforcement learning model based on (i) corresponding directed edges starting from the node and connecting subsequent nodes and (ii) the reward function; Determining that the updated value function converges; and In response to determining that the updated value function converges, providing the updated value function as the value function; and The user query is processed using the computational graph based on the determined cost function.

16. The computer-readable medium of claim 15, wherein: Processing the user query based on the determined value function includes: receiving, from a user device, a user query corresponding to an initial node in the computational graph; In response to receiving the user query, using the value function of the trained reinforcement learning model, generating a specific trajectory in the computation graph for the user query, the trajectory comprising a plurality of nodes and starting from an initial node in the computation graph to an end node; and Data associated with one or more nodes including the end node in the particular trace corresponding to the user query is provided to the user device.

17. The computer-readable medium of claim 15, wherein: Updating the value function of the reinforcement learning model for the node further includes: generating, using the reward function, (1) an immediate reward for the node and a corresponding directed edge of the node, and (2) a discounted reward for the subsequent node in the corresponding directed edge; and The value function is updated based on the immediate reward and the discounted reward.

18. The computer-readable medium of claim 15, wherein: The operation further includes: determining the corresponding directed edge starting from the node and connecting the subsequent nodes, wherein the determining includes: Determine a plurality of candidate random walks for the node, wherein each candidate random walk travels from the node to a corresponding subsequent node along a different directed edge in the computation graph; sampling a random walk from a plurality of candidate random walks for the node based on the cost function; and The corresponding directed edges are determined based on the sampled random walks.

19. The computer-readable medium of claim 18, wherein: Updating the value function of the reinforcement learning model for the node further includes: generating a score for the sampled random walk using a scoring function of the reinforcement learning model; generating a discounted reward for the node in the corresponding directed edge using the reward function; and The value function is updated based on a weighted sum of the score and the discounted reward.

20. The computer readable medium of claim 16, wherein: Determining the specific trajectory includes determining a sequence of directed edges in the specific trajectory, wherein each directed edge in the sequence is determined by an operation comprising: for each node that is not the end node in the specific trajectory: determining a corresponding maximum value generated by the cost function for the node; and Based on the maximum value, an edge is selected from a plurality of directed edges starting from the node as the directed edge.

21. A method for processing a user query, comprising: receiving data representing a computational graph, wherein the computational graph comprises a plurality of nodes and a plurality of directed edges, wherein each directed edge connects two adjacent nodes, wherein the plurality of nodes comprises (i) one or more initial nodes, each initial node corresponding to a user query, and (i) one or more end nodes, each end node being associated with data corresponding to the user query; receiving data representing a particular user query; processing the specific user query using a specific Markov Chain Monte Carlo algorithm to generate output data responsive to the specific user query based on weights assigned to the plurality of directed edges, wherein the weights are determined by training the specific Markov Chain Monte Carlo algorithm using reinforcement learning; as well as The output data is provided in response to the particular user query to be displayed on a user device.

22. The method of claim 21, wherein: Processing the specific user query includes: A trajectory including a sequence of edges among the plurality of edges is generated by repeatedly sampling a random walk from a current node to a subsequent node, wherein the sampling is performed according to a probability distribution generated based on at least a subset of the weights assigned to the plurality of directed edges.

23. The method of claim 21, wherein: Using reinforcement learning to train this particular Markov Chain Monte Carlo algorithm involves: determining initial weights for the plurality of directed edges based on user feedback information for historical output data provided for historical user queries, and The weights are updated based on the initial weights by updating a cost function.

24. The method of claim 23, wherein: Determining initial weights for the plurality of directed edges based on the user feedback information includes: For each of the plurality of edges, a weighted sum of the user feedback information is determined, wherein the user feedback information comprises an occurrence of a user interaction type with the historical output data provided for the historical user query, and wherein the user interaction type comprises clicking, adding to a shopping cart, or purchasing a listing of items represented in the historical output data.

25. The method of claim 23, wherein: Updating the value function includes: receiving training data including a reward function; Initializing the cost function, wherein the cost function is configured to generate a respective value for each pair of a node and a respective directed edge starting from the node and terminating at a corresponding end node in the computation graph; and For each node that is not an end node of the plurality of nodes, the value function for the computational graph is repeatedly updated based on (i) a corresponding directed edge starting from the node and connecting subsequent nodes and (ii) the reward function until it is determined that the value function converges.

26. The method of claim 25, wherein: Updating the value function for the node further includes: generating scores for edges sampled by the Markov Chain Monte Carlo algorithm from a scoring function; generating a discounted reward for the node in the sampled edge using the reward function; and The value function is updated based on a weighted sum of the score and the discounted reward.

27. The method of claim 21, wherein: The Markov Chain Monte Carlo algorithm is the Metropolis-Hastings algorithm.

28. A system comprising: one or more memory devices storing instructions; as well as One or more data processing devices configured to interact with one or more memory devices and, when executing the instructions, perform operations comprising: receiving data representing a computational graph, wherein the computational graph comprises a plurality of nodes and a plurality of directed edges, wherein each directed edge connects two adjacent nodes, wherein the plurality of nodes comprises (i) one or more initial nodes, each initial node corresponding to a user query, and (i) one or more end nodes, each end node being associated with data corresponding to the user query; receiving data representing a particular user query; processing the specific user query using a specific Markov Chain Monte Carlo algorithm to generate output data responsive to the specific user query based on weights assigned to the plurality of directed edges, wherein the weights are determined by training the specific Markov Chain Monte Carlo algorithm using reinforcement learning; and The output data is provided in response to the particular user query to be displayed on a user device.

29. The system of claim 28, wherein: Processing the specific user query includes: A trajectory including a sequence of edges among the plurality of edges is generated by repeatedly sampling a random walk from a current node to a subsequent node, wherein the sampling is performed according to a probability distribution generated based on at least a subset of the weights assigned to the plurality of directed edges.

30. The system of claim 28, wherein: Using reinforcement learning to train this particular Markov Chain Monte Carlo algorithm involves: determining initial weights for the plurality of directed edges based on user feedback information for historical output data provided for historical user queries, and The weights are updated based on the initial weights by updating a cost function.

31. The system of claim 30, wherein: Determining initial weights for the plurality of directed edges based on the user feedback information includes: For each of the plurality of edges, a weighted sum of the user feedback information is determined, wherein the user feedback information comprises an occurrence of a user interaction type with the historical output data provided for the historical user query, and wherein the user interaction type comprises clicking, adding to a shopping cart, or purchasing a listing of items represented in the historical output data.

32. The system of claim 30, wherein: Updating the value function includes: receiving training data including a reward function; Initializing the cost function, wherein the cost function is configured to generate a respective value for each pair of a node and a respective directed edge starting from the node and terminating at a corresponding end node in the computation graph; and For each node that is not an end node of the plurality of nodes, the value function for the computational graph is repeatedly updated based on (i) a corresponding directed edge starting from the node and connecting subsequent nodes and (ii) the reward function until it is determined that the value function converges.

33. The system of claim 32, wherein: Updating the value function for the node further includes: generating scores for edges sampled by the Markov Chain Monte Carlo algorithm from a scoring function; generating a discounted reward for the node in the sampled edge using the reward function; and The value function is updated based on a weighted sum of the score and the discounted reward.

34. The system of claim 28, wherein: The Markov Chain Monte Carlo algorithm is the Metropolis-Hastings algorithm.

35. A non-transitory computer-readable medium storing instructions that, when executed by one or more data processing devices, cause the one or more data processing devices to perform operations comprising: receiving data representing a computational graph, wherein the computational graph comprises a plurality of nodes and a plurality of directed edges, wherein each directed edge connects two adjacent nodes, wherein the plurality of nodes comprises (i) one or more initial nodes, each initial node corresponding to a user query, and (i) one or more end nodes, each end node being associated with data corresponding to the user query; receiving data representing a particular user query; processing the specific user query using a specific Markov Chain Monte Carlo algorithm to generate output data responsive to the specific user query based on weights assigned to the plurality of directed edges, wherein the weights are determined by training the specific Markov Chain Monte Carlo algorithm using reinforcement learning; as well as The output data is provided in response to the particular user query to be displayed on a user device.

36. The computer readable medium of claim 35, wherein: Processing the specific user query includes: A trajectory including a sequence of edges among the plurality of edges is generated by repeatedly sampling a random walk from a current node to a subsequent node, wherein the sampling is performed according to a probability distribution generated based on at least a subset of the weights assigned to the plurality of directed edges.

37. The computer readable medium of claim 35, wherein: Using reinforcement learning to train this particular Markov Chain Monte Carlo algorithm involves: determining initial weights for the plurality of directed edges based on user feedback information for historical output data provided for historical user queries, and The weights are updated based on the initial weights by updating a cost function.

38. The computer readable medium of claim 37, wherein: Determining initial weights for the plurality of directed edges based on the user feedback information includes: For each of the plurality of edges, a weighted sum of the user feedback information is determined, wherein the user feedback information comprises an occurrence of a user interaction type with the historical output data provided for the historical user query, and wherein the user interaction type comprises clicking, adding to a shopping cart, or purchasing a listing of items represented in the historical output data.

39. The computer readable medium of claim 37, wherein: Updating the value function includes: receiving training data including a reward function; Initializing the cost function, wherein the cost function is configured to generate a respective value for each pair of a node and a respective directed edge starting from the node and terminating at a corresponding end node in the computation graph; and For each node that is not an end node of the plurality of nodes, the value function for the computational graph is repeatedly updated based on (i) a corresponding directed edge starting from the node and connecting subsequent nodes and (ii) the reward function until it is determined that the value function converges.

40. The computer readable medium of claim 39, wherein: Updating the value function for the node further includes: generating scores for edges sampled by the Markov Chain Monte Carlo algorithm from a scoring function; generating a discounted reward for the node in the sampled edge using the reward function; and The value function is updated based on a weighted sum of the score and the discounted reward.

41. The computer readable medium of claim 35, wherein: The Markov Chain Monte Carlo algorithm is the Metropolis-Hastings algorithm.

Images