Resource navigation using neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for resource navigation using neural networks.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 343,964, filed on May 19, 2022. The disclosure of that prior application is considered part of and incorporated by reference into the disclosure of the present application. Background Art

[0002] This specification relates to using neural networks to perform information retrieval.

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs for received inputs. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from the received input according to the current values ​​of the corresponding set of parameters. Summary of the invention

[0004] This specification describes a system implemented as a computer program on one or more computers in one or more locations that performs an information retrieval task to retrieve information within a collection of resources in response to a received query.

[0005] Embodiments of the system solve the problem of efficiently searching a graph data storage structure for a target in response to a query. A graph is defined by nodes that hold information and links between nodes that define relationships between nodes. Embodiments of the system are able to identify a target by efficiently traversing a path through the nodes via the links to identify a specific information storage location.

[0006] The described techniques can be used to search for any type of information; they do not depend on the kind of data. Targets can be soft targets or approximate targets. No human intervention is required during the search process.

[0007] As an example, the system can retrieve information from a collection of one or more websites, where each resource represents a web page or a portion of a web page in one or more websites. In this context, efficiently traversing paths through nodes results in reduced communication bandwidth (because fewer web pages need to be retrieved) and fast, i.e., low-latency searches.

[0008] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following additional advantages.

[0009] Embodiments of the described techniques provide a technical advantage of lower memory and computation requirements when searching a graph storage structure. This is achieved by using a navigation policy neural network to process, among other things, an embedding of information at a node to generate a score for outgoing links from that node, which is then used to select one of the links to follow.

[0010] By using a neural network to generate responses to queries by navigating through a collection of resources, the described techniques can generate effective responses to various queries even when traditional information retrieval techniques fail. For example, the described techniques can generate effective responses to queries by searching a collection of resources that may not be effectively searched by an Internet search engine. For example, the collection of resources can be one or more firewalled websites that are inaccessible to an Internet search engine. As another example, the collection of resources can be one or more websites that have not yet been indexed by an Internet search engine (and, therefore, the Internet search engine cannot generate results identifying pages from unindexed websites). As another example, the collection of resources can be one or more websites that are frequently updated and therefore have content that is significantly different from the content when they were last crawled by an Internet search engine. As another example, the described techniques can be used to supplement an Internet search engine or other information retrieval system, for example, by using an information retrieval system to find the correct vicinity of a collection relative to a query, and then using the described techniques to find the desired target starting from the current resource identified by the information retrieval system.

[0011] In general implementations, the described system can process resources including unstructured data, such as text in natural language. Implementations of the system can operate efficiently even when graph patterns change over time.

[0012] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 An example resource navigation system is shown.

[0014] Figure 2 is a flow chart of an example process for performing information retrieval.

[0015] Figure 3 is a flow chart of an example process for performing a navigation iteration.

[0016] Figure 4 The operation of the system at a navigation iteration is shown.

[0017] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0018] Figure 1 An example resource navigation system 100 is shown.

[0019] Resource navigation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.

[0020] The resource navigation system 100 performs information retrieval tasks that require retrieving information within the resource collection 110 in response to the received query 120 .

[0021] For example, an information retrieval task may be fact verification, and the query may indicate a fact that needs to be verified. The goal may then indicate verification of the fact.

[0022] As another example, the information retrieval task can be question answering, and the query can represent a question that requires a response. The goal can then be a response to the question.

[0023] As yet another example, the information retrieval task may be a general information retrieval task, and the system may generate one or more search results as a response, each search result identifying a corresponding resource from the collection 110. More specifically, each search result identifies a resource, includes a link to the resource, and optionally includes a snippet of content from the resource. Then, a target may be defined as information that is a relevant response to the search, i.e., a result.

[0024] In general, resource collection 110 may include any resource that can be represented by a graph, where resources are associated with nodes of the graph, and the nodes, i.e., resources, are connected by links, each link pointing from one resource at one node to another resource at another node. Each of the resources has associated content, which may include any type of data, including but not limited to: text data, such as for natural language text; image or video data, such as pixels defining one or more images or videos; audio data, such as waveforms defining one or more sounds, such as speech or music; or any other type of entity or knowledge.

[0025] For example, resource collection 110 may represent one or more websites, and each resource may represent a web page in one or more websites or a portion of a web page in one or more websites.

[0026] Specifically, the system 100 receives a query 120, for example, from a client device. For example, the query can be a natural language query.

[0027] In some cases, the set of resources 110 to be searched by the system 100 is fixed, ie, the system 100 searches the same set of resources 120 for each query 110 .

[0028] In some other cases, the resource collection 110 may be specified along with the query 120. For example, the system 100 may also receive, in response to the query 120, a URL or other resource locator identifying the resource collection 110 to be searched.

[0029] As a specific example, query 120 may be submitted via an Internet search engine.

[0030] In this example, the one or more websites represented by set 110 may be websites that have not been indexed by an Internet search engine and are therefore not searchable by an Internet search engine.

[0031] As another example, the content of one or more of the web pages in the one or more websites may have changed since the one or more websites have been indexed by the Internet search engine, i.e., such that searching the websites using the Internet search engine may potentially result in outdated or incorrect information being retrieved.

[0032] As another example, one or more websites represented by set 110 may be firewalled or otherwise inaccessible to search engine crawlers of Internet search engines. Therefore, one or more websites cannot be effectively searched by Internet search engines. Please note that this system cannot be used to provide unauthorized access or use of the content of resources from behind any access control mechanism (e.g., firewall), for example, authorized access to the resources of the access control mechanism is still required to determine the content of the resources and identify links between resources. Therefore, the use of access to these resources and any content obtained thereby is intended to be strictly within the boundaries of law and ethics.

[0033] System 100 then retrieves information from resource collection 110 in response to query 120 using query encoder neural network 140 , resource encoder neural network 150 , and navigation strategy neural network 160 .

[0034] The query encoder neural network 140 is a neural network configured to process the query 120 to generate an embedding of the query ("query embedding") 142. For example, the query encoder neural network 140 may be a text encoder neural network, such as a Transformer or a recurrent neural network. As a specific example, the query encoder neural network 140 may include a self-attention neural network that is configured to process text tokens from the query to generate a corresponding token embedding for each of the text tokens. The query encoder neural network 140 may then generate the query embedding 142 by combining (e.g., taking the mean or applying different pooling operations) the token embeddings and then optionally applying a non-linear activation function (e.g., a tanh function) to the combined embeddings.

[0035] As used in this specification, an "embedding" is an ordered set of numerical values ​​having a predetermined dimension. For example, an embedding can be a vector of numerical values ​​(eg, floating point values ​​or other numerical values) having a fixed number of dimensions.

[0036] Resource encoder neural network 150 is a neural network configured to process information characterizing a resource 148 to generate an embedding 152 of the resource (“resource embedding”).

[0037] As a specific example, the resource encoder neural network 150 can be a text encoder neural network, such as a Transformer or a recurrent neural network, that processes the text from the current resource (and optionally the title of the resource) to generate a resource embedding 152. As a specific example, the resource encoder neural network 150 can include a self-attention neural network (a neural network with one or more self-attention layers) that is configured to process text tokens from the resource to generate a corresponding token embedding for each text token. Typically, the self-attention layer is configured to apply an attention mechanism on the attention layer input to generate an attention layer output for each element of the input; there are many possible attention mechanisms. The resource encoder neural network 150 can then generate the resource embedding 152 by combining (e.g., taking the mean or applying different pooling operations) the token embeddings and then optionally applying a non-linear activation function (e.g., a tanh function) to the combined embeddings.

[0038] As another example, the resource encoder neural network 150 can be a multimodal encoder, such as a multimodal Transformer or other multimodal representation generation neural network, that processes both text and images from a resource (and optionally the title of the resource) to generate a resource embedding 152.

[0039] The navigation policy neural network 160 is a neural network configured to receive inputs including at least the resource embedding 152 of the current resource, the embeddings of the candidate resources linked to from the current resource, and the query embedding 142, and generate as output the scores 162 of the candidate resources. As an example, the navigation policy neural network 160 may include a feed-forward neural network.

[0040] The score of a candidate resource generally represents the likelihood that navigating from the current resource to the candidate resource will lead to successful completion of the information retrieval task given the query embedding 120 .

[0041] The following will refer to Figure 2 and Figure 3 The navigation policy neural network 160 is described in more detail.

[0042] More specifically, system 100 uses query encoder neural network 140, resource encoder neural network 150, and navigation strategy neural network 160 to navigate to a final resource within resource set 110, and then generates response 170 characterizing the final resource.

[0043] Reference below Figures 2 to 4 Navigating through a collection of resources 110 using a neural network is described.

[0044] The system 100 may then provide a response 170 , for example, to the client device from which the query 120 was received.

[0045] In some implementations, the response 170 characterizes only the final resource, ie, not other current resources for previous navigation iterations.

[0046] The format of response 170 will generally depend on the information retrieval task being performed by system 100 .

[0047] For example, when the query 120 represents a fact, the response may include content extracted from the final resource that the system 100 has determined to verify the fact.

[0048] As another example, when the query 120 represents a question, the response 170 may include content extracted from the ultimate resource that the system 100 has determined answers the question.

[0049] As another example, for a general search engine query task, the system 100 may generate search results that identify resources, include links to the resources, and optionally include snippets of content from the resources, and provide the search results as part of the response 170 to the query 120 .

[0050] Figure 22 is a flow chart of an example process 200 for generating a response to a received query. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed resource navigation system (e.g., Figure 1 The resource navigation system 100 may perform process 200 .

[0051] The system receives a query (step 202).

[0052] The system processes the query using a query encoder neural network to generate an embedding for the query (step 204).

[0053] The system identifies a current resource in the resource collection (step 206 ).

[0054] For example, the current resource may be the home page of a collection of resources, such as the home page of a website, or a randomly selected resource from the collection.

[0055] As another example, the system may use information retrieval technology to identify resources within the resource collection that are relevant to the query. For example, the information retrieval technology may be a TF-IDF-based technology or any other appropriate information retrieval technology.

[0056] The system then performs a number of navigation iterations to generate a response to the query (step 208 ).

[0057] At a high level, the system updates the current resource at each navigation iteration, ie, "navigates" to another resource linked to by the current resource as of the navigation iteration.

[0058] The following will refer to Figure 3 Performing a navigation iteration is described in more detail.

[0059] After the last navigation iteration, the system generates a response characterizing the current resource after the last navigation iteration (the "final resource" of the iteration of process 200) (step 210).

[0060] In particular, after performing each navigation iteration, the system can determine whether a termination criterion for generating a response has been met. Once the termination criterion has been met, the system determines that the navigation iteration is the last iteration and generates a response using the current resources after the iteration.

[0061] The system may determine that the termination criteria have been met in any of a variety of ways.

[0062] For example, the system may determine that termination criteria are met when a threshold (ie, a predetermined maximum number) of navigation iterations have been performed.

[0063] As another example, the system can determine that the termination criteria are met when the similarity between the embedding of the query and the embedding of the current resource after the last navigation iteration meets a threshold. For example, after each navigation iteration, the system can calculate the dot product between the two embeddings and determine that the navigation iteration should be the last iteration when the dot product is above a threshold.

[0064] The system may then provide a response, for example, to the client device from which the query was received.

[0065] In some implementations, the response only characterizes the current resources after the last navigation iteration, ie, not other current resources for previous navigation iterations.

[0066] Typically, the response may include data identifying the current resource after the last navigation iteration and, in some embodiments, include (i) content from the current resource after the last navigation iteration, (ii) a summary of content from the current resource after the last navigation iteration, or (iii) both.

[0067] The format of the response will typically depend on the information retrieval task being performed by the system.

[0068] For example, when the query represents a fact, the response may include content extracted from a resource that the system has determined to verify the fact. For example, the system may use a neural network (e.g., a neural network that has been trained to perform fact verification) to process the query and multiple text snippets from the resource to generate a respective score for each of the multiple snippets, and then provide the highest scoring text snippet as the extracted content.

[0069] As another example, when the query represents a question, the response may include content extracted from a resource that the system has determined answers the question. For example, the system may use a neural network (e.g., a neural network that has been trained to perform question answering) to process the query and multiple text snippets from the resource to generate a respective score for each of the multiple snippets, and then provide the highest scoring text snippet as the extracted content.

[0070] As another example, for a general search engine query task, the system may generate search results that identify resources, include links to the resources, and optionally include snippets of content from the resources, and provide the search results in response to the query.

[0071] In some implementations, the system may perform multiple iterations of process 200 for each received query.

[0072] That is, the system may identify multiple current resources for each received query, and perform, for each of the identified current resources, a corresponding iteration of process 200. The response generated by the system may then feature multiple different ones of the "final" resources identified by different iterations of process 200, i.e., multiple different ones of the final current resources identified after the last navigation iteration, or only the most relevant extracted content across all final resources.

[0073] In other words, the system can perform the above-described iterations of process 200. The system can also identify one or more other current resources and perform navigation iterations as described above to determine one or more additional final resources.

[0074] The response may then also characterize some or all of the additional final resources in addition to the "final resources" identified above, or may characterize only the most relevant final resources, or may characterize only the most relevant extracted content across all final resources.

[0075] Figure 3 300 is a flow chart of an example process 300 for performing navigation iterations. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a resource navigation system (e.g., Figure 1 The resource navigation system 100 may perform process 300 .

[0076] The system identifies a plurality of outgoing links from a current resource as of a navigation iteration (step 302). Each outgoing link is a link from the current resource to a corresponding other resource in a resource collection.

[0077] In some cases, when the resource collection is one or more websites, each resource represents a different web page. In these cases, each outgoing link is a hyperlink to a web page different from the current web page.

[0078] In some other cases, the system may divide certain web pages into multiple resources. For example, the system may divide each web page that includes more than a threshold amount of text tokens into multiple resources, each resource including less than the threshold amount of text tokens. In these cases, outgoing links may include hyperlinks to other web pages as well as links to other parts of the same web page. For example, for a given part of a given web page, links may include any other web pages linked to by any hyperlinks within the given part and linked to resources representing adjacent parts of the given web page.

[0079] The system processes content from the current resource using the resource encoder neural network to generate an embedding of the current resource (step 304).

[0080] As a specific example, the system may use a text encoder (e.g., a Transformer neural network) to process text from the current resource (and optionally the title of the current web page). As another example, the system may use a multimodal encoder (e.g., a multimodal Transformer or other multimodal representation generation neural network) to process both text and images from the current resource (and optionally the title of the current web page).

[0081] The system generates a corresponding embed for each outgoing link (step 306).

[0082] For example, the system may generate an embedding for a given outgoing link from the content of a resource linked to by the outgoing link, or from text on the current resource corresponding to the link (ie, from the text of the hyperlink), or from both.

[0083] As a specific example, the system may use a resource encoder neural network to process the content of a resource linked to by a given outgoing link, and optionally process text from a current resource corresponding to the link, to generate an embedding for that outgoing link.

[0084] As another specific example, the system can use a resource encoder neural network to generate a corresponding word-meta embedding for each text word in the current resource. The system can then generate an embedding for the resource linked to by the outgoing link from the word-meta embeddings of the words in the text on the current resource corresponding to the link, e.g., by averaging or otherwise pooling these word-meta embeddings, and optionally applying a non-linear activation function to the resulting pooled embeddings.

[0085] In some implementations, the system also uses additional information characterizing the links to generate embeddings, e.g., by including the additional information in any of the inputs to the resource encoder neural network described above.

[0086] For example, the additional information may include a first indicator indicating whether the corresponding other resource has been selected as the current resource at a previous navigation iteration, a second indicator indicating whether the corresponding other resource is another part of the same web page as the current resource or a different web page from the current resource, or both.

[0087] The system generates a corresponding score for each outgoing link using the navigation policy neural network from at least the embedding of the outgoing link, the embedding of the current resource, and the embedding of the query (step 308).

[0088] For example, the system can process navigation input including an embedding of the current resource and an embedding of the query using one or more neural network layers of a navigation policy neural network (e.g., using a multilayer perceptron) to generate a combined embedding. In some cases, to provide additional context, the navigation input can also include the corresponding embedding of the current resource as of any previous navigation iteration.

[0089] For each outgoing link, the system can then determine a corresponding score for the outgoing link based on the similarity between the combined embedding and the embedding of the outgoing link. For example, the system can calculate a dot product or cosine similarity between the combined embedding and the embedding of the outgoing link. As another example, the system can process the combined embedding and the embedding of the outgoing link using one or more additional layers of a neural network that outputs a similarity score.

[0090] Then, the system selects one of the outgoing links based on the corresponding scores of the outgoing links, and updates the current resource to the corresponding other resource linked to by the outgoing link (step 310 ).

[0091] As an example, the system may select the outgoing link with the highest corresponding score.

[0092] As another example, the system may generate a probability distribution including a corresponding probability for each outgoing link from the corresponding scores, such as by applying softmax to the corresponding scores, and sample the outgoing links from the probability distribution.

[0093] After the last navigation interaction, the system generates a response representing the current resource after the last navigation iteration.

[0094] Figure 4 The operation of the system 100 when performing a navigation iteration is shown.

[0095] like Figure 4 , a current resource ("node") 402 for a navigation iteration includes links to a first resource ("node 1") 404 and a second resource ("node 2") 406. The purpose of the navigation iteration is to obtain information that is available from a destination resource ("destination node") 408 that is linked to by the second resource 406 but not by the first resource 404. For example, the destination resource 408 may contain an answer to a query, or may contain statements that verify a fact raised by the query.

[0096] To perform the navigation iteration, the system processes the respective contents 410 (“node text”) of the current resource 402 , the first resource 404 , and the second resource 406 to generate an embedding s t , embedding a1 of the first resource 404 and embedding a2 of the second resource 406 .

[0097] exist Figure 4 In the example of FIG. 4 , because the destination resource 408 is known, the system processes the content of the destination resource 408 to generate an embedded s gWhen the destination resource 408 is known, the system can process the content of the destination resource 408 using the resource encoder neural network to generate an embedding. As described above, at inference time, because the destination resource 408 is typically not known a priori, the system typically uses the embedding of the query instead of the embedding of the destination resource 408.

[0098] The system then generates a score p1 for the first resource 404 and a score p2 for the second resource 406 and uses the scores to select one of the first resource and the second resource to navigate to at the navigation iteration.

[0099] For example, to generate a score for the first resource 404, the system may process s t , a1 and s g To generate a score for the second resource 406, the system may process s t , a2 and s g .

[0100] Thus, because the scores are generated using embeddings that represent not only the content of the first and second resources, but also the content of the current resource and the destination resource (or, as a proxy, the query), the system can efficiently navigate between resources to perform information retrieval tasks.

[0101] The system 100 or another training system may train various neural networks used for navigation in any of a variety of ways.

[0102] In some embodiments, the system jointly trains all of the above neural network components on training data for an information retrieval task that the system is configured to perform. For example, the system can train the neural network through reinforcement learning, where the system receives a reward of one for successfully completing a task (as indicated in the training data) at a given navigation iteration and a reward of zero for not completing the task. As another example, the system can obtain training data that includes trajectories that successfully perform an information retrieval task, and then train the neural network on the trajectories through imitation learning (e.g., using behavioral cloning (BC) or other imitation learning objectives).

[0103] In some other implementations, the system pre-trains some or all of the components on the graph navigation task.

[0104] For example, the system may pre-train a navigation policy neural network on a graph navigation task (e.g., a task where the destination resource is known in advance) and, optionally, pre-train a resource encoder neural network.

[0105] In these cases, the system may use the embedding of the destination resource or the embedding of a portion of the content in the destination resource instead of the above referenced Figure 4Embeddings for the queries shown. For example, the system can use a resource encoder neural network to generate these embeddings.

[0106] When the resource encoder neural network is not pre-trained jointly with the navigation policy neural network, the system can keep the parameters of the resource encoder neural network frozen to the values ​​learned through the unsupervised or semi-supervised representation learning objective.

[0107] Specifically, the system can execute multiple instances of the graph navigation task on the resource set to generate a trajectory set of the graph navigation task. The system can then pre-train a neural network on the trajectory set, for example, by imitation learning or reinforcement learning as described above.

[0108] The system can use any of a variety of graph navigation tasks for pre-training.

[0109] As an example, the system can generate trajectories for a graph navigation task by randomly selecting a starting node (resource) in the set of training resources and then performing a T-step random walk from the starting node, where the end node of the random walk is the destination resource. The system can keep T fixed to a predetermined number throughout pre-training, or sample values ​​of T from a distribution for each trajectory. When training on this task, the system can (e.g., using a resource encoder neural network) provide the entire content of the end node of the random walk or a fragment (e.g., a sentence or other text fragment) from the end node as the content from the destination resource to be embedded. This has the advantage of being simple, i.e., memory and compute efficient, but is (surprisingly) sufficient to train effective navigation policies, such as using behavioral cloning.

[0110] The system can then perform fine-tuning on the training data for the information retrieval task. For example, the system can train the query encoder neural network on the training data for the information retrieval task using reinforcement learning or imitation learning as described above and optionally fine-tune the navigation policy neural network, the resource encoder neural network, or both.

[0111] The following table shows the success rate (%) for navigating a graph to a goal with different numbers of navigation steps required to reach the goal. The table compares implementations of the described method - Randomized Forward Behavior Cloning (RFBC) and RFBC+RL (Reinforcement Learning) - with various other methods - specifically a greedy strategy that selects the action with the minimum cosine distance to the goal, and two types of depth-first search (DFS). The described technique performs much better than the other techniques.

[0112] This specification uses the term "configured" in connection with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0113] The embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by a data processing device or for controlling the operation of the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical signal, optical signal, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0114] The term "data processing apparatus" refers to data processing hardware and covers all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to the hardware, the apparatus may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0115] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and the computer program may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines or code portions). A computer program may be deployed to execute on one or more computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0116] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way or at all, and it may be stored on a storage device in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed differently.

[0117] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same computer or multiple computers.

[0118] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit (e.g., an FPGA or ASIC) or by a combination of a special purpose logic circuit and one or more programmed computers.

[0119] Computers suitable for executing computer programs can be based on general or special microprocessors or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by or incorporated into a special logic circuit. Typically, the computer will also include or be operably coupled to receive data from or transmit data to or both of one or more large-capacity storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0120] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

[0121] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device (e.g., a smart phone running a messaging application) and receiving a response message from the user.

[0122] The data processing apparatus for implementing a machine learning model may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts of machine learning training or production (i.e., inference) workloads.

[0123] Machine learning models can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework or the Jax framework).

[0124] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0125] A computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device, for example, to display data to a user interacting with the device acting as a client and to receive user input from it. Data generated at the user device, such as the result of a user interaction, can be received from the device at the server.

[0126] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or on the scope that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve a sub-combination or a variation of the sub-combination.

[0127] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0128] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more computers, the method comprising: Receive inquiries; processing the query using a query encoder neural network to generate an embedding for the query; Identifies the current resource in the resource collection; as well as Generate a response to the query, including: At each of multiple navigation iterations: identifying a plurality of outgoing links from the current resource as of the navigation iteration, each outgoing link being a link from the current resource to a respective other resource in the collection of resources; Processing content from the current resource using a resource encoder neural network to generate an embedding of the current resource; Generate the corresponding embedding for each outgoing link; generating, using a navigation policy neural network, a respective score for each outgoing link from at least the embedding of the outgoing link, the embedding of the current resource, and the embedding of the query; selecting one of the outgoing links based on the respective scores of the outgoing links; and updating the current resource to the corresponding other resource linked to by the outgoing link; and After a last navigation interaction, a response is generated that represents the current resource after the last navigation iteration.

2. The method according to claim 1, wherein: The resource collection includes one or more websites, and wherein each resource in the collection represents at least a portion of a web page in the one or more websites.

3. The method according to claim 2, wherein: For at least one of the navigation iterations, the current resource as of the navigation iteration represents a portion of a particular web page, and the plurality of outgoing links from the current resource as of the navigation iteration include (i) a respective link to each of one or more other portions of the particular web page and (ii) one or more hyperlinks to other web pages in the collection.

4. The method according to any one of claims 2 or 3, wherein: The query is submitted via an Internet search engine, and wherein (i) the one or more websites have not been indexed by the Internet search engine, or (ii) the content of one or more of the web pages in the one or more websites has changed since the one or more websites have been indexed by the Internet search engine.

5. The method according to any one of claims 2 to 4, wherein: The one or more websites are inaccessible to search engine crawlers.

6. A method according to any preceding claim, wherein: The query is received from a client device, and wherein the method further comprises: The response to the query is provided to the client device.

7. A method according to any preceding claim, wherein: The response to the query includes a link to the current resource after the last navigation iteration.

8. A method according to any preceding claim, wherein the response to the query comprises (i) content from the current resource after the last navigation iteration, (ii) a summary of the content from the current resource after the last navigation iteration, or (iii) both.

9. A method according to any preceding claim, wherein: The resource encoder neural network includes a self-attention neural network configured to process text tokens from the resource to generate a corresponding token embedding for each of the text tokens.

10. A method according to any preceding claim, wherein: Selecting one of the outgoing links based on the respective scores of the outgoing links comprises: The outgoing link with the highest corresponding score is selected, or a probability distribution including the corresponding probability of each outgoing link is generated from the corresponding scores and the outgoing links are sampled from the probability distribution.

11. The method according to any preceding claim, further comprising: After performing the last navigation iteration, it is determined that a termination criterion for generating the response has been met.

12. The method according to claim 11, wherein: Determining that the termination criteria have been met includes: It is determined that a threshold number of navigation iterations have been performed.

13. The method according to claim 11, wherein: Determining that the termination criteria have been met includes: A determination is made as to whether a similarity between the embedding of the query and the embedding of the current resource after the last navigation iteration satisfies a threshold.

14. A method according to any preceding claim, wherein: Generating a corresponding score for each outgoing link from at least the embedding of the outgoing link, the embedding of the current resource, and the embedding of the query using a navigation policy neural network includes: processing a navigation input comprising the embedding of the current resource and the embedding of the query using one or more neural network layers to generate a combined embedding; and For each outgoing link, the respective score for the outgoing link is determined based on a similarity between the embedding of the combination and the embedding of the outgoing link.

15. The method according to claim 14, wherein: The navigation input also includes a corresponding embedding of the current resource as of any previous navigation iteration.

16. A method according to any preceding claim, wherein: Generating the corresponding embedding for each outgoing link involves: The respective embedding is generated from at least (i) content of the respective other resource linked to by the outgoing link, (ii) text in the current resource corresponding to the outgoing link, or (iii) both.

17. The method according to claim 16, wherein: Generating the corresponding embedding from at least (i) content of the corresponding other resource linked to by the outgoing link, (ii) text in the current resource corresponding to the outgoing link, or (iii) both includes: The corresponding embedding is generated from at least (i) content of the corresponding other resource linked to by the outgoing link, (ii) text in the current resource corresponding to the outgoing link, or (iii) both, and (iv) one or more of the following: a first indicator indicating whether the corresponding other resource has been selected as the current resource at a previous navigation iteration, or A second indicator indicates whether the corresponding other resource is another part of the same web page as the current resource or a web page different from the current resource.

18. The method according to claim 16 or claim 17, wherein: Generating the corresponding embedding from at least (i) content of the corresponding other resource linked to by the outgoing link, (ii) text in the current resource corresponding to the outgoing link, or (iii) both includes processing (i), (ii), or (iii) using the resource encoder neural network.

19. A method according to any preceding claim, wherein: The current resources in the identified resource collection include: The current resource is identified by applying information retrieval techniques to identify resources in the collection that are relevant to the query.

20. The method of any preceding claim, further comprising: identifying one or more other resources in the resource collection; as well as For each identified other resource, a navigation iteration is performed to identify final other current resources, wherein the response characterizes one or more of the final other current resources.

21. A method of training a navigation policy neural network and a query encoder neural network according to any preceding claim, the method comprising: pre-training at least the navigation strategy neural network on a first graph navigation task; as well as After the pre-training, at least the query encoder neural network is fine-tuned on an information retrieval task.

22. The method according to claim 21, wherein: Pre-training at least the navigation policy neural network on a first graph navigation task includes pre-training the navigation policy neural network by reinforcement learning of a reward, the reward measuring whether the first graph navigation task was successfully completed.

23. The method according to any one of claims 21 to 22, wherein: Each instance of the first graph navigation task requires reaching a target resource of the instance within a corresponding threshold number of navigation iterations starting from a source resource of the instance.

24. The method according to claim 23, wherein: Different instances of the task have different corresponding thresholds.

25. The method according to any one of claims 21 to 24, wherein: Pre-training at least the navigation policy neural network on a first graph navigation task includes pre-training the resource encoder neural network.

26. A system comprising: one or more computers; as well as One or more storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform the corresponding operations according to any one of claims 1 to 25.

27. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform corresponding operations of the method of any one of claims 1 to 25.

Images