A transformer-based text encoder for passage retrieval

Abstract

The computing system includes a logic subsystem and a storage subsystem that holds instructions executable by the logic subsystem to implement a transformer-based text encoder. The transformer-based text encoder includes a plurality of transformer blocks pre-trained to apply encoding operations to computer-readable text representations of input text strings, the computer-readable text representations including a computer-readable question representation of an input text question and a computer-readable passage representation of an input text passage. The plurality of transformer blocks include a shared transformer block trained for both the computer-readable question representation and the computer-readable passage representation, and dedicated transformer blocks including two or more input-specific subnetworks and a routing function for selecting an input-specific subnetwork from the two or more input-specific subnetworks for each of the computer-readable text representations.

Description

[Background technology]

[0001] background Passage retrieval is used to identify text passages or specific portions of text within a passage that are relevant to an input question. After receiving an input question such as "Who is the current President of the United States?", a passage retrieval system evaluates many different text passages (e.g., documents, web pages) to determine which are most relevant to the input question and therefore most likely to contain the requested information.

[0002] This can be achieved using a transformer-based text encoder that converts input text sequences (e.g., strings of letters, numbers, punctuation, and other characters) into computer-readable representations of text sequences that can be compared with each other. For example, the input question and each text passage can be encoded as a vector. The passage retrieval system then selects one particular passage as most relevant to the input question by computing the dot product of the vector representing the question with the vectors representing the various passages to determine which passage vector is most similar to the question vector. Summary of the Invention [Means for solving the problem]

[0003] overview This Summary is provided to introduce selected concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Moreover, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.

[0004] This disclosure generally describes a transformer-based text encoder that includes a shared transformer block and a dedicated transformer block. The dedicated transformer block includes a routing function that routes input text representations between different input-specific subnetworks. In one embodiment, the routing function routes question representations to the question-specific subnetworks of the dedicated transformer block, while passage representations are routed to the passage-specific subnetworks of the dedicated transformer block. The shared transformer block is trained for both the question representation and the passage representation. In this way, a single transformer-based text encoder can be used to encode both the input text question and the input text passage. [Brief explanation of the drawings]

[0005] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] 1 illustrates a schematic of passage retrieval based on an input text query. [Figure 2] 1 illustrates schematically the use of a transformer-based text encoder to encode an input text string as an output vector representation. [Figure 3] 1 illustrates an exemplary method for computer passage retrieval. [Figure 4A] 1 illustrates a schematic diagram of text encoding using a shared transformer block in a transformer-based text encoder. [Figure 4B] 1 illustrates a schematic diagram of text encoding using a dedicated transformer block of a transformer-based text encoder. [Figure 5] 10 illustrates schematically the routing of different tokens of an input text representation to different input-specific sub-networks of a dedicated transducer block. [Figure 6] 1 illustrates a schematic diagram of an exemplary transformer-based text encoder including dedicated and shared transformer blocks interleaved therewith; [Figure 7] 10 illustrates schematically the implementation of vector comparison between an output vector question representation and a plurality of output vector passage representations. [Figure 8] 1 illustrates an exemplary method for training a transformer-based text encoder. [Figure 9A] 1 illustrates schematically the updating of parameters of a transformer-based text encoder during different training passes. [Figure 9B] 1 illustrates schematically the updating of parameters of a transformer-based text encoder during different training passes. [Figure 10] 1 illustrates a schematic diagram of an exemplary computing system. DETAILED DESCRIPTION OF THE INVENTION

[0006] Detailed Description Dense search frameworks are very useful for fetching external knowledge in various text processing tasks. For open-domain question answering (ODQA), a dense searcher that performs well is a bi-encoder architecture that includes a question encoder and a passage encoder. In some implementations, the two encoders are isomorphic but separately parameterized, as they are initialized from the same pre-trained model and then fine-tuned for the task.

[0007] Despite its usefulness, the bi-encoder architecture has notable drawbacks. First, from an efficiency perspective, bi-encoder parameterization can cause scaling bottlenecks both during training and inference. Second, bi-encoder parameters are believed to underperform coarse information retrieval models, such as BM25, in various settings. Bi-encoder parameters are also believed to be more sensitive to distribution shifts than coarse information retrieval models, resulting in poor generalization for questions involving rare entities.

[0008] Thus, this disclosure describes a transformer-based text encoder that can be used for both input questions and input passages. Instead of using two different, identical but separate encoders for questions and passages, the transformer-based text encoder described herein includes at least one "shared" transformer block and at least one "dedicated" transformer block. The shared transformer block is trained for both questions and passages, while the dedicated transformer block includes different input-specific sub-networks used for different encoding tasks. In one example, the dedicated transformer block includes a question-specific sub-network for encoding the input question and a passage-specific sub-network for encoding the input passage.

[0009] In some embodiments, the text encoder includes some shared transformer blocks and some dedicated transformer blocks interleaved with each other - e.g., the output from a shared transformer block is passed to a dedicated transformer block, the output of a dedicated transformer block is passed to another shared transformer block, etc. For simplicity, this disclosure primarily describes the transformer-based text encoder as having one shared transformer block and one dedicated transformer block, but it will be understood that this is not a limitation; rather, the transformer-based text encoder has at least one shared transformer block and at least one dedicated transformer block.

[0010] The use of shared and dedicated transformer blocks in this manner provides the technical benefit of reducing computational resource consumption because at least some network parameters are shared between the encoding operations applied to questions and passages. This is in contrast to bi-encoder architectures in which the question encoder and the passage encoder are parameterized separately. Furthermore, the use of transformer-based text encoders as described herein provides the technical benefit of improving human-computer interaction by providing more accurate and resource-efficient passage retrieval across a wide range of question domains compared to other passage retrieval solutions.

[0011] Figure 1 illustrates an exemplary passage retrieval scenario. It is understood that Figure 1 is non-limiting and highly simplified for illustrative purposes. Figure 1 illustrates an exemplary passage retrieval system 100. A passage retrieval system may typically be used to identify text passages or specific portions of text passages that are appropriate for an input text query. This is done via a transformer-based text encoder having a shared transformer block and a dedicated transformer block, as described in detail below.

[0012] In FIG. 1 , a passage retrieval system receives an input text query 102. The passage retrieval system encodes the input text query and compares it to encoded representations of several different input text passages, including passages 104A, 104B, and 104C. As shown, passages 104A and 104C share some similarity to the input text query but do not contain the ultimately requested information. In contrast, text passage 104B contains the answer to the input text query, and thus, a portion 106 of the text passage is retrieved by the passage retrieval system as a response to the input text query. In another example, instead of retrieving only a portion of the passage, as shown in FIG. 1 , the passage retrieval system retrieves the entire passage identified as most relevant to the input text query.

[0013] The passage retrieval system may be implemented via any suitable computing system of one or more computing devices, each having any suitable capabilities, hardware configuration, and form factor. In some embodiments, the passage retrieval system may be implemented via two or more different computing devices communicating over a network, such as the Internet. As one non-limiting example, the passage retrieval system may be implemented as computing system 1000, described below with respect to FIG. 10.

[0014] It will be understood that the input text question and the input text passage each may take any suitable form and have any suitable source. In some examples, the input text question is created by a human user—e.g., typed into a search bar or spoken into a microphone and subjected to speech-to-text processing. In other examples, the input text question has another suitable source—e.g., loaded from computer storage or received over a computer network. Similarly, the input text passage, in some cases, is held in local storage of the passage retrieval computing system—e.g., the input text passage is a text document or database entry. Additionally or alternatively, the input text passage may be accessed over a computer network, such as the Internet. For example, in one example, the input text passage takes the form of a web page accessed from a remote server. It will be understood that this disclosure is primarily concerned with the use of a transformer-based text encoder to perform passage retrieval, regardless of the particular source and content of the questions and passages being encoded.

[0015] Figure 2 illustrates generally the operation of a transformer-based text encoder in more detail. In particular, Figure 2 illustrates generally an input text string 200, which generally represents both an input text question and an input text passage. The input text string may take the form of any suitable sequence of text characters, including any or all of letters, numbers, spaces, punctuation, etc.

[0016] Before being provided to the transformer-based text encoder, the input text string is converted into a computer-readable text representation 202. As one non-limiting example, the computer-readable text representation takes the form of an input vector representation generated by tokenizing the input text string into a plurality of input tokens, each token representing a different character or sequence of characters in the input text string. Values ​​corresponding to each input token are then embedded into a vector representing the input text string, and this input vector representation is passed to the transformer-based text encoder for encoding as an output vector representation. In this manner, an input text question may be converted into a computer-readable question representation, and an input text passage may be converted into a computer-readable passage representation. It will be understood that in other examples, the computer-readable text representation may take other suitable forms.

[0017] Continuing with FIG. 2, a computer-readable text representation 202 is input to a transformer-based text encoder 204. The transformer-based text encoder includes multiple transformer blocks 206 that are pre-trained to apply encoding operations to the computer-readable text representation. The multiple transformer blocks of the transformer-based text encoder include at least a shared transformer block 208 and a dedicated transformer block 210. The shared transformer block is trained for both the computer-readable question representation and the computer-readable passage representation. The dedicated transformer block includes two or more different input-specific sub-networks—e.g., one question-specific sub-network and one passage-specific sub-network. More specific details regarding the shared and dedicated transformer blocks are provided below.

[0018] Generally, a transformer-based text encoder is constructed by multiple transformer blocks, as described herein. In some embodiments, each transformer block includes a multi-head self-attention (MHA) sublayer and at least one feed-forward network (FFN) sublayer, with residual connections and layer normalization applied to both sublayers. An input vector h∈R is d Given , the FFN sublayer generates the output vector: FFN(h)=W z max{0,W1h+b1}+b2 where W1∈R m×d , W2∈R d×m , b1∈R m and b2∈R d is a learnable parameter. For a sequence of N tokens, each transformer block generates N corresponding vectors that, along with a vector of special prefix tokens [CLS], can be used as a representation of the sequence of N tokens. In other embodiments, the transformer blocks are implemented in other suitable ways. Non-limiting examples of suitable machine learning (ML) and / or artificial intelligence (AI) techniques that can be used to implement the transformer blocks are described below with respect to FIG. 10.

[0019] After the encoding operations are applied by the transformer block, the transformer-based text encoder outputs an encoded version of the input text string as an output vector representation 212. As explained further below, the various vector representations output by the transformer-based text encoder are compared to each other, for example, to determine which of several text passages is most appropriate for the input text question. In some embodiments, the transformer-based text encoder uses the final output vectors of [CLS] from the corresponding encoders to encode the question and passage, q∈R, respectively. d and p∈R dThe relevance between a question and a passage can then be measured in the vector space by computing the dot product between the corresponding vectors. sim(q,p) = qTp.

[0020] 3 illustrates an exemplary method 300 of computer passage searching. Method 300 may be implemented by any suitable computing system of one or more computing devices. Any computing device-implemented method 300 may have any suitable capabilities, hardware configuration, and form factor. In some embodiments, different steps of method 300 are performed by two or more different computing devices communicating over a computer network. As one non-limiting example, method 300 is implemented by computing system 1000, described below with respect to FIG. 10.

[0021] At 302, method 300 includes receiving a computer-readable question representation representing an input text question. This may be done substantially similarly as described with respect to Figures 1 and 2. As one non-limiting example, a human user formulates an input text question, which is then converted into an input vector representation by tokenizing it into a sequence of input tokens. Values ​​representing the input tokens are then embedded within a vector representing the input question.

[0022] At 304, the method 300 includes passing the computer-readable question representation to a transformer-based text encoder that is pre-trained to encode the computer-readable question representation and the computer-readable passage representation using shared and dedicated transformer blocks.

[0023] This is illustrated schematically with respect to Figures 4A and 4B. Specifically, Figure 4A schematically illustrates an input text question 400Q being converted into a computer-readable question representation 402Q, and an input text passage 400P being converted into a computer-readable passage representation 402P. Both the question representation 402Q and the passage representation 402P are input to the shared transformer block 208 of the transformer-based text encoder 204 shown in Figure 2.

[0024] The shared transformer block is trained to perform encoding operations on both the computer-readable question representation and the computer-readable passage representation. In some cases, the shared transformer block is the first transformer block of a transformer-based text encoder. In other cases, the shared transformer block receives a text representation that has already been at least partially encoded by one or more preceding transformer blocks of the transformer-based text encoder. The output from the shared transformer block is then passed to the next transformer block of the transformer-based text encoder, or alternatively, if the shared transformer block is the last transformer block of the text encoder, is output as an output vector representation of the input text string.

[0025] 4A is shown as if the computer-readable question representation and the computer-readable passage representation are both provided to a shared transducer block simultaneously. However, it will be understood that this need not be the case. Rather, in some embodiments, the computer-readable question representation and the computer-readable passage representation are provided to the transducer-based text encoder at different times. In some cases, the computer-readable question representation and the computer-readable passage representation are encoded simultaneously by different instances of a transducer-based text encoder implemented by the same or different computing devices.

[0026] In FIG. 4A, a shared transformer block performs the encoding operation via a neural network 404. By way of example, this takes the form of a feedforward network (FFN). The neural network is parameterized with a shared set of parameters 406 that are used to apply the encoding operation to both the computer-readable question representation and the computer-readable passage representation. This contrasts with a bi-encoder architecture, in which two independent sets of parameters are used for separate question and passage encoders. The use of a shared transformer block in a transformer-based text encoder therefore provides the technical benefit of conserving computational resources by reducing the number of parameters used to perform text encoding and passage retrieval. Exemplary techniques for training the shared transformer block, and thereby modifying the shared set of parameters used in the neural network, are provided below with respect to FIGS. 8-9B.

[0027] The transformer-based text encoder also includes a dedicated transformer block that is used to apply encoding operations to the question and passage representations. This is shown schematically with respect to FIG. 4B, which illustrates the dedicated transformer block 210 of the transformer-based text encoder 204. Like the shared transformer block, the dedicated transformer block in some embodiments is the first transformer block of the transformer-based text encoder. In other embodiments, the dedicated transformer block receives a text representation that has already been at least partially encoded by one or more previous transformer blocks of the transformer-based text encoder—for example, the dedicated transformer block receives the output from a previous shared transformer block. The output from the dedicated transformer block is then passed to the next transformer block of the transformer-based text encoder, or, if the dedicated transformer block is the last transformer block of the text encoder, is alternatively output as an output vector representation of the input text string.

[0028] In general, the dedicated transducer block includes two or more input-specific subnetworks and a routing function for selecting an input-specific subnetwork from among the two or more input-specific subnetworks for each of the computer-readable text representations. In Figure 4B, the dedicated transducer block includes a routing function 408, a question-specific subnetwork 410Q, and a passage-specific subnetwork 410P. In some embodiments, the dedicated transducer block may be described as using a mixture of experts (MOE) approach, in which multiple expert FFN sublayers are used in parallel and a routing function is used to select among these expert FFN sublayers.

[0029] To this end, different input-specific sub-networks in some embodiments are parameterized differently to apply different encoding operations to different types of input text representations. In other words, a first input-specific sub-network is parameterized with a first set of parameters, and a second input-specific sub-network is parameterized with a second set of parameters that differ from the first set of parameters. This provides the technical benefit of improving the performance of transformer-based text encoders for both question and passage representations, thereby improving human-computer interaction.

[0030] The dedicated transducer block includes a routing function for routing different text representations to different input-specific subnetworks. In the example of Figure 4B, the routing function routes computer-readable question representations through question-specific subnetworks and routes computer-readable passage representations through passage-specific subnetworks. In other words, the routing function is a deterministic routing function that automatically routes computer-readable question representations through question-specific subnetworks and computer-readable passage representations through passage-specific subnetworks.

[0031] In other examples, other suitable routing functions are used. As additional non-limiting examples, the routing function uses sequence-based routing (Seq-R) or token-based routing (Tok-R). In the case of Seq-R and Tok-R, the routing function is a parameterized routing function trained to route computer-readable text representations such that each computer-readable question representation is routed through one or more question-specific sub-networks, and each computer-readable passage representation is routed through one or more passage-specific sub-networks. Such a parameterized routing function may be expressed as follows: R(u) = Gumbel Softmax(Au + c) where Gumbel Softmax is d×I and c∈R I It outputs an I-dimensional one-hot vector based on a linear projection parameterized by, where I is the number of input eigensubnetworks in the dedicated transformer block, and u∈R d is the input of the routing function. In some embodiments, the routing function is learned jointly with other network parameters of the transformer-based text encoder using discrete reparameterization.

[0032] As discussed above, in some embodiments, the computer-readable text representation input to a transformer-based text encoder comprises multiple input tokens. In the case of Seq-R, routing is performed at the sequence level, where all tokens in a sequence are routed to an FFN input vector h representing a special prefix token [CLS]. [cls] In other words, each computer-readable text representation includes a sequence of tokens, and all tokens of a given computer-readable text representation are routed through the same input-specific sub-network of two or more input-specific sub-networks via a pre-trained parameterized routing process.

[0033] In contrast, in Tok-R, the routing function routes each token independently—e.g., for the jth token in the sequence, u routes the corresponding FFN input vector h j In other words, each computer-readable text representation includes a sequence of tokens, and the routing function routes two or more different tokens of a given computer-readable text representation through different ones of the two or more input-specific sub-networks.

[0034] This scenario is illustrated generally with respect to Figure 5, which shows an exemplary computer-readable text representation 500. Representation 500 includes a sequence of tokens 502A-502C. As discussed above, in some embodiments, the computer-readable text representation is generated by applying a tokenization process to the input text string, which converts individual characters or strings of characters in the input text into a sequence of token representations. Values ​​representing the tokens may then be embedded in the input vector representation.

[0035] A computer-readable text representation 500 is input to a dedicated transformer block 504, which is one of several different transformer blocks in a transformer-based text encoder. A routing function 506 in the dedicated transformer block routes tokens of the computer-readable text representation between different input-specific subnetworks 508A-508C of the dedicated transformer block. In this example, the routing function uses Tok-R routing, in which different tokens of the same text representation are routed to different subnetworks of the dedicated transformer block. For example, token 502A is routed to subnetwork 508A, while tokens 508B and 508C are routed to subnetworks 508B and 508C. This provides the technical benefit of improving the performance of the transformer-based text encoder, as the final generated output vector representation of the input text string will more accurately capture the relationships and context between the different input characters and words in the input text string.

[0036] As discussed above, this disclosure primarily describes a transformer-based text encoder as having one shared transformer block and one dedicated transformer block. It will be understood that this is done for simplicity only. Rather, in some embodiments, a "shared transformer block" described herein is one of two or more shared transformer blocks of a transformer-based text encoder, and a "dedicated transformer block" is one of two or more dedicated transformer blocks of a transformer-based text encoder. Specifically, in some embodiments, two or more shared transformer blocks are interleaved with two or more dedicated transformer blocks.

[0037] Such a scenario is illustrated schematically with respect to Figure 6, which shows another exemplary transformer-based text encoder 600. The transformer-based text encoder 600 includes at least four shared transformer blocks 602A-602D and at least two dedicated transformer blocks 604A and 604B. Furthermore, the shared and dedicated transformer blocks are interleaved, such that the output of shared transformer block 602B passes to dedicated transformer block 604A, the output of dedicated transformer block 604A passes to shared transformer block 602C, etc. Such an interleaved arrangement provides the technical benefit of improving the encoding and passage retrieval performance of the transformer-based text encoder, thereby improving human-computer interaction.

[0038] In the example of Figure 6, the transformer blocks have a ratio of two shared transformer blocks for every one dedicated transformer block. In other examples, other suitable ratios are used. Furthermore, in other examples, block interleaving need not be used. Rather, for example, another exemplary scenario uses a group of two or more shared transformer blocks followed by a group of two or more dedicated transformer blocks before the transformer-based text encoder outputs a vector representation of the input text string.

[0039] 3, at 306, the method 300 includes receiving an output vector representation of the computer-readable question expression. As discussed above, in some embodiments, the transformer-based text encoder uses the final output vectors of [CLS] from the corresponding encoders to convert the question and passage, respectively, q∈R d and p∈R d In other embodiments, the vector representation output by the transformer-based text encoder takes any other suitable form.

[0040] Continuing with FIG. 3 , at 308, method 300 includes performing a vector comparison of the output vector question representation with the plurality of output vector passage representations. This enables a computing system to evaluate relevance between the input text question, which has also been encoded via the transformer-based text encoder, and the plurality of input text passages. As one non-limiting example, the output vector question representation and the output vector passage representation are compared in vector space by calculating a dot product between corresponding vectors. More generally, in some embodiments, performing the vector comparison includes calculating a relevance score between the output vector question representation and the plurality of output vector passage representations, where the dot product is one suitable example of a relevance score.

[0041] At 310, method 300 includes identifying one or more input text passages as being relevant to the input text question. As an example, this includes identifying the text passage with the highest relevance score to the input text question—e.g., identifying the output vector passage representation with the largest dot product with the output vector question representation. Additionally or alternatively, in some scenarios, none of the encoded text passages may be relevant to the input text question. Thus, in some examples, one or more input text passages identified as being relevant to the input text question are associated with output vector passage representations that have a relevance score higher than a threshold score. If none of the output vector passage representations have a relevance score higher than the threshold score, then no input text passages are identified as being relevant to the input text question. In various examples, any suitable relevance score threshold is used, depending on the particular method by which the relevance scores are calculated and the desired confidence that the identified text passages are indeed relevant to the input text question.

[0042] This process is illustrated generally with respect to FIG. 7, which schematically illustrates an exemplary computer-readable question representation 700. Representation 700 is input to a transformer-based text encoder 702, which, as described above, encodes the question representation as an output vector question representation 704. The computing system then performs a vector comparison 706 between the output vector question representation and a plurality of output vector passage representations 708A-D corresponding to input text passages previously encoded by the transformer-based text encoder. As a result of the vector comparison, an output vector passage representation 706C is identified as being appropriate for the input text question based on a relevance score (e.g., maximum dot product) calculated via the vector comparison.

[0043] 3 , at 312, method 300 includes outputting an indication of one or more input text passages identified as relevant to the input text question. Such indications may take any suitable form, depending on the implementation. In one example, the indications specify unique identifiers or addresses (e.g., file system addresses, network addresses) of the text passages identified as relevant to the question. As another example, the indications include a duplication or reproduction of at least a portion of the text passages identified as relevant—e.g., the entire text passage or only a portion of the text passage that provides relevant information to the input text question is presented for display.

[0044] At 314, method 300 includes determining whether there are additional input text questions to encode via the transformer-based text encoder. If yes, method 300 returns to step 302 to encode the next input text question. If no, method 300 ends until another input text question is provided.

[0045] This disclosure has thus focused on runtime use of transformer-based text encoders. However, prior to such runtime use, the encoder is first trained on several input training examples. Over a series of training passes, the network parameters of the transformer-based text encoder are modified to improve the encoder's performance. In some embodiments, the transformer-based text encoder is trained using a set of training text questions and a set of training text passages, each containing information corresponding to the training text questions, such that each training text question is associated with one or more training text passages. In some embodiments, the transformer-based text encoder is further trained using a set of negative training text passages, such that each training text question is associated with one or more negative training text passages containing information inappropriate to the training text question.

[0046] FIG. 8 illustrates an exemplary method 800 for training a transformer-based text encoder that encodes input text questions and input text passages. Similar to method 300, method 800 may be implemented via any suitable computing system of one or more computing devices. Any computing device-implemented method 800 may have any suitable capabilities, hardware configuration, and form factor. In some embodiments, various steps of method 800 are performed by two or more different computing devices communicating over a computer network. As one non-limiting example, method 800 is implemented by computing system 1000, described below with respect to FIG. 10.

[0047] At 802, method 800 includes receiving a set of training computer-readable text representations representing training text strings. These include training computer-readable question representations representing training text questions and training computer-readable passage representations representing training text passages. In particular, in this case, the training text passages are known to contain information corresponding to the training text questions, such that each training text question is associated with one or more training text passages. Thus, the performance of the transformer-based text encoder can be evaluated during training by comparing output vector representations corresponding to input text questions with input passage questions known to contain information relevant to the input text questions.

[0048] At 804, method 800 includes providing training computer-readable text representations to a transformer-based text encoder over a series of training passes. At 806, method 800 includes outputting, in the transformer-based text encoder, a corresponding output vector representation for each of the training computer-readable text representations. The transformer-based text encoder is implemented substantially as described above, except that the encoder is still being trained and, therefore, its performance may not yet be sufficient for practical use. Thus, the transformer-based text encoder includes a plurality of transformer blocks for collectively encoding the training computer-readable text representations as output vector representations, such that training text questions are encoded as output vector question representations and training text passages are encoded as output vector passage representations.

[0049] The plurality of transformer blocks includes a shared transformer block and a dedicated transformer block, as described above. Specifically, the shared transformer block is used to apply encoding operations to both the training text questions and the training text passages. The dedicated transformer block includes two or more input-specific sub-networks and a routing function for selecting an input-specific sub-network from the two or more input-specific sub-networks for each of the training computer-readable text representations.

[0050] This is illustrated schematically with respect to Figures 9A and 9B. Specifically, Figure 9A illustrates the training of a transformer-based text encoder during one training phase of a series of training phases. In Figure 9A, the set of training text representations 900 includes a training computer-readable question representation 902Q and a training computer-readable passage representation 902P. As discussed above, these represent corresponding input text questions and input passage representations. However, in this case, the training passages are known to contain information relevant to the training questions. In some cases, the training questions are also paired with one or more negative passage examples that are inappropriate for the training questions. It will be appreciated that the set of training text representations may include any suitable number of different training question representations, each associated with any suitable number of different training passage representations.

[0051] The training text representation is input to a transformer-based text encoder 904. In Figure 9A, the transformer-based text encoder is parameterized with an initial set of network parameters 906A. These generally represent the parameters of any or all of the different transformer blocks of the transformer-based text encoder - for example, the parameters used for the different shared transformer blocks, the parameters used for the different input-specific sub-networks of the dedicated transformer blocks, and the parameters used for the routing functions of the dedicated transformer blocks if Seq-R or Tok-R routing functions are used.

[0052] The transformer-based text encoder outputs vector question representations 908Q and vector passage representations 908P corresponding to the input training question and passage representations 902Q and 902P. The computing system then calculates a relevance score 910A between the vector question representations and the vector passage representations. As discussed above, the training text passages are known to contain relevant information relative to the input text passages, so the relevance score serves as an indicator of the performance of the encoder during training.

[0053] 8, at 808, method 800 includes modifying one or more parameters of the transformer-based text encoder before a next training pass in the series of training passes, in order to increase the relevance score between an output vector question representation corresponding to the training text question (e.g., vector representation 908Q) and an output vector passage representation corresponding to a training text passage related to the training text question (e.g., vector representation 908P) compared to a previous training pass in the series of training passes.

[0054] This is shown schematically in Figure 9B, which again shows training text representations 900 provided to the transformer-based text encoder 904. However, Figure 9B represents a later training pass in the series of training passes compared to Figure 9A. Between the scenario depicted in Figure 9A and that depicted in Figure 9B, the parameters of the transformer-based text encoder have been updated to a set of updated parameters 906B. This affects the performance of the transformer-based text encoder, causing the encoder to output an updated vector question representation 912Q and an updated vector passage representation 912P, which are compared to give an updated relevance score 910B that is higher than relevance score 910A.

[0055] The specific training procedure used to modify the parameters of a transformer-based text encoder varies depending on the implementation. Various embodiments use single-set or multi-set training procedures. In single-set training, the transformer-based text encoder is trained using only a single dataset and then evaluated on that dataset. This offers the technical benefit of faster and more efficient training of the transformer-based text encoder, especially if the encoder is intended for runtime use within a relatively specific domain captured by the dataset. However, single-set training may negatively impact the encoder's performance on other datasets it was not trained on. In contrast, in multi-set training, the encoder is trained by combining training data from multiple datasets. This offers the technical benefit of improving the encoder's performance across a wide range of question and passage domains. In some embodiments, during training, the transformer-based text encoder is optimized based on the following contrastive learning objectives:

number

[0056] In such cases, L sim is a negative passage for each question. - Use the set p - There are several ways to construct the mini-batch. As one non-limiting example, two negative passages per question are used: one is the top passage retrieved by BM25 that does not contain the answer to the question, and the other is selected from the gold positive passages of other questions in the same mini-batch.

[0057] It has been shown that mining more hard-negative examples in iterative training can lead to better performance. Thus, in some embodiments, a transformer-based text encoder is trained using hard-negative mining. Specifically, the encoder trains a negative passage p1 generated as described above. - Then, the negative passage p2 - The second set of p1 is generated by searching the top 100 ranked passages (or another suitable number of top ranked passages) for each question, excluding the gold passages. In a single-set training scenario, p1 - and p2 - are combined to train the encoder. In multi-set training scenarios, p2 offers the technical benefit of improving training efficiency. - Only ∑ i = 1 ...

[0058] As discussed above, in some cases the routing function used by the dedicated transducer block of a transducer-based text encoder is a trained parameterized routing function. In some embodiments, such a parameterized routing function is used to train the contrastive learning objective L described above. sim The routing functions are trained using entropy normalization. For the Seq-R and Tok-R routing functions, entropy normalization is applied to avoid routing all inputs to the same expert FFN sublayer. In other words, training the routing functions involves using entropy normalization to vary which of two or more input-specific subnetworks is selected for each of a plurality of training computer-readable text representations that are input to the transformer-based text encoder over a series of training passes.

[0059] This is the following L ent This can be done by calculating:

number

[0060] It is understood that the above training procedures are non-limiting examples.

[0061] Another example of a suitable training procedure is described below with respect to FIG.

[0062] The methods and processes described herein may be coupled to the computing systems of one or more computing devices. In particular, such methods and processes may be implemented as executable computer application programs, network-accessible computing services, application programming interfaces (APIs), libraries, and combinations of the above and / or other computing resources.

[0063] 10 schematically illustrates a simplified representation of a computing system 1000 configured to provide any or all of the computing functionality described herein. Computing system 1000 may take the form of one or more personal computers, network-accessible server computers, tablet computers, home entertainment computers, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), virtual / augmented / mixed reality computing devices, wearable computing devices, Internet of Things (IoT) devices, embedded computing devices, and / or other computing devices.

[0064] The computing system 1000 includes a logic subsystem 1002 and a storage subsystem 1004 .

[0065] Computing system 1000 may optionally include a display subsystem 1006, an input subsystem 1008, a communication subsystem 1010, and / or other subsystems not shown in FIG.

[0066] The logical subsystem 1002 includes one or more physical devices configured to execute instructions. For example, a logical subsystem may be configured to execute instructions that are part of one or more applications, services, or other logical constructs. A logical subsystem may include one or more hardware processors configured to execute software instructions. Additionally or alternatively, a logical subsystem may include one or more hardware or firmware devices configured to execute hardware or firmware instructions. The processors of a logical subsystem may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Individual components of a logical subsystem may optionally be distributed among two or more separate devices that may be remotely located and / or configured for collaborative processing. Aspects of a logical subsystem may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.

[0067] The storage subsystem 1004 includes one or more physical devices configured to temporarily and / or permanently hold computer information, such as data and instructions, executable by the logical subsystem. When the storage subsystem includes two or more devices, these devices may be coupled and / or remotely located. The storage subsystem 1004 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location-addressable, file-addressable, and / or content-accessible devices. The storage subsystem 1004 may include removable and / or embedded devices. When the logical subsystem executes instructions, the state of the storage subsystem 1004 may be transformed, for example, to hold different data.

[0068] Aspects of logic subsystem 1002 and aspects of storage subsystem 1004 may be integrated together into one or more hardware logic components, which may include, for example, program application specific integrated circuits and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on a chip (SOC), and complex programmable logic devices (CPLD).

[0069] The logic subsystem and storage subsystem may cooperate to instantiate one or more logical machines. As used herein, the term "machine" is used to collectively refer to a combination of hardware, firmware, software, instructions, and / or any other components that cooperate to provide computer functionality. In other words, a "machine" is never an abstract idea and therefore always has a tangible form. A machine may be instantiated by a single computing device, or a machine may include two or more subcomponents instantiated by two or more different computing devices. In some implementations, a machine includes a local component (e.g., a software application executed by a computer processor) that cooperates with a remote component (e.g., a cloud computing service provided by a network of server computers). Software and / or other instructions that provide that functionality for a particular machine may optionally be stored as one or more unexecuted modules on one or more suitable storage devices.

[0070] The machine may be implemented using any suitable combination of conventional and / or future machine learning (ML), artificial intelligence (AI), and / or natural language processing (NLP) techniques. Non-limiting examples of techniques that may be incorporated into one or more machine implementations include support vector machines, multi-layer neural networks, convolutional neural networks (e.g., including spatial convolutional networks for processing images and / or video, temporal convolutional neural networks for processing audio signals and / or natural language sentences, and / or any other suitable convolutional neural network configured to convolve and pool features across one or more temporal and / or spatial dimensions), recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural Turing machines and / or neural random access memories), word embedding models (e.g., GloVe or Word2Vec), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis and / or k-means clustering), graphical models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields and / or AI knowledge bases), and / or natural language processing techniques (e.g., tokenization, stemming, composition and / or dependency analysis, and / or intent-aware segment models, and / or super-segment models (e.g., hidden dynamic models)).

[0071] In some examples, the methods and processes described herein may be implemented using one or more differentiable functions, and the gradient of the differentiable function may be calculated and / or estimated with respect to the inputs and / or outputs (e.g., training data and / or objective function) of the differentiable function. Such methods and processes may be determined at least in part by a set of trainable parameters. Thus, the trainable parameters of a particular method or process may be adjusted via any suitable training procedure to continuously improve the function of the method or process.

[0072] Non-limiting examples of training procedures for adjusting trainable parameters include supervised training (e.g., using gradient descent or any other suitable optimization scheme), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on classes derived from unsupervised clustering methods), reinforcement learning (e.g., feedback-based deep Q-learning) and / or generative adversarial neural network training methods, belief propagation, RANSAC (random sample consensus), contextual bandit methods, maximum likelihood methods, and / or expectation-maximization. In some examples, multiple methods, processes, and / or components of the systems described herein may be trained simultaneously with respect to an objective function (e.g., reinforcement feedback and / or labeled training data) that measures the performance of a collective function of the multiple components. Training multiple methods, processes, and / or components simultaneously may improve such collective function. In some examples, one or more methods, processes, and / or components may be trained independently of other components (e.g., offline training on historical data).

[0073] A language model may utilize vocabulary features to guide the sampling / searching of words for speech recognition. For example, a language model may be defined at least in part by a statistical distribution of words or other vocabulary features. For example, a language model may be defined by a statistical distribution of n-grams that define transition probabilities between candidate words according to vocabulary statistics. A language model may further be based on any other suitable statistical features and / or the results of processing the statistical features by one or more machine learning and / or statistical algorithms (e.g., confidence values ​​resulting from such processing). In some examples, a statistical model may constrain which words can be recognized for a speech signal, for example, based on an assumption that the words in the speech signal may come from a particular vocabulary.

[0074] Alternatively or additionally, the language model may be based on one or more neural networks pre-trained to represent the speech input and words in a shared latent space, e.g., a vector space learned by one or more speech and / or word models (e.g., wav2letter and / or word2vec). Thus, finding candidate words involves searching the shared latent space based on vectors encoded by the speech model for the speech input to find candidate word vectors for decoding by the word model. The shared latent space may be utilized, for one or more candidate words, to assess a confidence that the candidate word is a feature within the speech utterance.

[0075] The language model may be used in conjunction with an acoustic model configured to evaluate, for a candidate word and the speech signal, a confidence that the candidate word is included in a speech utterance in the speech signal based on acoustic features of the word (e.g., Mel-frequency cepstral coefficients, formants, etc.). Optionally, in some examples, the language model may incorporate the acoustic model (e.g., the evaluation and / or training of the language model may be based on the acoustic model). The acoustic model defines a mapping between the acoustic signal and basic speech units such as phonemes, for example, based on labeled speech utterances. The acoustic model may be based on any suitable combination of conventional or future machine learning (ML) and / or artificial intelligence (AI) models, such as deep neural networks (e.g., long short-term memory, temporal convolutional neural networks, constrained Boltzmann machines, deep belief networks), hidden Markov models (HMMs), conditional random fields (CRFs) and / or Markov random fields, Gaussian mixture models, and / or other graphical models (e.g., deep Bayesian networks). The speech signal processed by the acoustic model may be preprocessed in any suitable manner, such as encoding at any suitable sampling rate, Fourier transform, band-pass filtering, etc. The acoustic model may be trained to recognize a mapping between the acoustic signal and speech units based on training with labeled speech data. For example, the acoustic model may be trained based on labeled speech data including speech and corrected text to learn a mapping between the speech speech signal and the speech units represented by the corrected text. Thus, the acoustic model may be continuously refined to improve its usefulness for correctly recognizing speech.

[0076] In some examples, in addition to statistical models, neural networks, and / or acoustic models, the language model may incorporate any suitable graphical model, such as a hidden Markov model (HMM) or a conditional random field (CRF). The graphical model may utilize statistical features (e.g., transition probabilities) and / or confidence values ​​to determine the probability of recognizing a word given previously recognized speech and / or other words. Thus, the graphical model may utilize statistical features, pre-trained machine learning models, and / or acoustic models to define transition probabilities between states represented within the graphical model.

[0077] If included, the display subsystem 1006 may be used to present a visual representation of the data maintained by the storage subsystem 1004. This visual representation may take the form of a graphic user interface (GUI). The display subsystem 1006 may include one or more display devices utilizing nearly any type of technology. In some implementations, the display subsystem may include one or more virtual reality, augmented reality, or mixed reality displays.

[0078] If included, the input subsystem 1008 may include or interface with one or more input devices. Input devices may include sensor devices or user input devices. Examples of user input devices include a keyboard, mouse, touchscreen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and input act transformation and / or processing may be handled on-board or off-board. Exemplary NUI components may include microphones for speech and / or voice recognition, infrared cameras, color cameras, stereo cameras, and / or depth cameras for machine vision and / or gesture recognition, head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition.

[0079] If included, communications subsystem 1010 may be configured to communicatively couple computing system 1000 with one or more other computing devices. Communications subsystem 1010 may include wireless and / or wireless communication devices compatible with one or more different communications protocols. Communications subsystem may be configured for communication over personal, local, and / or wide area networks.

[0080] The methods and processes disclosed herein can be configured to give users and / or any other human beings control over any private and / or potentially sensitive data. Whenever data is accessed, stored, and / or processed, the data can be handled in accordance with privacy and / or security standards. When user data is collected, the user or other interested party can direct how the data will be used and / or stored. Whenever user data is collected for any purpose, it can only be collected with the utmost respect for user privacy (e.g., user data can only be collected when the user owning the data provides affirmative consent, and / or the user owning the data can be notified whenever user data is collected). If the data is to be released for access by persons other than the user or used for any decision-making process, the user's consent can be collected before using and / or releasing the data. The user can choose to participate in and / or opt out of data collection at any time. After the data is collected, the user can issue a command to delete the data and / or restrict access to the data. All potentially sensitive data may optionally be encrypted and / or anonymized where feasible to further protect user privacy. The user may designate portions of data, metadata, or statistics / results of processed data for release to other parties, e.g., for further processing. Private and / or confidential data may be kept completely private, e.g., only temporarily decrypted for processing, or decrypted only for processing on the user device, or otherwise stored in encrypted form. The user may retain and control the encryption keys for the encrypted data. Alternatively or additionally, the user may designate a trusted third party to retain and control the encryption keys for the encrypted data, e.g., to provide the user with access to the data according to a suitable authentication protocol.

[0081] Where the methods and processes described herein incorporate an ML and / or AI component, the ML and / or AI component may make decisions based at least in part on training of the component on training data. Thus, the ML and / or AI component may be trained on diverse and representative datasets that include sufficient relevant data for diverse users and / or user populations. In particular, the training dataset may be comprehensive with respect to different human individuals and groups, such that when the ML and / or AI component is trained, its performance is improved with respect to the user experience of the user and / or user population.

[0082] The ML and / or AI components can be further trained to make decisions that minimize potential bias toward human individuals and / or groups. For example, when an AI system is used to evaluate any qualitative and / or quantitative information about human individuals or groups, the AI ​​system can be trained to be invariant to differences between individuals or groups that are not intended to be measured by the qualitative and / or quantitative evaluation, e.g., so that any decisions are not influenced in an unintended way by differences between individuals or groups.

[0083] ML and / or AI components can be designed to provide context about how they operate, so that implementers of ML and / or AI systems can be held accountable for decisions / assessments made by the system. For example, ML and / or AI systems can be configured for reproducible behavior; for example, when making pseudo-random decisions, random seeds can be used and recorded to allow for later replay of the decisions. As another example, data used to train and / or test ML and / or AI systems can be curated and maintained to facilitate future investigation of the ML and / or AI system's behavior with respect to the data. Furthermore, ML and / or AI systems can be continuously monitored to identify potential biases, errors, and / or unintended consequences.

[0084] This disclosure is presented by way of example and with reference to the associated drawing figures. Components, process steps, and other elements that may be substantially the same in one or more of the figures are generally identified and described with minimal repetition. It should also be noted, however, that generally identified elements may differ to some extent. It should further be noted that some figures may be schematic and therefore not drawn to scale. The various depiction scales, aspect ratios, and many of the components shown in the figures may be intentionally distorted to more clearly show some features or relationships.

[0085] In one example, a computing system includes a logic subsystem and a storage subsystem that holds instructions executable by the logic subsystem to implement a transformer-based text encoder, the transformer-based text encoder including a plurality of transformer blocks pre-trained to apply encoding operations to a computer-readable text representation of an input text string, the computer-readable text representation including a computer-readable question representation of an input text question and a computer-readable passage representation of an input text passage, the plurality of transformer blocks including: a shared transformer block trained for both the computer-readable question representation and the computer-readable passage representation; and dedicated transformer blocks including two or more input-specific subnetworks and a routing function for selecting, for each of the computer-readable text representations, an input-specific subnetwork from among the two or more input-specific subnetworks. In this example or any other example, the two or more input-specific subnetworks include a question-specific subnetwork and a passage-specific subnetwork, and the routing function is a deterministic routing function that automatically routes the computer-readable question representation through the question-specific subnetwork and automatically routes the computer-readable passage representation through the passage-specific subnetwork. In this example or any other example, the routing function is a parameterized routing function trained to route computer-readable text representations such that each computer-readable question representation is routed through one or more question-specific sub-networks and each computer-readable passage representation is routed through one or more passage-specific sub-networks. In this example or any other example, training the routing function includes using entropy normalization to vary which of the two or more input-specific sub-networks is selected for each of a plurality of training computer-readable text representations that are input to the transformer-based text encoder over a series of training passes.In this example or any other example, each computer-readable text representation includes a sequence of tokens, and all tokens of a given computer-readable text representation are routed through the same input-specific subnetwork of the two or more input-specific subnetworks. In this example or any other example, each computer-readable text representation includes a sequence of tokens, and the routing function routes two or more different tokens of a given computer-readable text representation through different input-specific subnetworks of the two or more input-specific subnetworks. In this example or any other example, the shared transformer block includes a neural network parameterized with a shared set of parameters used to apply encoding operations to both the computer-readable question representation and the computer-readable passage representation. In this example or any other example, a first input-specific subnetwork of the two or more input-specific subnetworks is parameterized with a first set of parameters, and a second input-specific subnetwork of the two or more input-specific subnetworks is parameterized with a second set of parameters different from the first set of parameters. In this example or any other example, the shared transformer block is one of two or more shared transformer blocks of the transformer-based text encoder, and the dedicated transformer block is one of two or more dedicated transformer blocks of the transformer-based text encoder, and the two or more shared transformer blocks are interleaved with the two or more dedicated transformer blocks. In this example or any other example, the multiple transformer blocks of the transformer-based text encoder have a ratio of two shared transformer blocks for every one dedicated transformer block. In this example or any other example, the transformer-based text encoder is trained using a set of training text questions and a set of training text passages each containing information corresponding to the training text questions, such that each training text question is associated with one or more training text passages.In this or any other example, the transformer-based text encoder is further trained using a set of negative training text passages such that each training text question is associated with one or more negative training text passages that contain information that is irrelevant to the training text question. In this or any other example, the computer-readable text representation is an input vector representation generated by tokenizing each input text string into input tokens and embedding the value of each input token into a vector that represents the input text string.

[0086] In one example, a method for computer passage retrieval includes receiving a computer-readable question representation of an input text question; passing the computer-readable question representation to a transformer-based text encoder, the transformer-based text encoder being pre-trained to encode the computer-readable question representation and the computer-readable passage representation using a shared transformer block and a dedicated transformer block, the shared transformer block being trained for both the computer-readable question representation and the computer-readable passage representation, and the dedicated transformer block including a routing function for routing the computer-readable question representation through a question-specific sub-network and for routing the computer-readable question representation through a passage-specific sub-network; receiving an output vector question representation of the computer-readable question representation as an output of the transformer-based text encoder; performing a vector comparison between the output vector question representation and a plurality of output vector passage representations corresponding to a plurality of different pre-encoded input text passages; identifying one or more of the input text passages as being appropriate for the input text question based at least in part on the vector comparison; and outputting an indication of the one or more input text passages identified as being appropriate for the input text question. In this example or any other example, the routing function is a deterministic routing function that automatically routes computer-readable question representations through question-specific sub-networks and computer-readable passage representations through passage-specific sub-networks. In this example or any other example, the routing function is a parameterized routing function that is trained to route computer-readable question representations through question-specific sub-networks and computer-readable passage representations through passage-specific sub-networks. In this example or any other example, the shared transformer block includes a neural network parameterized with a shared set of parameters used to apply encoding operations to both the computer-readable question representations and the computer-readable passage representations.In this example or any other example, the shared transformer block is one of two or more shared transformer blocks of the transformer-based text encoder, and the dedicated transformer block is one of two or more dedicated transformer blocks of the transformer-based text encoder, and the two or more shared transformer blocks are interleaved with the two or more dedicated transformer blocks. In this example or any other example, performing the vector comparison includes calculating a relevance score between the output vector question representation and a plurality of output vector passage representations, and one or more input text passages identified as being relevant to the input text question are associated with an output vector passage representation having a relevance score higher than a threshold score.

[0087] In one example, a method for training a transformer-based text encoder implemented by a computing system includes receiving a set of training computer-readable text representations of training text strings, the training computer-readable text representations including training computer-readable question representations of training text questions and training computer-readable passage representations of training text passages, each of the training text passages including information corresponding to a training text question such that each training text question is associated with one or more training text passages; providing the training computer-readable text representations to the transformer-based text encoder over a series of training passes; and providing as output in the transformer-based text encoder a corresponding output vector representation of each of the training computer-readable text representations, the transformer-based text encoder encoding the training text questions as the output vector question representations and the training text passages. providing a plurality of transformer blocks for collectively encoding the training computer-readable text representations as output vector representations such that training text passages related to the training text questions are encoded as output vector passage representations, the plurality of transformer blocks including a shared transformer block for applying an encoding operation to both the training text questions and the training text passages, and dedicated transformer blocks including two or more input-specific sub-networks and a routing function for selecting an input-specific sub-network from the two or more input-specific sub-networks for each of the training computer-readable text representations; and modifying, before a next training pass in the series of training passes, one or more parameters of the transformer-based text encoder to increase a relevance score between the output vector question representation corresponding to the training text question and the output vector passage representation corresponding to the training text passage related to the training text question compared to a previous training pass in the series of training passes.

[0088] It is understood that the configurations and / or techniques described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various acts shown and / or described may be performed in the sequence shown and / or described, in other sequences, in parallel, or omitted. Similarly, the order of the processes described above may be altered.

[0089] The subject matter of the present disclosure includes all novel and non-obvious combinations and subcombinations of the various processes, systems, and configurations as well as other features, functions, acts, and / or properties disclosed herein and any equivalents thereof.

Claims

1. a logical subsystem; a storage subsystem holding instructions executable by said logic subsystem to implement a transformer-based text encoder; 1. A computing system comprising: a plurality of transducer blocks pre-trained to apply encoding operations to computer-readable text representations of input text strings, the computer-readable text representations including computer-readable question representations of input text questions and computer-readable passage representations of input text passages; wherein the plurality of transducer blocks include: a shared transducer block trained for both the computer-readable question representation and the computer-readable passage representation; a dedicated transducer block including two or more input-specific sub-networks and a routing function for selecting an input-specific sub-network from the two or more input-specific sub-networks for each of the computer-readable text representations; a computing system including:

2. 2. The computing system of claim 1, wherein the two or more input-specific subnetworks include a question-specific subnetwork and a passage-specific subnetwork, and the routing function is a deterministic routing function that automatically routes the computer-readable question representation through the question-specific subnetwork and automatically routes the computer-readable passage representation through the passage-specific subnetwork.

3. 2. The computing system of claim 1, wherein the routing function is a parameterized routing function trained to route the computer-readable text representations such that each computer-readable question representation is routed through one or more question-specific sub-networks and each computer-readable passage representation is routed through one or more passage-specific sub-networks.

4. 4. The computing system of claim 3, wherein training the routing function comprises using entropy normalization to vary which of the two or more input-specific sub-networks is selected for each of a plurality of training computer-readable text representations input to the transformer-based text encoder over a series of training passes.

5. 4. The computing system of claim 3, wherein each computer-readable text representation comprises a sequence of tokens, and all tokens of a given computer-readable text representation are routed through the same input-specific sub-network of the two or more input-specific sub-networks.

6. 4. The computing system of claim 3, wherein each computer-readable text representation includes a sequence of tokens, and the routing function routes two or more different tokens of a given computer-readable text representation through different input-specific subnetworks of the two or more input-specific subnetworks.

7. 2. The computing system of claim 1, wherein the shared transformer block includes a neural network parameterized with a shared set of parameters used to apply encoding operations to both the computer-readable question representation and the computer-readable passage representation.

8. 2. The computing system of claim 1, wherein a first input-specific subnetwork of the two or more input-specific subnetworks is parameterized with a first set of parameters, and a second input-specific subnetwork of the two or more input-specific subnetworks is parameterized with a second set of parameters different from the first set of parameters.

9. 2. The computing system of claim 1, wherein the shared transformer block is one of two or more shared transformer blocks of the transformer-based text encoder, and the dedicated transformer block is one of two or more dedicated transformer blocks of the transformer-based text encoder, and the two or more shared transformer blocks are interleaved with the two or more dedicated transformer blocks.

10. 10. The computing system of claim 9, wherein the plurality of transformer blocks of the transformer-based text encoder has a ratio of two shared transformer blocks for every one dedicated transformer block.

11. 2. The computing system of claim 1, wherein the transformer-based text encoder is trained using a set of training text questions and a set of training text passages each containing information corresponding to the training text questions, such that each training text question is associated with one or more training text passages.

12. 12. The computing system of claim 11, wherein the transformer-based text encoder is further trained using a set of negative training text passages such that each training text question is associated with one or more negative training text passages that contain information irrelevant to the training text question.

13. 2. The computing system of claim 1, wherein the computer-readable text representation is an input vector representation generated by tokenizing each input text string into input tokens and embedding the value of each input token into a vector representing the input text string.

14. 1. A method of computer passage retrieval, comprising: receiving a computer-readable question representation of an input text question; passing the computer-readable question representation to a transformer-based text encoder, the transformer-based text encoder being pre-trained to encode computer-readable question representations and computer-readable passage representations using a shared transformer block and a dedicated transformer block, the shared transformer block being trained for both the computer-readable question representations and the computer-readable passage representations, and the dedicated transformer block including a routing function for routing the computer-readable question representation through a question-specific sub-network and for routing the computer-readable question representation through a passage-specific sub-network; receiving an output vector question representation of the computer-readable question representation as an output of the transformer-based text encoder; performing a vector comparison between the output vector question representation and a plurality of output vector passage representations corresponding to a plurality of different pre-encoded input text passages; identifying one or more of the input text passages as being relevant to the input text question based at least in part on the vector comparison; outputting an indication of the one or more input text passages identified as being relevant to the input text question; A method comprising:

15. 15. The method of claim 14, wherein the routing function is a deterministic routing function that automatically routes the computer-readable question representations through the question-specific subnetworks and automatically routes the computer-readable passage representations through the passage-specific subnetworks.

Images