Converter-based text encoder for paragraph retrieval

Abstract

A computing system includes a logic subsystem and a storage subsystem that holds instructions executable by the logic subsystem to implement a translator-based text encoder. A translator-based text encoder includes a plurality of translator blocks previously trained to apply an encoding operation to a computer-readable text representation of an input text string, the computer-readable text representation includes a computer-readable question representation of an input text question and a computer-readable paragraph representation of an input text paragraph. The plurality of converter blocks includes a shared converter block trained for both computer-readable question representations and computer-readable paragraph representations, and a dedicated converter block comprising two or more input-specific sub-networks, and a routing function, the shared converter block being trained for both computer-readable question representations and computer-readable paragraph representations, and the dedicated converter block being trained for both computer-readable question representations and computer-readable paragraph representations. The routing function is to select an input-specific sub-network of the two or more input-specific sub-networks for each computer-readable text representation.

Description

Background Art

[0001] Passage retrieval is used to identify a passage of text, or a specific portion of text within a passage, that is relevant to an input question. Upon receiving an input question such as "Who is the current president of the United States?", a passage retrieval system evaluates a number of different passages of text (e.g., documents, web pages) to determine which passage of text is most relevant to the input question, and therefore most likely to include the requested information.

[0002] This can be accomplished using a transformer-based text encoder that converts an input text sequence (e.g., a string of letters, numbers, punctuation, and other characters) into a computer-readable representation of the text sequence that can be compared to one another. For example, the input question and each text paragraph can be encoded as a vector. The paragraph retrieval system then selects a specific paragraph that is most relevant to the input question by calculating the dot product between the vector representing the question and the vectors representing different paragraphs to determine which paragraph vector is most similar to the question vector. Summary of the invention

[0003] This summary is provided to introduce a selection of concepts further described below in the detailed description in a simplified form. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages mentioned in any part of this disclosure.

[0004] The present disclosure generally describes a converter-based text encoder that includes a shared converter block and a dedicated converter block. The dedicated converter block includes a routing function that routes input text representations between different input specific subnetworks. In one embodiment, the routing function routes question representations to the question specific subnetwork of the dedicated converter block, while paragraph representations are routed to the paragraph specific subnetwork of the dedicated converter block. The shared converter block is trained for both question representations and paragraph representations. In this way, a single converter-based text encoder can be used to encode both input text questions and input text paragraphs. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 A schematic diagram shows paragraph retrieval based on an input text question.

[0006] Figure 2 Schematic diagram of the use of a transformer-based text encoder to encode an input text string into an output vector representation.

[0007] Figure 3 An example method for computer passage retrieval is shown.

[0008] Figure 4A and Figure 4B Schematic diagram of text encoding using shared and dedicated transformer blocks of a transformer-based text encoder.

[0009] Figure 5 The routing of different tokens of the input text representation to different input-specific sub-networks of dedicated transformer blocks is schematically shown.

[0010] Figure 6 An example transformer-based text encoder is schematically illustrated, the example transformer-based text encoder comprising shared transformer blocks interleaved with dedicated transformer blocks.

[0011] Figure 7 Performing a vector comparison between an output vector question representation and a plurality of output vector paragraph representations is schematically shown.

[0012] Figure 8 An example method for training a transformer-based text encoder is shown.

[0013] Fig.9A and Fig. 9B Schematic diagram showing updating the parameters of a transformer-based text encoder during different training passes.

[0014] Fig.10 An example computing system is schematically illustrated. DETAILED DESCRIPTION

[0015] The dense retrieval framework is very useful for acquiring external knowledge in various text processing tasks. For open domain question answering (ODQA), a well-performing dense retriever is a dual encoder architecture, which includes a question encoder and a paragraph encoder. In some embodiments, the two encoders are isomorphic but separately parameterized, in that they are initialized from the same pre-trained model and then fine-tuned on the task.

[0016] The dual encoder architecture has significant drawbacks despite its usefulness. First, from an efficiency perspective, the dual encoder parameterization can cause scaling bottlenecks during both training and inference. Second, the dual encoder dense retriever is believed to perform poorly for sparse information retrieval models in various settings (such as BM25). The dual encoder dense retriever is also believed to be more sensitive to distribution shift than sparse information retrieval models, leading to poor generalization to problems with rare entities.

[0017] Therefore, the present disclosure describes a transformer-based text encoder that can be used for both input questions and input paragraphs. The transformer-based text encoder described herein includes at least one "shared" transformer block and at least one "dedicated" transformer block, rather than using two different isomorphic and decoupled encoders for questions and paragraphs. The shared transformer block is trained for both questions and paragraphs, while the dedicated transformer block includes different input-specific subnetworks for different encoding tasks. In one example, the dedicated transformer block includes a question-specific subnetwork for encoding the input question and a paragraph-specific subnetwork for encoding the input paragraph.

[0018] In some embodiments, the text encoder includes several shared converter blocks and several dedicated converter blocks interleaved with each other, for example, the output from the shared converter block is passed to the dedicated converter block, and the output of the dedicated converter block is passed to another shared converter block, etc. For simplicity, the present disclosure mainly describes the converter-based text encoder as having one shared converter block and one dedicated converter block, but it should be understood that this is not limiting, but the converter-based text encoder has at least one shared converter block and at least one dedicated converter block.

[0019] Using shared and dedicated transformer blocks in this manner provides a technical benefit of reduced computer resource consumption because at least some network parameters are shared between encoding operations applied to questions and paragraphs. This is in contrast to a dual encoder architecture in which the question encoder and paragraph encoder are parameterized separately. Furthermore, the use of transformer-based text encoders described herein provides a technical benefit of improved human-computer interaction by providing more accurate and resource-efficient paragraph retrieval for use across a wide range of question domains compared to other paragraph retrieval solutions.

[0020] Figure 1 An example paragraph retrieval scenario is schematically shown. It should be understood that Figure 1 are non-limiting and highly simplified for purposes of explanation. Figure 1 An example paragraph retrieval system 100 is schematically shown. The paragraph retrieval system can generally be used to identify a text paragraph or a specific portion of a text paragraph that is relevant to an input text question. This is accomplished via a converter-based text encoder having a shared converter block and a dedicated converter block, as will be described in more detail below.

[0021] exist Figure 1, a paragraph retrieval system receives an input text question 102. The paragraph retrieval system encodes the input text question and compares it to encoded representations of several different input text paragraphs, including paragraphs 104A, 104B, and 104C. As shown, paragraphs 104A and 104C share some similarities with the input text question, but ultimately do not include the requested information. In contrast, text paragraph 104B includes the answer to the input text question, and therefore a portion 106 of the text paragraph is retrieved by the paragraph retrieval system as a response to the input text question. In other examples, the paragraph retrieval system retrieves the entire paragraph identified as being most relevant to the input text question, rather than the entire paragraph identified as being most relevant to the input text question. Figure 1 As shown, only a portion of the paragraph is retrieved.

[0022] The paragraph retrieval system is implemented via any suitable computing system of one or more computing devices, each having any suitable capabilities, hardware configuration, and form factor. In some embodiments, the paragraph retrieval system is implemented via two or more different computing devices communicating over a network such as the Internet. As a non-limiting example, the paragraph retrieval system is implemented as follows with respect to Fig.10 A computing system 1000 is described.

[0023] It should be understood that the input text question and the input text paragraph each take any suitable form and have any suitable source. In some examples, the input text question is written by a human user, for example, typed into a search bar, or spoken aloud in a microphone and processed by speech to text. In other examples, the input text question has another suitable source, for example, loaded from a computer storage or received over a computer network. Similarly, in some cases, the input text paragraph is stored in a local storage of a paragraph retrieval computing system, for example, the input text paragraph is a text document or a database entry. Additionally or alternatively, the input text paragraph can be accessed through a computer network such as the Internet. For example, as an example, the input text paragraph takes the form of a web page accessed from a remote server. It should be understood that the present disclosure is primarily directed to performing paragraph retrieval using a converter-based text encoder, regardless of the specific source and content of the questions and paragraphs to be encoded.

[0024] Figure 2 The operation of the converter-based text encoder is schematically shown in more detail. Specifically, Figure 2 An input text string 200 is schematically shown, which generally represents an input text question and an input text paragraph. The input text string takes the form of any suitable sequence of text characters, including any or all of letters, numbers, spaces, punctuation marks, etc.

[0025] The input text string is converted into a computer-readable text representation 202 before being provided to a text encoder based on a converter. As a non-limiting example, the computer-readable text representation takes the form of an input vector representation generated by tokenizing the input text string into a plurality of input tags, wherein each tag represents a different character or character sequence of the input text string. The value corresponding to each input tag is then embedded in the vector representing the input text string, and the input vector representation is passed to the text encoder based on the converter for encoding into an output vector representation. In this way, the input text question can be converted into a computer-readable question representation, and the input text paragraph can be converted into a computer-readable paragraph representation. It should be understood that in other examples, the computer-readable text representation takes other suitable forms.

[0026] continue Figure 2 , the computer readable text representation 202 is input to a converter-based text encoder 204. The converter-based text encoder includes a plurality of converter blocks 206 previously trained to apply encoding operations to computer readable text representations. The plurality of converter blocks of the converter-based text encoder include at least a shared converter block 208 and a dedicated converter block 210. The shared converter block is trained for both computer readable question representations and computer readable paragraph representations. The dedicated converter block includes two or more different input specific subnetworks, for example, a question specific subnetwork and a paragraph specific subnetwork. More specific details about the shared converter block and the dedicated converter block will be given below.

[0027] Typically, a transformer-based text encoder according to the present invention is constructed with multiple transformer blocks. In some embodiments, each transformer block includes a multi-head self-attention (MHA) sublayer and at least one feed-forward network (FFN) sublayer, where residual connections and layer normalization are applied to both sublayers. Given an input vector The FFN sublayer produces the following output vector:

[0028] FFN(h)=W2max{0,W1h+b1}+b2

[0029] in, and is a learnable parameter. For a sequence of N tokens, each transformer block generates N corresponding vectors and a vector for a special prefix token [CLS], which can be used as a representation of the sequence. In other embodiments, the transformer block is implemented in other suitable ways. Fig.10 Describes non-limiting examples of suitable machine learning (ML) and / or artificial intelligence (AI) techniques that may be used to implement the converter block.

[0030] After the transformer block applies the encoding operation, the transformer-based text encoder outputs the encoded version of the input text string as an output vector representation 212. As will be described in more detail below, different vector representations output by the transformer-based text encoder are compared to each other, for example, to determine which text paragraph among a number of text paragraphs is most relevant to the input text question. In some embodiments, the transformer-based text encoder uses the final output vector for [CLS] from the corresponding encoder to encode the question and the paragraph into d-dimensional vectors, represented as and The relevance between the question and the paragraph can then be measured in vector space by computing the dot product between corresponding vectors:

[0031] sim(q,p)=qTp.

[0032] Figure 3 An example method 300 for computer passage retrieval is shown. Method 300 is implemented by any suitable computing system of one or more computing devices. Any computing device implementing method 300 may have any suitable capabilities, hardware configurations, and form factors. In some embodiments, different steps of method 300 are performed by two or more different computing devices communicating over a computer network. As a non-limiting example, method 300 is implemented by the following description of Fig.10 The computing system 1000 described herein implements.

[0033] At 302, method 300 includes receiving a computer-readable question representation representing a question for an input text. This may be substantially as described above with respect to Figure 1 and 2 As a non-limiting example, a human user writes an input text question, which is then converted into an input vector representation by tokenizing the input question into a sequence of input tokens. The values ​​representing the input tokens are then embedded into the vector representing the input question.

[0034] At 304, method 300 includes passing the computer-readable question representation to a transformer-based text encoder. The transformer-based text encoder is previously trained to encode the computer-readable question representation and the computer-readable paragraph representation using shared transformer blocks and dedicated transformer blocks.

[0035] This is relative to Figure 4A and 4B Specifically, Figure 4A The schematic diagram shows an input text question 400Q converted into a computer-readable question representation 402Q and an input text paragraph 400P converted into a computer-readable paragraph representation 402P. Both the question representation 402Q and the paragraph representation 402P are input into Figure 2208 of the transformer-based text encoder 204 shown in FIG.

[0036] The shared transformer block is trained to perform encoding operations for both a computer-readable question representation and a computer-readable paragraph representation. In some cases, the shared transformer block is the first transformer block of a transformer-based text encoder. In other cases, the shared transformer block receives a text representation that has been at least partially encoded by one or more previous transformer blocks of the transformer-based text encoder. In the case where the shared transformer block is the last transformer block of the text encoder, the output from the shared transformer block is then passed to the next transformer block of the transformer-based text encoder, or alternatively output as an output vector representation of the input text string.

[0037] Figure 4A It seems that the computer-readable problem representation and the computer-readable paragraph representation are both provided to the shared converter block at the same time as illustrated. However, it should be understood that this is not necessarily the case. On the contrary, in some embodiments, the computer-readable problem representation and the computer-readable paragraph representation are provided to the text encoder based on the converter at different times. In some cases, the computer-readable problem representation and the computer-readable paragraph representation are encoded simultaneously by different instances of the text encoder based on the converter, which are realized by the same or different computing devices.

[0038] exist Figure 4A In , the shared transformer block performs the encoding operation via a neural network 404. As an example, this takes the form of a feed-forward network (FFN). The neural network is parameterized using a shared parameter set 406, which is used to apply the encoding operation to both the computer-readable question representation and the computer-readable paragraph representation. This is in contrast to a dual encoder architecture, in which two independent parameter sets are used for separate question encoders and paragraph encoders. Therefore, the use of shared transformer blocks in a transformer-based text encoder provides a technical benefit of saving computational resources by reducing the number of parameters used to perform text encoding and paragraph retrieval. Reference will be made below to Figures 8 to 9B An example method is given for training a shared transformer block and thereby changing a set of shared parameters used in a neural network.

[0039] The transformer-based text encoder also includes dedicated transformer blocks for applying the encoding operation to question representations and paragraph representations. Figure 4BSchematically illustrated, a dedicated converter block 210 of a converter-based text encoder 204 is shown. As with the shared converter block, in some embodiments, the dedicated converter block is the first converter block of the converter-based text encoder. In other embodiments, the dedicated converter block receives a text representation that has been at least partially encoded by one or more previous converter blocks of the converter-based text encoder, e.g., the dedicated converter block receives the output from a previous shared converter block. The output from the dedicated converter block is then passed to the next converter block of the converter-based text encoder, or alternatively output as an output vector representation of the input text string, in the case where the dedicated converter block is the last converter block of the text encoder.

[0040] Typically, the dedicated converter block includes two or more input specific subnetworks, and a routing function that selects an input specific subnetwork from the two or more input specific subnetworks for each computer readable text representation. Figure 4B , the dedicated converter block includes routing functionality 408, question-specific subnetwork 410Q, and paragraph-specific subnetwork 410P. In some embodiments, the dedicated converter block may be described as using a Mixture of Experts (MOE) approach, where multiple expert FFN sublayers are used in parallel and the routing functionality is used to select among these expert FFN sublayers.

[0041] To this end, in some embodiments, different input-specific subnetworks are separately parameterized to apply different encoding operations to different types of input text representations. In other words, a first input-specific subnetwork is parameterized using a first set of parameters, and a second input-specific subnetwork is parameterized using a second set of parameters that are different from the first set of parameters. This provides a technical benefit of improving the performance of transformer-based text encoders on both question representations and paragraph representations, thereby improving human-computer interaction.

[0042] The dedicated transformer block includes a routing function that routes different text representations to different input-specific subnetworks. Figure 4B In the example of , the routing function routes the computer readable question representation through the question specific subnetwork, and routes the computer readable paragraph representation through the paragraph specific subnetwork. In other words, the routing function is a deterministic routing function (Det-R) that automatically routes the computer readable question representation through the question specific subnetwork, and automatically routes the computer readable paragraph representation through the paragraph specific subnetwork.

[0043] In other examples, other suitable routing functions are used. As additional non-limiting examples, the routing function uses sequence-based routing (Seq-R) or tag-based routing (Tok-R). In the case of Seq-R and Tok-R, the routing function is a parameterized routing function trained to route computer-readable text representations such that each computer-readable question representation is routed through one or more question-specific subnetworks, and each computer-readable paragraph representation is routed through one or more paragraph-specific subnetworks.

[0044] Such a parameterized routing function can be expressed as:

[0045] R(u) = GumbelSoftmax(Au+c),

[0046] Among them, GumbelSoftmax is based on and The parameterized linear projection outputs an I-dimensional one-hot vector, where I is the number of input-specific subnetworks in the dedicated transformer block, and is the input of the routing function. In some embodiments, the routing function is jointly learned with other network parameters of the transformer-based text encoder using discrete reparameterization.

[0047] As described above, in some embodiments, the computer-readable text representation input to the transformer-based text encoder includes multiple input tokens. In the case of Seq-R, routing is performed at the sequence level, and all tokens in the sequence share the same u, which is the FFN input vector h representing the special prefix token [CLS] [cls] In other words, each computer-readable text representation comprises a sequence of tokens, and each token of a given computer-readable text representation is routed through the same input-specific subnetwork of the two or more input-specific subnetworks via a pre-trained and parameterized routing process.

[0048] In contrast, in the case of Tok-R, the routing function routes each token independently, e.g., for the jth token in the sequence, u is set to the corresponding FFN input vector h j In other words, each computer-readable text representation comprises a sequence of tokens, and the routing function routes two or more different tokens of a given computer-readable text representation through different ones of the two or more input-specific subnetworks.

[0049] about Figure 5The scenario is schematically illustrated, showing an example computer-readable text representation 500. Representation 500 includes a sequence of tokens 502A to 502C. As described above, in some embodiments, a computer-readable text representation is generated by applying a tokenization process to an input text string, which converts individual characters or strings of input text into a sequence of token representations. The values ​​representing the tokens can then be embedded into the input vector representation.

[0050] The computer-readable text representation 500 is input to a dedicated converter block 504, which is one of a plurality of different converter blocks of a converter-based text encoder. The routing function 506 of the dedicated converter block routes the tokens of the computer-readable text representation between different input-specific subnetworks 508A to 508C of the dedicated converter block. In this example, the routing function uses Tok-R routing, in which different tokens of the same text representation are routed to different subnetworks of the dedicated converter block. For example, token 502A is routed to subnetwork 508A, while tokens 508B and 508C are routed to subnetworks 508B and 508C. This provides a technical benefit of improving the performance of the converter-based text encoder, because the output vector representation generated for the input text string will more accurately capture the relationship and context between different input characters and words of the input text string.

[0051] As described above, the present disclosure mainly describes a converter-based text encoder as having one shared converter block and one dedicated converter block. It should be understood that this is only for simplicity. On the contrary, in some embodiments, the "shared converter block" described herein is one of two or more shared converter blocks of a converter-based text encoder, and the "dedicated converter block" is one of two or more dedicated converter blocks of a converter-based text encoder. More specifically, in some embodiments, two or more shared converter blocks are interleaved with two or more dedicated converter blocks.

[0052] about Figure 6 Such a scenario is schematically illustrated, which shows another example converter-based text encoder 600. The converter-based text encoder 600 includes at least four shared converter blocks 602A to 602D and at least two dedicated converter blocks 604A and 604B. In addition, the shared and dedicated converter blocks are interleaved so that the output of the shared converter block 602B is passed to the dedicated converter block 604A, and the output of the dedicated converter block 604A is passed to the shared converter block 602C, and so on. This interleaved arrangement provides technical benefits of improved human-computer interaction by improving the encoding and paragraph retrieval performance of the converter-based text encoder.

[0053] exist Figure 6In the example of , the converter blocks have a ratio of two shared converter blocks for each dedicated converter block. In other examples, other suitable ratios are used. Furthermore, in other examples, block interleaving need not be used. Instead, for example, another example scenario uses a set of two or more shared converter blocks followed by a set of two or more dedicated converter blocks before a converter-based text encoder outputs a vector representation of an input text string.

[0054] Return briefly Figure 3 At 306, method 300 includes receiving an output vector representation of a computer-readable question representation. As described above, in some embodiments, the transformer-based text encoder uses the final output vector of [CLS] from the corresponding encoder to encode the question and the paragraph into d-dimensional vectors, represented as and In other embodiments, the vector representation output by the transformer-based text encoder takes other suitable forms.

[0055] continue Figure 3 At 308, method 300 includes performing a vector comparison between the output vector question representation and the plurality of output vector paragraph representations. This enables the computing system to evaluate the relevance between the input text question and the plurality of input text paragraphs that are also encoded via the transformer-based text encoder. As a non-limiting example, the output vector question representation and the output vector paragraph representation are compared in vector space by computing a dot product between corresponding vectors. More generally, in some embodiments, performing the vector comparison includes computing a relevance score between the output vector question representation and the plurality of output vector paragraph representations, wherein the dot product is a suitable example of a relevance score.

[0056] At 310, method 300 includes identifying one or more input text paragraphs as being relevant to the input text question. As an example, this includes identifying a text paragraph having the highest relevance score to the input text question, for example, identifying an output vector paragraph representation having the maximum dot product with the output vector question representation. Additionally or alternatively, in some scenarios, no encoded text paragraph may be relevant to the input text question. Therefore, in some examples, one or more input text paragraphs identified as being relevant to the input text question are associated with an output vector paragraph representation having a relevance score above a threshold score. In the event that no output vector paragraph representation has a relevance score above the threshold score, then no input text paragraph is identified as being relevant to the input text question. In various examples, any suitable relevance score threshold may be used, depending on the particular manner in which the relevance score is calculated, and depending on the desired confidence that the identified text paragraph is actually relevant to the input text question.

[0057] about Figure 7The process is schematically illustrated, schematically showing an example computer-readable question representation 700. Representation 700 is input to a transformer-based text encoder 702, which encodes the question representation into an output vector question representation 704 as described above. The computing system then performs a vector comparison 706 between the output vector question representation and a plurality of output vector paragraph representations 708A to D, which correspond to input text paragraphs previously encoded by the transformer-based text encoder. As a result of the vector comparison, an output vector paragraph representation 706C is identified as being relevant to the input text question based on a relevance score (e.g., maximum dot product) calculated via the vector comparison.

[0058] Return briefly Figure 3 At 312, method 300 includes outputting an indication of one or more input text paragraphs identified as being relevant to the input text question. Depending on the implementation, such indications may take any suitable form. In one example, the indication specifies a unique identifier or address (e.g., a file system address, a network address) of the text paragraph identified as being relevant to the question. As another example, the indication includes duplication or recreation of at least a portion of the text paragraph identified as being relevant, e.g., the entire text paragraph is presented for viewing, or only a portion of the text paragraph that provides information relevant to the input text question.

[0059] At 314, method 300 includes determining whether there are additional input text questions for encoding via the converter-based text encoder. If so, method 300 returns to step 302 to encode the next input text question. If not, method 300 ends until another input text question is provided for encoding.

[0060] The present disclosure has so far focused on the runtime use of a converter-based text encoder. However, prior to such runtime use, it is first trained on input training examples. In a series of training processes, the network parameters of the converter-based text encoder are changed to improve the performance of the encoder. In some embodiments, the converter-based text encoder is trained using a set of training text questions and a set of training text paragraphs, each training text paragraph including information corresponding to the training text question so that each training text question is associated with one or more training text paragraphs. In some embodiments, the converter-based text encoder is further trained using a set of negative training text paragraphs so that each training text question is associated with one or more negative training text paragraphs that include information unrelated to the training text question.

[0061] Figure 8An example method 800 for training a transformer-based text encoder to encode an input text question and an input text paragraph is shown. As with method 300, method 800 is implemented via any suitable computing system of one or more computing devices. Any computing device implementing method 800 may have any suitable capabilities, hardware configurations, and form factors. In some embodiments, different steps of method 800 are performed by two or more different computing devices communicating over a computer network. As a non-limiting example, method 800 is implemented by the following description of Fig.10 The computing system 1000 described herein implements.

[0062] At 802, method 800 includes receiving a set of training computer-readable text representations representing training text strings. These training computer-readable text representations include training computer-readable question representations representing training text problems and training computer-readable paragraph representations representing training text paragraphs. It is worth noting that in this case, it is known that the training text paragraphs include information corresponding to the training text problems so that each training text problem is associated with one or more training text paragraphs. In this way, the performance of the text encoder based on the converter can be evaluated during training by comparing the output vector representation corresponding to the input text problem with the input paragraph problem known to include information related to the input text problem.

[0063] At 804, method 800 includes providing training computer-readable text representations to a text encoder based on a converter through a series of training passes. At 806, method 800 includes outputting corresponding output vector representations for each training computer-readable text representation in the training computer-readable text representations at a text encoder based on a converter. The text encoder based on a converter is implemented substantially as described above, except that the encoder is still being trained, and therefore its performance may not be sufficient for actual use. Like this, the text encoder based on a converter includes a plurality of converter blocks to encode the training computer-readable text representations into output vector representations together, so that the training text problems are encoded into output vector problem representations, and the training text paragraphs are encoded into output vector paragraph representations.

[0064] As described above, the plurality of transformer blocks include shared transformer blocks and dedicated transformer blocks. More specifically, the shared transformer blocks are used to apply encoding operations to both training text questions and training text paragraphs. The dedicated transformer blocks include two or more input-specific subnetworks, and a routing function that selects an input-specific subnetwork of the two or more input-specific subnetworks for each training computer-readable text representation.

[0065] This is about Fig.9A and Fig. 9B Specifically, Fig.9AThe training of a transformer-based text encoder is shown during one of a series of training stages. Fig.9A , a set of training text representations 900 includes training computer-readable question representations 902Q and training computer-readable paragraph representations 902P. As described above, they represent corresponding input text questions and input paragraph representations. However, in this case, it is known that the training paragraphs include information related to the training questions. In some cases, the training questions are also paired with one or more negative paragraph examples that are unrelated to the training questions. It will be understood that a set of training text representations can include any suitable number of different training question representations, each training question representation being associated with any suitable number of different training paragraph representations.

[0066] The training text representation is input to a transformer-based text encoder 904. Fig.9A , the transformer-based text encoder is parameterized by a set of initial network parameters 906 A. These generally represent any or all parameters of the different transformer blocks of the transformer-based text encoder, for example, parameters for different shared transformer blocks, parameters for different input-specific subnetworks of dedicated transformer blocks, and parameters for routing functions of dedicated transformer blocks in the case where Seq-R or Tok-R routing functions are used.

[0067] The transformer-based text encoder outputs a vector question representation 908Q and a vector paragraph representation 908P corresponding to the input training question representation 902Q and the input training paragraph representation 902P. The computing system then calculates a correlation score 910A between the vector question representation and the vector paragraph representation. As described above, the correlation score is used as an indicator of the performance of the encoder during training because it is known that the training text paragraphs include information related to the input text paragraphs.

[0068] Return briefly Figure 8 At 808, method 800 includes changing one or more parameters of a transformer-based text encoder prior to a next training pass in a series of training passes. This is done to increase a correlation score between an output vector question representation corresponding to a training text question (e.g., vector representation 908Q) and an output vector paragraph representation corresponding to a training text paragraph associated with the training text question (e.g., vector representation 908P) as compared to a previous training pass in the series of training passes.

[0069] This is Fig. 9B , again showing a training text representation 900 being provided to a transformer-based text encoder 904. Fig.9A compared to, Fig. 9B represents a later training pass in a series of training passes. Fig.9A and Fig. 9B , the parameters of the transformer-based text encoder have been updated to an updated set of parameters 906B. This affects the performance of the transformer-based text encoder, causing the encoder to output an updated vector question representation 912Q and an updated vector paragraph representation 912P. These updated vector question representations and updated vector paragraph representations are compared to give an updated relevance score 910B, which is higher than the relevance score 910A.

[0070] The specific training process for changing the parameters of the converter-based text encoder varies depending on the implementation. In various embodiments, a single-set or multi-set training process is used. In single-set training, only a single data set is used to train the converter-based text encoder, and then evaluated on the same data set. This provides the technical benefit of training the converter-based text encoder faster and more efficiently, especially when the encoder is intended for runtime use in a relatively specific domain captured by the data set. However, single-set training may have a negative impact on the performance of the encoder on other data sets for which it has not been trained. In contrast, in multi-set training, the encoder is trained by combining training data from multiple data sets. This provides the technical benefit of improving the performance of the encoder across a wider range of problems and paragraph domains.

[0071] In some embodiments, during training, a transformer-based text encoder is optimized based on a contrastive learning objective:

[0072]

[0073] Among them, p + is the relevant (positive) paragraph for the given input question, and p - is the set of irrelevant (negative) paragraphs. All paragraphs are pre-converted into vectors using a paragraph encoder. Then, each incoming question is encoded using a text encoder and the top K lists of the most relevant paragraphs are retrieved based on their relevance scores with respect to the question.

[0074] In this case, L sim Use a set of negative paragraphs for each question - There are several constructions of p - As a non-limiting example, two negative paragraphs are used per question: one negative paragraph is a preceding paragraph retrieved by BM25 that does not include an answer to the question, and the other negative paragraph is selected from the ground truth positive paragraphs for other questions in the same mini-batch.

[0075] It has been shown that mining more difficult negative examples using iterative training can lead to better performance. Therefore, in some embodiments, difficulty mining is used to train a transformer-based text encoder. Specifically, the encoder uses the first set of negative paragraphs p1 generated as described above - Then, a second set of negative paragraphs p2 is generated by retrieving the top 100 paragraphs (or another suitable number of top-ranked paragraphs) for each question (excluding the standard answer paragraphs). - In a single-episode training scenario, the combination p1 - and p2 - To train the encoder. In the multi-set training scenario, only p2 is used - to train the encoder, which provides the technical benefit of improving training efficiency.

[0076] As described above, in some cases, the routing function used by the dedicated transformer block of the transformer-based text encoder is a trained parameterized routing function. In some embodiments, the above contrastive learning objective L is used. sim to train such parameterized routing functions. In the case of Seq-R and Tok-R routing functions, entropy regularization is applied to avoid routing all inputs to the same expert FFN sublayer. In other words, training the routing function includes using entropy regularization to change which of two or more input-specific subnetworks is selected for each of a plurality of training computer-readable text representations input to the transformer-based text encoder over a series of training passes.

[0077] This can be calculated by ent To complete:

[0078]

[0079] Where P(i) = Softmax(Ah+c) i is the probability that the i-th expert FFN sublayer is selected. Therefore, the joint training objective is:

[0080] L joint =L sim +βL ent

[0081] Where β is a scalar hyperparameter. As a non-limiting example, β = 0.01

[0082] It should be understood that the above training process is a non-limiting example. Fig.10 Describe other examples of suitable training procedures.

[0083] The methods and processes described herein may be bound to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as an executable computer application, a network accessible computing service, an application programming interface (API), a library, or a combination of the above and / or other computing resources.

[0084] Fig.10 Schematically illustrated is a simplified representation of a computing system 1000 configured to provide any or all of the computing functionality described herein. The computing system 1000 may take the form of one or more personal computers, network accessible server computers, tablet computers, home entertainment computers, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), virtual / augmented / mixed reality computing devices, wearable computing devices, Internet of Things (IoT) devices, embedded computing devices, and / or other computing devices.

[0085] The computing system 1000 includes a logic subsystem 1002 and a storage subsystem 1004. The computing system 1000 may optionally include a display subsystem 1006, an input subsystem 1008, a communication subsystem 1010, and / or Fig.10 Other subsystems not shown.

[0086] The logic subsystem 1002 includes one or more physical devices configured to execute instructions. For example, the logic subsystem may be configured to execute instructions as part of one or more applications, services, or other logical constructs. The logic subsystem may include one or more hardware processors configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware devices configured to execute hardware or firmware instructions. The processor of the logic subsystem may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The various components of the logic subsystem may optionally be distributed in two or more separate devices, which may be remotely located and / or configured for coordinated processing. Aspects of the logic subsystem may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0087] The storage subsystem 1004 includes one or more physical devices configured to temporarily and / or permanently hold computer information such as data and instructions that can be executed by the logic subsystem. When the storage subsystem includes two or more devices, the devices can be collocated and / or remotely located. The storage subsystem 1004 can include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices. The storage subsystem 1004 can include removable and / or built-in devices. When the logic subsystem executes instructions, the state of the storage subsystem 1004 can be converted, for example, to hold different data.

[0088] Aspects of the logic subsystem 1002 and the storage subsystem 1004 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include programmable and application specific integrated circuits (PASIC / ASIC), programmable and application specific standard products (PSS / ASSP), systems on chips (SOC), and complex programmable logic devices (CPLD).

[0089] The logic subsystem and the storage subsystem can collaborate to instantiate one or more logic machines. As used herein, the term "machine" is used to collectively refer to a combination of hardware, firmware, software, instructions, and / or any other components that collaborate to provide computer functions. In other words, a "machine" is never an abstract concept and always has a tangible form. A machine can be instantiated by a single computing device, or a machine can include two or more subcomponents instantiated by two or more different computing devices. In some embodiments, a machine includes a local component (e.g., a software application executed by a computer processor) that collaborates with a remote component (e.g., a cloud computing service provided by a server computer network). The software and / or other instructions that give a particular machine its functionality can optionally be stored on one or more suitable storage devices as one or more unexecuted modules.

[0090] The machine may be implemented using any suitable combination of existing and / or future machine learning (ML), artificial intelligence (AI), and / or natural language processing (NLP) techniques. Non-limiting examples of techniques that may be incorporated into embodiments of one or more machines include support vector machines, multi-layer neural networks, convolutional neural networks (e.g., including spatial convolutional networks for processing images and / or videos, temporal convolutional neural networks for processing audio signals and / or natural language sentences, and / or any other suitable convolutional neural network configured to convolve and pool features across one or more temporal and / or spatial dimensions), recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural Turing machines, and / or neural random access memories), word embedding models (e.g., GloVe or Word2Vec), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and / or k-means clustering), graphical models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields, and / or AI knowledge bases), and / or natural language processing techniques (e.g., tokenization, stemming, composition and / or dependency parsing, and / or intent recognition, segmentation models, and / or super-segmentation models (e.g., hidden dynamic models)).

[0091] In some examples, the methods and processes described herein can be implemented using one or more differentiable functions, wherein the gradient of the differentiable function can be calculated and / or estimated with respect to the input and / or output of the differentiable function (e.g., with respect to training data and / or with respect to an objective function). Such methods and processes can be determined, at least in part, by a set of trainable parameters. Thus, the trainable parameters for a particular method or process can be adjusted by any suitable training process to continuously improve the functionality of the method or process.

[0092] Non-limiting examples of training processes for adjusting trainable parameters include supervised training (e.g., using gradient descent or any other suitable optimization method), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on categories derived from unsupervised clustering methods), reinforcement learning (e.g., feedback-based deep Q-learning), and / or generative adversarial neural network training methods, belief propagation, RANSAC (random sample consensus), contextual strong-weak methods, maximum likelihood methods, and / or expectation maximization. In some examples, multiple methods, processes, and / or components of the systems described herein can be trained simultaneously with respect to an objective function that measures the performance of the collective function of multiple components (e.g., with respect to reinforced feedback and / or with respect to labeled training data). Training multiple methods, processes, and / or components simultaneously can improve such collective function. In some examples, one or more methods, processes, and / or components can be trained independently of other components (e.g., offline training on historical data).

[0093] The language model can use vocabulary features to guide the sampling / search of words to recognize speech. For example, the language model can be defined at least in part by the statistical distribution of words or other vocabulary features. For example, the language model can be defined by the statistical distribution of n-grams, defining the transition probabilities between candidate words based on vocabulary statistics. The language model can also be based on any other suitable statistical features, and / or the results of processing statistical features with one or more machine learning and / or statistical algorithms (e.g., confidence values ​​generated by such processing). In some examples, the statistical model can constrain what words can be recognized for the audio signal, for example based on the assumption that the words in the audio signal are from a specific vocabulary.

[0094] Alternatively or additionally, the language model can be based on one or more neural networks previously trained to represent audio input and words in a shared latent space, for example, a vector space learned by one or more audio and / or word models (e.g., wav2letter and / or word2vec). Thus, finding a candidate word can include searching a shared latent space based on a vector encoded by an audio model to obtain the audio input in order to find a candidate word vector for decoding using a word model. The shared latent space can be used to evaluate, for one or more candidate words, a confidence that the candidate word has a feature in the speech audio.

[0095] The language model can be used in conjunction with an acoustic model, which is configured to evaluate the confidence of the candidate word in the speech audio included in the audio signal for the candidate word and the audio signal based on the acoustic features of the word (e.g., Mel-frequency cepstral coefficients, formants, etc.). Optionally, in some examples, the language model can be combined with the acoustic model (e.g., the evaluation and / or training of the language model can be based on the acoustic model). The acoustic model defines the mapping between the acoustic signal and the basic sound unit such as the phoneme based on the labeled speech audio. The acoustic model can be based on any suitable combination of existing or future machine learning (ML) and / or artificial intelligence (AI) models, such as: deep neural networks (e.g., long short-term memory, temporal convolutional neural networks, restricted Boltzmann machines, deep belief networks), hidden Markov models (HMMs), conditional random fields (CRFs) and / or Markov random fields, Gaussian mixture models and / or other graphical models (e.g., deep Bayesian networks). The audio signal to be processed using the acoustic model can be preprocessed in any suitable manner, for example, encoded at any suitable sampling rate, Fourier transformed, bandpass filtered, etc. The acoustic model can be trained to identify the mapping between the acoustic signal and the sound unit based on training using labeled audio data. For example, the acoustic model can be trained based on labeled audio data including speech audio and correction text to learn the mapping between the speech audio signal and the sound unit represented by the correction text. Therefore, the acoustic model can be continuously improved to improve its utility for correctly identifying speech audio.

[0096] In some examples, in addition to statistical models, neural networks, and / or acoustic models, the language model can incorporate any suitable graphical model, such as a hidden Markov model (HMM) or a conditional random field (CRF). The graphical model can utilize statistical features (e.g., transition probabilities) and / or confidence values ​​to determine the probability of recognizing a word given the speech audio and / or other words recognized to date. Thus, the graphical model can utilize statistical features, previously trained machine learning models, and / or acoustic models to define transition probabilities between states represented in the graphical model.

[0097] When display subsystem 1006 is included, it can be used to present a visual representation of the data maintained by storage subsystem 1004. The visual representation can take the form of a graphical user interface (GUI). Display subsystem 1006 can include one or more display devices utilizing nearly any type of technology. In some embodiments, the display subsystem may include one or more virtual reality, augmented reality, or mixed reality displays.

[0098] When input subsystem 1008 is included, it may include or interface with one or more input devices. An input device may include a sensor device or a user input device. Examples of user input devices include a keyboard, a mouse, a touch screen, or a game controller. In some embodiments, the input subsystem may include or interface with a selected natural user input (NUI) component. Such components may be integrated or peripheral, and the conversion and / or processing of input actions may be processed on-board or off-board. Example NUI components may include microphones for voice and / or speech recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition.

[0099] When included, the communication subsystem 1010 can be configured to communicatively couple the computing system 1000 with one or more other computing devices. The communication subsystem 1010 can include wired and / or wireless communication devices compatible with one or more different communication protocols. The communication subsystem can be configured to communicate via a personal area network, a local area network, and / or a wide area network.

[0100] The methods and processes disclosed herein can be configured to give users and / or any other human control over any privacy and / or potentially sensitive data. Whenever data is stored, accessed and / or processed, data can be processed according to privacy and / or security standards. When collecting user data, users or other stakeholders can specify how to use and / or store data. Whenever user data is collected for any purpose, only the most important user data about user privacy can be collected (for example, user data can be collected only when the user with the data provides affirmative consent, and / or the user with the data can be notified whenever the user data is collected). If the data is to be made public for anyone other than the user to access or for any decision-making process, user consent can be collected before using and / or disclosing the data. Users can choose to join and / or opt out of data collection at any time. After the data has been collected, users can issue commands to delete the data and / or restrict access to the data. All potentially sensitive data can optionally be encrypted and / or anonymized when feasible to further protect user privacy. Users can specify data, metadata, or parts of the statistics / results of processing data to be published to other parties, such as for further processing. Private and / or confidential data may be kept completely private, e.g., only temporarily decrypted for processing, or only decrypted for processing on a user device, and stored in encrypted form. The user may retain and control the encryption key for the encrypted data. Alternatively or additionally, the user may designate a trusted third party to retain and control the encryption key used to encrypt the data, e.g., to provide the user with access to the data in accordance with an appropriate authentication protocol.

[0101] When the methods and processes described herein incorporate ML and / or AI components, the ML and / or AI components may make decisions based at least in part on the training of the components on training data. Thus, the ML and / or AI components may be trained on different representative data sets that include sufficient relevant data for different users and / or user groups. In particular, the training data sets may be inclusive for different human individuals and groups, such that when the ML and / or AI components are trained, their performance with respect to the user experience of the users and / or user groups is improved.

[0102] ML and / or AI components may additionally be trained to make decisions so as to minimize potential bias against human individuals and / or groups. For example, when AI systems are used to evaluate any qualitative and / or quantitative information about human individuals or groups, they may be trained so as to be invariant to differences between individuals or groups that are not intended to be measured by the qualitative and / or quantitative evaluations, e.g., so that any decision is not affected in an unintended manner by differences between individuals and groups.

[0103] ML and / or AI components can be designed to provide context about how they operate so that implementers of ML and / or AI systems can be responsible for decisions / evaluations made by the systems. For example, ML and / or AI systems can be configured for replicable behavior, e.g., when they make pseudo-random decisions, a random seed can be used and recorded to enable the decision to be replicated later. As another example, data used to train and / or test ML and / or AI systems can be curated and maintained to facilitate future investigations of the behavior of ML and / or AI systems with respect to the data. Furthermore, ML and / or AI systems can be continuously monitored to identify potential biases, errors, and / or unintended outcomes.

[0104] The present disclosure is presented by way of example and with reference to the associated drawings. Components, process steps and other elements that may be substantially the same in one or more of the drawings are identified equally and described with minimal repetition. However, it should be noted that the elements identified equally may also be different to some extent. It will be further noted that some of the figures may be schematic and not drawn to scale. The various drawing scales, aspect ratios and number of parts shown in the drawings may be intentionally deformed to make certain features or relationships easier to see.

[0105] In an example, a computing system includes: a logic subsystem; and a storage subsystem, the storage subsystem storing instructions, the instructions executable by the logic subsystem to implement a converter-based text encoder, the converter-based text encoder including: a plurality of converter blocks, the plurality of converter blocks previously trained to apply encoding operations to computer-readable text representations of input text strings, the computer-readable text representations including computer-readable question representations of input text questions and computer-readable paragraph representations of input text paragraphs, the plurality of converter blocks including: a shared converter block, the shared converter block being trained for both the computer-readable question representations and the computer-readable paragraph representations; and a dedicated converter block, the dedicated converter block including two or more input-specific subnetworks, and a routing function, the routing function being used to select an input-specific subnetwork of the two or more input-specific subnetworks for each computer-readable text representation in the computer-readable text representation. In this example or any other example, the two or more input-specific subnetworks include a question-specific subnetwork and a paragraph-specific subnetwork, and the routing function is a deterministic routing function, the deterministic routing function automatically routes the computer-readable question representation through the question-specific subnetwork, and automatically routes the computer-readable paragraph representation through the paragraph-specific subnetwork. In this example or any other example, the routing function is a parameterized routing function that is trained to route computer-readable text representations such that each computer-readable question representation is routed through one or more question-specific subnetworks and each computer-readable paragraph representation is routed through one or more paragraph-specific subnetworks. In this example or any other example, training the routing function includes using entropy regularization to change which input-specific subnetwork of two or more input-specific subnetworks is selected for each of a plurality of training computer-readable text representations input to a transformer-based text encoder over a series of training passes. In this example or any other example, each computer-readable text representation includes a sequence of tokens, and each token of a given computer-readable text representation is routed through the same input-specific subnetwork of two or more input-specific subnetworks. In this example or any other example, each computer-readable text representation includes a sequence of tokens, and the routing function routes two or more different tokens of a given computer-readable text representation through different input-specific subnetworks of two or more input-specific subnetworks. In this example or any other example, the shared transformer block includes a neural network parameterized using a shared parameter set for applying encoding operations to both computer-readable question representations and computer-readable paragraph representations. In this or any other example, a first one of the two or more input specific subnetworks is parameterized using a first set of parameters, and a second one of the two or more input specific subnetworks is parameterized using a second set of parameters different from the first set of parameters.In this example or any other example, a shared converter block is one of two or more shared converter blocks of a converter-based text encoder, and a dedicated converter block is one of two or more dedicated converter blocks of a converter-based text encoder, and two or more shared converter blocks are interleaved with two or more dedicated converter blocks. In this example or any other example, a plurality of converter blocks of a converter-based text encoder have a ratio of two shared converter blocks for each dedicated converter block. In this example or any other example, a converter-based text encoder is trained using a set of training text questions and a set of training text paragraphs, each training text paragraph including information corresponding to the training text question, so that each training text question is associated with one or more training text paragraphs. In this example or any other example, a converter-based text encoder is also trained using a set of negative training text paragraphs, so that each training text question is associated with one or more negative training text paragraphs, and one or more negative training text paragraphs include information unrelated to the training text question. In this example or any other example, a computer-readable text representation is an input vector representation, which is generated by tokenizing each input text string into input tokens and embedding the value for each input token into a vector representing the input text string.

[0106] In an example, a method for computer passage retrieval includes: receiving a computer-readable question representation of an input text question; passing the computer-readable question representation to a transformer-based text encoder, the transformer-based text encoder previously trained to encode the computer-readable question representation and the computer-readable passage representation using a shared transformer block and a dedicated transformer block, wherein the shared transformer block is trained for both the computer-readable question representation and the computer-readable passage representation, and the dedicated transformer block includes a routing function that routes the computer-readable question representation through a question-specific subnetwork and routes the computer-readable question representation through a passage-specific subnetwork; receiving an output vector question representation of the computer-readable question representation as an output of the transformer-based text encoder; performing a vector comparison between the output vector question representation and a plurality of output vector passage representations corresponding to a plurality of different previously encoded input text passages; identifying one or more of the input text passages as being relevant to the input text question based at least in part on the vector comparison; and outputting an indication of the one or more input text passages identified as being relevant to the input text question. In this example or any other example, the routing function is a deterministic routing function that automatically routes the computer-readable question representation through the question-specific subnetwork and automatically routes the computer-readable passage representation through the passage-specific subnetwork. In this example or any other example, the routing function is a parameterized routing function that is trained to route computer-readable text representations such that each computer-readable question representation is routed through a question-specific subnetwork and each computer-readable paragraph representation is routed through a paragraph-specific subnetwork. In this example or any other example, the shared transformer block includes a neural network parameterized using a shared parameter set for applying encoding operations to both the computer-readable question representation and the computer-readable paragraph representation. In this example or any other example, the shared transformer block is one of two or more shared transformer blocks of a transformer-based text encoder, and the dedicated transformer block is one of two or more dedicated transformer blocks of a transformer-based text encoder, and wherein the two or more shared transformer blocks are interleaved with the two or more dedicated transformer blocks. In this example or any other example, performing the vector comparison includes calculating a relevance score between the output vector question representation and a plurality of output vector paragraph representations, and wherein one or more input text paragraphs identified as being relevant to the input text question are associated with an output vector paragraph representation having a relevance score above a threshold score.

[0107] In an example, a method implemented by a computing system for training a transformer-based text encoder includes: receiving a set of training computer-readable text representations of training text strings, the training computer-readable text representations including training computer-readable question representations of training text questions, and training computer-readable paragraph representations of training text paragraphs, and the training text paragraphs each include information corresponding to the training text questions, so that each training text question is associated with one or more training text paragraphs; after a series of training passes, providing the training computer-readable text representations to the transformer-based text encoder; at the transformer-based text encoder, providing a corresponding output vector representation as an output for each training computer-readable text representation, the transformer-based text encoder including a plurality of transformer blocks to jointly encode the training computer-readable text representations into the output vector representations so that the training text questions are The invention relates to a method for encoding a text encoder based on a transformer in a plurality of embodiments of the present invention. The method comprises: encoding a text encoder based on a transformer in a plurality of embodiments of the present invention into an output vector question representation, and encoding a training text paragraph into an output vector paragraph representation, wherein the plurality of transformer blocks include: a shared transformer block for applying the encoding operation to both the training text question and the training text paragraph; and a dedicated transformer block, the dedicated transformer block including two or more input-specific subnetworks and a routing function, the routing function selecting an input-specific subnetwork of the two or more input-specific subnetworks for each training computer-readable text representation in the training computer-readable text representation; and before a next training pass in a series of training passes, changing one or more parameters of a transformer-based text encoder to increase a correlation score between the output vector question representation corresponding to the training text question, and the output vector paragraph representation corresponding to the training text paragraph associated with the training text question, compared to a previous training pass in the series of training passes.

[0108] It should be understood that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered as restrictive, because many variations are possible. The specific routine or method described herein can represent one or more of any number of processing strategies. Therefore, the various actions shown and / or described can be performed in the order shown and / or described, in other orders, in parallel or omitted. Equally, the order of the above process can be changed.

[0109] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computing system comprising: Logical subsystem; as well as a storage subsystem storing instructions executable by the logic subsystem to implement a converter-based text encoder, the converter-based text encoder comprising: a plurality of transformer blocks previously trained to apply encoding operations to computer-readable text representations of input text strings, the computer-readable text representations comprising computer-readable question representations of input text questions and computer-readable paragraph representations of input text paragraphs, the plurality of transformer blocks comprising: a shared transformer block trained on both the computer-readable question representation and the computer-readable paragraph representation; and A dedicated converter block comprising two or more input specific subnetworks and a routing function for selecting an input specific subnetwork of the two or more input specific subnetworks for each of the computer readable text representations.

2. The computing system of claim 1 , wherein the two or more input specific subnetworks include a question-specific subnetwork and a paragraph-specific subnetwork, and the routing function is a deterministic routing function that automatically routes the computer-readable question representation through the question-specific subnetwork and automatically routes the computer-readable paragraph representation through the paragraph-specific subnetwork.

3. The computing system of claim 1 , wherein the routing function is a parameterized routing function trained to route the computer-readable text representations such that each computer-readable question representation is routed through one or more question-specific subnetworks and each computer-readable paragraph representation is routed through one or more paragraph-specific subnetworks.

4. The computing system of claim 3 , wherein training the routing function comprises using entropy regularization to vary which of the two or more input-specific subnetworks is selected for each of a plurality of training computer-readable text representations input to the transformer-based text encoder over a series of training passes.

5. The computing system of claim 3, wherein each computer-readable text representation comprises a sequence of tokens, and each token of a given computer-readable text representation is routed through the same input-specific subnetwork of the two or more input-specific subnetworks.

6. A computing system according to claim 3, wherein each computer-readable text representation includes a sequence of tokens, and the routing function routes two or more different tokens of a given computer-readable text representation through different input specific subnetworks of the two or more input specific subnetworks.

7. The computing system of claim 1 , wherein the shared transformer block comprises a neural network parameterized with a shared set of parameters for applying encoding operations to both the computer-readable question representation and the computer-readable paragraph representation.

8. The computing system of claim 1 , wherein a first input specific subnetwork of the two or more input specific subnetworks is parameterized using a first set of parameters, and a second input specific subnetwork of the two or more input specific subnetworks is parameterized using a second set of parameters different from the first set of parameters.

9. The computing system of claim 1 , wherein the shared converter block is one of two or more shared converter blocks of the converter-based text encoder, and the dedicated converter block is one of two or more dedicated converter blocks of the converter-based text encoder, and wherein the two or more shared converter blocks are interleaved with the two or more dedicated converter blocks.

10. The computing system of claim 9, wherein the plurality of converter blocks of the converter-based text encoder have a ratio of two shared converter blocks for each dedicated converter block.

11. The computing system of claim 1 , wherein the transformer-based text encoder is trained using a set of training text questions and a set of training text paragraphs, each training text paragraph including information corresponding to the training text question, such that each training text question is associated with one or more training text paragraphs.

12. The computing system of claim 11, wherein the transformer-based text encoder is also trained using a set of negative training text passages, such that each training text question is associated with one or more negative training text passages, the one or more negative training text passages including information unrelated to the training text question.

13. The computing system of claim 1, wherein the computer-readable text representation is an input vector representation generated by tokenizing each input text string into input tokens and embedding a value for each input token into a vector representing the input text string.

14. A method for computer paragraph retrieval, the method comprising: receiving a computer-readable problem representation of an input text problem; passing the computer-readable question representation to a transformer-based text encoder previously trained to encode the computer-readable question representation and the computer-readable paragraph representation using a shared transformer block and a dedicated transformer block, wherein the shared transformer block is trained for both the computer-readable question representation and the computer-readable paragraph representation, and the dedicated transformer block includes a routing function that routes the computer-readable question representation through a question-specific subnetwork and routes the computer-readable question representation through a paragraph-specific subnetwork; receiving an output vector problem representation of the computer-readable problem representation as an output of the transformer-based text encoder; performing a vector comparison between the output vector question representation and a plurality of output vector paragraph representations corresponding to a plurality of different previously encoded input text paragraphs; Based at least in part on the vector comparison, identifying one or more of the input text passages as relevant to the input text question; and An indication of the one or more input text passages identified as being relevant to the input text question is output.

15. The method of claim 14, wherein the routing function is a deterministic routing function that automatically routes the computer-readable question representation through the question-specific subnetwork and automatically routes the computer-readable paragraph representation through the paragraph-specific subnetwork.

Images