System and method for highlighting keywords and phrases from answers

Abstract

In an embodiment, a method of displaying an answer to a question-answer pair in response to a natural language search query includes receiving an array of attention matrices of the question-answer pair from a Transform-based bidirectional encoder representation (BERT) model, where each attention matrix of the array of attention matrices includes an array of attribution values; generating a total cause value for each word of the answer of the question-answer pair according to the attention matrix array; and displaying the answer on an electronic display, where one or more words of the answer are highlighted based on the total cause value of each word.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 593,363, filed on October 26, 2023, the contents of which are incorporated herein by reference in their entirety. Background Art

[0002] Question-answer pair search involves users submitting questions to an embedding model in the form of natural language queries. Each piece of content in the document corpus is indexed into the database using the same embedding model. The system then retrieves and generates one or more question-answer pairs, where the answer is a potential answer to the submitted question. In a second step, the answers are re-ranked using another model that takes question-answer pairs as input and returns a score as output. The scored answers are presented to the user in a result list or other graphical interface. However, some answers may have long text spans, and it may be difficult for users to quickly see and understand the important parts of the text, and it is also difficult to have confidence that the model can produce reliable answers. Summary of the invention

[0003] In order to solve the above problems, the embodiments of the present disclosure highlight important words in the obtained answers so that users can both trust the underlying model and quickly understand the important words and phrases in the given answers.

[0004] In one embodiment, a method for displaying answers to question-answer pairs in response to a natural language search query, the method comprising: receiving a natural language query from a graphical user interface; generating a plurality of question-answer pairs from the natural language query; and inputting the plurality of question-answer pairs into a Transformer-based bidirectional encoder representation (BERT) model. The BERT model can generate an attention matrix array for each question-answer pair in the plurality of question-answer pairs, wherein each attention matrix in the attention matrix array produces an attribution value array. The method further comprises: inputting the output of the BERT model into a classifier, wherein the classifier classifies each question-answer pair as a satisfactory answer or an unsatisfactory answer; and displaying each satisfactory answer. One or more words of each satisfactory answer are highlighted based at least in part on the attribution value array.

[0005] In another embodiment, a system for displaying answers to question-answer pairs in response to a natural language search query includes: one or more processors; an electronic display; and a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the following operations: receiving a natural language query from a graphical user interface; generating a plurality of question-answer pairs from the natural language query; inputting the plurality of question-answer pairs into a Bidirectional Encoder Representation Based on Transformer (BERT) model. The BERT model may generate an array of attention matrices for each of the plurality of question-answer pairs, wherein each attention matrix in the array of attention matrices produces an array of attribution values. The instructions further cause the one or more processors to perform the following operations: inputting the output of the BERT model into a classifier, wherein the classifier classifies each question-answer pair as a satisfactory answer or an unsatisfactory answer; and displaying each satisfactory answer, wherein one or more words of the satisfactory answer are highlighted based at least in part on the array of attribution values.

[0006] In an embodiment, a method for displaying an answer to a question-answer pair in response to a natural language search query includes: receiving an attention matrix array for a question-answer pair from a Bidirectional Encoder Representations from Transformer (BERT) model, wherein each attention matrix in the attention matrix array includes an attribution value array; generating a total attribution value for each word of the answer to the question-answer pair based on the attention matrix array; and displaying the answer on an electronic display, wherein one or more words of the answer are highlighted based on the total attribution value of each word. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] To easily identify discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0008] Figure 1 The diagram illustrates a query / answer pair with attribution according to one or more embodiments described and illustrated herein.

[0009] Figure 2 An example process for analyzing query / answer pairs is illustrated in accordance with one or more embodiments described and illustrated herein.

[0010] Figure 3 Figure illustrates an example attention matrix according to one or more embodiments described and illustrated herein.

[0011] Figure 4 Figure illustrates an example BERT output including an attention matrix array according to one or more embodiments described and illustrated herein.

[0012] Figure 5A Figure illustrates an example BERT output including an attention matrix array and a selected subset of attention matrices according to one or more embodiments described and illustrated herein.

[0013] Figure 5B FIG. 1 illustrates another example BERT output including an attention matrix array and a selected subset of attention matrices according to one or more embodiments described and illustrated herein.

[0014] Figure 6 The diagram illustrates a process for comparing BERT output to ground truth to determine loss according to one or more embodiments described and illustrated herein.

[0015] Figure 7 The diagram illustrates a process for determining an optimal attention matrix subset for BERT output according to one or more embodiments described and illustrated herein.

[0016] Figure 8 Figure illustrates two example highlighted answers according to one or more embodiments described and illustrated herein.

[0017] Fig. 9 An example computing device for highlighting words in an answer is illustrated according to one or more embodiments described and illustrated herein. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure generate query / answer pairs with highlighted important words in response to a query posed by a user. The highlighted words presented in the answer allow the user to quickly view the important words, while also increasing the user's confidence that the system understands the important aspects of the question posed.

[0019] More specifically, embodiments of the present disclosure utilize an attention matrix generated by a trained Transformer-based Bidirectional Encoder Representation (BERT) model to compute attributions for each word token in the resulting query / answer pair. These attributions are used to determine the most relevant tokens in the answer so that the words represented by highly relevant word tokens can be highlighted. The highlighted words are the most important words in the answer, and they both increase the user's confidence in the accuracy of the model and provide the user with a quick visual cue to view the most important parts of the answer.

[0020] Figure 1An example is shown in which a user asks the question "What is the square root of nine?" The user can enter the question into a text box, or the user can speak the question using his or her voice. The spoken words can then be converted into text using any known or yet to be developed speech-to-text algorithm.

[0021] like Figure 1 As shown in the lower left corner of , the system displays the answer "As can be computed using Newton's method, the square root of nine is three." indicating that an attribution of the importance of each word token is calculated. Figure 1 The answer in the upper right corner box shows the attribution above each word token of the answer. For example, the word token "can" has an attribution of 0.1 and the word token "Newton's" has an attribution of 0.3.

[0022] To determine the most important word tokens in an answer, a threshold may be applied so that those word tokens with attributions exceeding the threshold are highlighted in the answer. Embodiments are not limited by any threshold. Figure 1 In the example above, the threshold is set to 0.4. Each word token with an attribution greater than 0.4 is highlighted in the answer presented to the user. Figure 1 In the example of , the phrase "square root of nine is three" is highlighted by highlighting each word token in the phrase because all word tokens in the phrase have an attribution greater than 0.4.

[0023] Previous methods of calculating attribution (gradient-based attribution methods) often require calculating multiple gradients, which is a computationally expensive and time-consuming process. This is because calculating each gradient takes longer than the forward propagation because it requires finding the partial derivative of each weight of the neural network. Therefore, it is not practical to apply gradient-based attribution methods every time a user asks a question to the question-answering system. Such a system would be slow to respond and would consume too much power. Therefore, gradient-based attribution methods cannot be utilized in large-scale data retrieval systems such as legal search platforms, where thousands of users are running thousands of queries that require almost immediate results.

[0024] The BERT model already computes an attention matrix as part of its forward propagation. The attention matrix is ​​the output of the self-attention layer of the BERT model and indicates how much attention each token in the sequence pays to other tokens. Embodiments of the present disclosure utilize these attention matrices of the BERT model to determine token attribution through a fast summation operation, rather than using computationally expensive operations such as gradient calculations. Therefore, the method of the present disclosure can be applied to large-scale data retrieval systems in a fast, economical, and reliable manner.

[0025] Reference now Figure 2 , Figure 2 The diagram illustrates an example process for analyzing a query / answer pair 102 (also referred to herein as a "question-answer pair"). A query / answer pair 102 for a query is provided as input to a BERT model 104. As a non-limiting example, the BERT model 104 may be a legal BERT model trained based on legal documents, such as case law. The output of the BERT model 104 is provided to a classifier 106, which produces a determination 108 as to whether the answer is a satisfactory answer or an unsatisfactory answer. The classifier 106 may be any known or yet to be developed classifier model, such as a feedforward neural network classifier. In some embodiments, the functionality of the classifier 106 is provided by the BERT model 104, rather than as a separate model. For example, the last layer of the BERT model 104 may be a classification layer. The BERT model 104 may output a probability for each category (i.e., a satisfactory answer or an unsatisfactory answer), and the probability value is used to determine whether the answer is satisfactory or unsatisfactory. In Figure 2 In the example of , the answer has a probability of 0.95 of being a satisfactory answer and is therefore selected as the satisfactory answer.

[0026] As described above, the BERT model 104 generates intermediate data in the form of an attention matrix that gives the importance of each token in the answer relative to other tokens. Embodiments of the present disclosure use the importance of each answer token to the query to find the attribution of each word of the query / answer pair 102.

[0027] Figure 3 A simple attention matrix 110 is shown as an example. The attention matrix 110 includes query word tokens in the columns and answer word tokens in the rows.

[0028] Sum the values ​​of each word token in the answer to produce a total value for each token. Figure 3In the example of FIG. 1 , for the word token “Apples”, it has a value of 0.2 for the word token “What” for the query, a value of 0.1 for the word token “is”, and a value of 0.5 for the word token “red”. Therefore, the word token “Apple” receives a total value of 0.8. In this example, according to the attention matrix 110, the word token “red” has a total value of 0.9 and is the most important word.

[0029] The BERT model 104 for classifying question-answer pairs generates many attention matrices 110. Therefore, it becomes desirable to select the attention matrix that will produce the best results in determining the most important words of the answer. Figure 4 An example 12 x 8 matrix of a single attention matrix is ​​shown, where each block is a single attention matrix 110.

[0030] Figure 4 The output of the example BERT model 104 in includes eight attention heads associated with different features and twelve sequential layers, and each combination of a single attention head and a single layer produces a single attention matrix. Some attention heads may be redundant, and some attention matrices may not be as valuable as other attention matrices. Therefore, it may be desirable to select the optimal combination of attention matrices to calculate the score of the answer tag.

[0031] The first attempt was to use the attention matrix of all attention heads in the last layer to calculate the labeled scores, such as Figure 4 The dark attention matrix is ​​shown in the last column of the BERT output 114 in Fig. 1. However, this approach did not work well, so a method for selecting the best attention matrix was developed and applied.

[0032] Figure 5A and Figure 5B The figure shows two different possible sets of selected attention matrices, where the dark blocks represent the selected attention matrices. Figure 5A An example case is shown where the attention matrix of an entire layer is selected to compute the attribution for a word token. Figure 5B In , only some attention matrices in each selected layer are selected to compute the attribution of word tokens.

[0033] To determine the optimal attention matrix to compute the attribution of word tokens, embodiments of the present disclosure utilize a loss function that uses layers and heads as hyperparameters. Figure 6, a training set with query / answer pairs 102 and highlighting of ground truth 116 is used. Each query / answer pair 102 is used to generate a BERT output 114, which is further used to generate an answer 118 with highlighted words. The loss is calculated between the highlighted words of the answer 118 and the highlighted words of the ground truth 116. Therefore, the loss is proportional to the number of inconsistencies (i.e., the Hamming distance) between the highlighted terms in the answer 118 and the highlighted terms in the ground truth. Figure 6 In the example above, there are two inconsistencies.

[0034] refer to Figure 7 , an optimizer 124 is used to combine hyperparameters 128 (i.e., heads and layers) to find an optimal subset of attention matrices 126 for the BERT output 114. Any known or yet to be developed optimization algorithm may be utilized. As a non-limiting example, the optimization algorithm of the optimizer 124 may be a Bayesian optimization, such as provided by OpenBox.

[0035] exist Figure 7 In the workflow of , BERT output 114 and the training set 120 of valid ground truth values ​​are provided as input to the loss function 122, which calculates the loss between BERT output 114 and the training set, as described above with reference to Figure 6 The loss is then provided to the optimizer 124 along with the hyperparameters (i.e., layers and heads) of the single BERT output 114 that produced the calculated loss. In other words, the highlighted word selections that are the output of the BERT model 104 and the highlighted word selections of the training set 120 are provided as input to the loss function. The output and hyperparameters of the loss function 122 are provided as input to the optimizer 124, which then calculates the optimal subset of the attention matrix 126.

[0036] This optimal subset of attention matrices 126 can then be used to select the words that are highlighted in the answer 118 presented to the user. As described above, the values ​​of the word tokens 112 of the answer are summed using all attention matrices of the optimal subset of attention matrices 126. Word tokens 112 that meet the threshold are highlighted in the answer presented to the user. The highlighted words provide a quick visual cue for the user to see the most important parts of the answer, and also increase the user's confidence in the accuracy of the model.

[0037] In some embodiments, post-processing can be used to provide better results for the user. In some cases, a word in a phrase may be omitted because it does not meet the threshold, or a word may be isolated, that is, although the word meets the threshold, there are no other words around it that meet the threshold. Figure 8In the left-hand example of FIG, the words "district", "to", and "Colorado" are not selected for highlighting. However, it makes sense for the reader to highlight these words along with the rest of the phrase. During the post-processing process, these words can be highlighted. In the right-hand example, the words "bearing", "child", "increase", "changes", "income", and "support an increase" are highlighted, but they are isolated and not coordinated with the other highlighted words. Therefore, these words can have their highlighting removed by the post-processing process.

[0038] Post-processing can use heuristic rules to determine when to add or remove highlighting to or from a word. For example, nearby spans that should be merged are merged with adjacent spans. If there are too many spans, the system can keep only the spans with the largest number.

[0039] Embodiments of the present disclosure may be implemented by a computing device and may be embodied as computer-readable instructions stored on a non-transitory memory device. Fig. 9 , schematically illustrates an example system for automatically recommending electronic documents configured as a computing device 130. The example computing device 130 provides a system for highlighting words in query answers and / or a non-transitory computer-usable medium having a computer-readable program code for highlighting words in query answers embodied as hardware, software, and / or firmware according to the embodiments shown and described herein. Although in some embodiments, the computing device 130 may be configured as a general-purpose computer with the necessary hardware, software, and / or firmware, in some embodiments, the computing device 130 may be configured as a special-purpose computer specifically designed for performing the functions described herein. It should be understood that Fig. 9 The software, hardware, and / or firmware components depicted in FIG. 1 may also be provided in other computing devices external to computing device 130 (eg, a data storage device, a remote server computing device, etc.).

[0040] Likewise Fig. 9As illustrated in , the computing device 130 (or other additional computing devices) may include a processor 146, input / output hardware 148, network interface hardware 150, a data storage component 152 (which may store BERT model data (e.g., data related to the BERT model 104), training data 154 (e.g., valid truth value 116 examples and data), and any other data 156 used to perform the functions described herein), and a non-transitory memory component 132. The memory component 132 may be configured as a volatile and / or non-volatile computer-readable medium, and thus, may include random access memory (including SRAM, DRAM, and / or other types of random access memory), flash memory, registers, compact disks (CDs), digital versatile disks (DVDs), and / or other types of storage components.

[0041] Additionally, the memory component 132 can be configured to store the operation logic 134, the BERT logic 136 for determining the attributed value and determining the answer, the classifier logic 138 for determining whether the answer is satisfactory, and the highlight logic 140 for highlighting the words of the answer (each of these logics can be embodied as computer-readable program code, firmware, or hardware, as examples). It should be understood that the data storage component 152 can reside locally and / or remotely from the computing device 130 and can be configured to store one or more data segments for access by the computing device 1002 and / or other components.

[0042] The local interface 144 is also included in Fig. 9 and may be implemented as a bus or other interface to facilitate communication between components of computing device 130 . Processor 146 may include any processing component configured to receive and execute computer-readable code instructions (such as from data storage component 152 and / or memory component 132). Input / output hardware 148 may include a virtual reality headset, a display device, a keyboard, a mouse, a printer, a camera, a microphone, a speaker, a touch screen, and / or other devices for receiving, sending, and / or presenting data. Network interface hardware 150 may include any wired or wireless networking hardware, such as a modem, a LAN port, a wireless fidelity (Wi-Fi) card, a WiMax card, mobile communication hardware, and / or other hardware for communicating with other networks and / or devices.

[0044] The memory component 132 may include operation logic 134, BERT logic 136, classifier logic 138, highlight logic 140, and training logic 142. Operation logic 134 may include an operating system and / or other software for managing components of computing device 1002. BERT logic 136 may reside in memory component 132 and may be configured to generate attributions and potential answers for a query. Classifier logic 138 may also reside in memory component 132 and may be configured to classify potential answers as satisfactory or unsatisfactory. Highlight logic 140 includes logic for applying an attribution threshold to select which words in the answer to highlight when displaying the answer to the user. Training logic 142 includes logic for training optimizer 124 to select an optimal subset of the attention matrix 126 output by BERT.

[0045] Fig. 9 The components illustrated in the figure are exemplary only and are not intended to limit the scope of the present disclosure. More specifically, although Fig. 9 The components in are shown as residing within computing device 130, but this is a non-limiting example. In some embodiments, one or more of these components may reside outside computing device 130.

[0046] It should be understood that embodiments of the present disclosure relate to systems and methods for generating question-answer pairs with highlighted important words in response to queries posed by users. The highlighted words presented in the answers allow users to quickly view important words while also increasing the user's confidence that the system understands important aspects of the questions posed. More specifically, embodiments of the present disclosure utilize an attention matrix generated by a trained Transformer-based Bidirectional Encoder Representation (BERT) model to calculate attributions for each word token in the resulting question / answer pair. These attributions are used to determine the most relevant tokens in the answer so that the words represented by highly relevant tokens can be highlighted. The highlighted words are the most important words in the answer, which both increases the user's confidence in the accuracy of the model and provides the user with a quick visual cue to view the most important part of the answer.

[0047] Note that statements herein that components of an embodiment are "configured" in a particular way to embody a particular characteristic, or to function in a particular manner, are structural statements, as opposed to statements of intended use. More specifically, references herein to the manner in which a component is "configured" refer to the existing physical condition of the component, and are therefore considered to be explicit statements of the structural characteristics of the component.

[0048] Note that one or more of the appended claims use the term "characterized by / wherein" as a transition phrase. For the purpose of defining the embodiments of the present disclosure, it should be noted that this term is introduced into the appended claims as an open transition phrase that is used to introduce a recitation of a series of characteristics of the described structure and should be interpreted in a manner similar to the more commonly used open preamble term "comprising".

[0049] Although the present disclosure has been illustrated and described herein with reference to exemplary embodiments and specific examples thereof, it will be readily apparent to those skilled in the art that other embodiments and examples may also implement similar functions and / or achieve similar results. All of these equivalent embodiments and examples are within the spirit and scope of the present disclosure and are intended to be covered by the appended claims. It is also apparent to those skilled in the art that various modifications and variations may be made to the disclosed concepts without departing from the spirit and scope of the present disclosure. Therefore, it is intended that the present application cover the modifications and variations provided, as long as these modifications and variations are within the scope of the appended claims and their equivalents.

Claims

1. A method for displaying answers to question-answer pairs in response to a natural language search query, the method comprising: receiving a natural language query from a graphical user interface; generating a plurality of question-answer pairs from the natural language query; Inputting the plurality of question-answer pairs into a Transformer-based bidirectional encoder representation BERT model, wherein the BERT model generates an array of attention matrices for each question-answer pair in the plurality of question-answer pairs, wherein each attention matrix in the array of attention matrices produces an array of attribution values; inputting the output of the BERT model into a classifier, wherein the classifier classifies each question-answer pair as a satisfactory answer or an unsatisfactory answer; and Each satisfying answer is displayed, wherein one or more words of each satisfying answer are highlighted based at least in part on the array of attributed values.

2. The method of claim 1, further comprising: For each satisfactory answer, a total attribution value is generated for each word of the satisfactory answer according to a single attention matrix array associated with the satisfactory answer, wherein each word having a total attribution value above a threshold is highlighted.

3. The method of claim 2, wherein the attribution value of each word of the satisfactory answer is generated by: selecting an attention matrix subset of the attention matrix array; and For each word of the satisfactory answer, the attribution values ​​of the attention matrix subset are summed.

4. The method according to claim 3, characterized in that The attention matrix subset is selected through a loss function and an optimization algorithm.

5. The method according to claim 4, characterized in that The attention matrix array includes multiple heads and multiple layers, which are provided as input to the optimization algorithm.

6. The method of claim 1, further comprising applying a post-processing process that performs one or more of the following operations: adding highlighting to one or more words adjacent to the word having the highlight; and, when the number of highlighted phrases exceeds the maximum number of highlighted phrases, removing highlighting from one or more words having the lowest total attribution.

7. The method according to claim 1, characterized in that The classifier is a layer of the BERT model.

8. A system for displaying answers to question-answer pairs in response to a natural language search query, the system comprising: one or more processors; Electronic displays; as well as A non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to: receiving a natural language query from a graphical user interface; generating a plurality of question-answer pairs from the natural language query; Inputting the plurality of question-answer pairs into a Transformer-based bidirectional encoder representation BERT model, wherein the BERT model generates an array of attention matrices for each question-answer pair in the plurality of question-answer pairs, wherein each attention matrix in the array of attention matrices produces an array of attribution values; inputting the output of the BERT model into a classifier, wherein the classifier classifies each question-answer pair as a satisfactory answer or an unsatisfactory answer; and Each satisfying answer is displayed, wherein one or more words of the satisfying answer are highlighted based at least in part on the array of attributed values.

9. The system of claim 8, wherein the instructions cause the one or more processors to further perform the following operations: generating a total attribution value for each word of the satisfactory answer from a single attention matrix array associated with the satisfactory answer, wherein Each word with a total attribution value above the threshold is highlighted.

10. The system of claim 9, wherein the attributed value of each word of the satisfactory answer is generated by: selecting an attention matrix subset of the attention matrix array; and For each word of the satisfactory answer, the attribution values ​​of the attention matrix subset are summed.

11. The system according to claim 10, characterized in that The attention matrix subset is selected through a loss function and an optimization algorithm.

12. The system according to claim 11, characterized in that The attention matrix array includes multiple heads and multiple layers, which are provided as input to the optimization algorithm.

13. The system of claim 8, further comprising applying a post-processing process that performs one or more of the following operations: adding highlighting to one or more words having a total attribution value less than a first threshold; and removing highlighting from one or more words having a total attribution value greater than a second threshold.

14. The system of claim 8, wherein: The classifier is a layer of the BERT model.

15. A method of displaying answers to question-answer pairs in response to a natural language search query, the method comprising: Receiving an array of attention matrices for the question-answer pair from a Transformer-based bidirectional encoder representation BERT model, wherein each attention matrix in the array of attention matrices comprises an array of attribution values; generating a total attribution value for each word of the answer to the question-answer pair based on the attention matrix array; and The answer is displayed on an electronic display, wherein one or more words of the answer are highlighted based on the total attributed value of each word.

16. The method according to claim 15, characterized in that Each word with a total attribution value above the threshold is highlighted.

17. The method of claim 16, wherein the attributed value of each word of the satisfactory answer is generated by: selecting an attention matrix subset of the attention matrix array; and For each word of the satisfactory answer, the attribution values ​​of the attention matrix subset are summed.

18. The method according to claim 17, characterized in that The attention matrix subset is selected through a loss function and an optimization algorithm.

19. The method according to claim 18, characterized in that The attention matrix array includes multiple heads and multiple layers, which are provided as input to the optimization algorithm.

20. The method of claim 15, further comprising applying a post-processing process that performs one or more of the following operations: adding highlighting to one or more words adjacent to the word having the highlight; and, when the number of highlighted phrases exceeds the maximum number of highlighted phrases, removing highlighting from one or more words having the lowest total attribution.

Images