Applying natural language processing to facilitate responses to regulatory questions

Abstract

In a system and method for processing a regulated question, textual data representing the regulated question is obtained by one or more processors. The system and method also classify the regulated question using one or more natural language processing models, generate an answer to the regulated question, generate a summary of the regulated question, and / or identify documents similar to the regulated question. The system and method also stores, transmits, and / or displays data indicative of the classification, the answer, the summary, and / or the similar documents.

Description

[Technical field]

[0001] This application relates generally to techniques for facilitating regulatory processing, and more particularly to systems and methods for classifying questions in regulatory documents (e.g., Health Assessment Questionnaires (HAQs) or Responses to Questions (RTQs)), e.g., to more efficiently respond to such questions. [Background technology]

[0002] In developed countries, there are regulatory authorities (e.g., the Food and Drug Administration in the United States) that rigorously review the safety and effectiveness of products offered by entities such as pharmaceutical or medical device companies. These regulatory authorities typically require extensive data to perform their evaluation. To this end, pharmaceutical and other companies are typically asked to submit various documents, and the regulatory authorities then issue documents requesting additional data. These regulatory documents (e.g., Health Assessment Questionnaires (HAQs) or Response to Questions (RTQs) documents) may contain many (e.g., hundreds) of detailed questions, and it can take a significant amount of time to provide complete and accurate answers. Summary of the Invention [Problem to be solved by the invention]

[0003] One of the most important sources of delay is the initial stage where the respondent must review all questions to determine which questions he or she is best able to answer. For example, a person with experience primarily in drug labeling may not be able to easily answer questions related to clinical trials or safety. Once the query is directed to the appropriate user, further delays occur due to the time it takes for the user to fully understand the information being sought. For example, the question may be very long (e.g., multiple paragraphs) and / or phrased as an article (e.g., a statement of a specific problem / issue) rather than an explicit question. Once the user understands the question, further delays may occur due to the time it takes for the user and / or others to determine the appropriate answer / response. This type of delay is costly not only in terms of consumed employee time, but also in the sense that it may prolong the entire regulatory approval process. Furthermore, manual review may lead to further delays because users are prone to errors, for example, they may skip or ignore questions that are actually relevant to their skills or experience, or they may misunderstand the question in the first place. [Means for solving the problem]

[0004] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The embodiments described herein relate to systems and methods that improve the efficiency, consistency, and / or accuracy in processing questions of the type found in regulatory documents (e.g., HAQ, RTQ, etc.) and / or in generating responses for submission to regulatory authorities. As used herein, and unless the context of use indicates a more specific meaning, terms such as "question," "query," and "interrogation" may refer to an explicit question (e.g., "What is the maximum dose of drug X?") or an implicit question or prompt (e.g., when describing a potential problem with the administration of drug X, with the understanding that the response should explain why the problem need not be of concern or how the problem has been mitigated, etc.), and may refer to a single sentence or a set of related sentences (e.g., "Drug Y is known to be associated with condition Z. How frequently has the condition occurred in pilot trials?"). Additionally, while the present specification refers to "regulatory documents," which may be the source of the particular question under consideration, it is understood that the source of the question may be in other ways, such as by a user (e.g., by cutting and pasting a regulatory question into a user interface, or by manually entering anticipated future regulatory questions, etc.). As used herein, the term "document" may refer to any electronic document or portion thereof (e.g., an original PDF, a PDF that is a scanned version of a paper document, a Word document, etc.), or more generally, any collection of textual data representing a question or other sentences and / or sentence fragments within that question.

[0005] In general, the techniques disclosed herein utilize natural language processing (NLP) and semantic search to process regulatory questions and provide specific outputs that help users prepare responses to the regulatory questions. To provide more accurate / useful results, these techniques can utilize deep learning models (i.e., neural networks), which in some embodiments can perform context-sensitive embedding and / or bidirectional "reading" of the text input (e.g., considering word order bidirectionally to better understand the relationships between multiple words in a question) rather than more straightforward approaches such as keyword search. Additionally, scientific language / knowledge specifically related to regulatory documents (e.g., drug regulatory documents) can be incorporated into deep learning models during the training stage to make the models more useful in that context.

[0006] In some embodiments, the systems and methods disclosed herein automatically classify regulated questions and facilitate the process of generating responses to those questions. For example, the classification unit may pre-process the text (e.g., by parsing the question, filtering out irrelevant words, tokenizing, etc.) and then use an NLP model to classify each question into a category that assists the user in identifying who is best qualified to answer it. Exemplary categories may include "medical", "safety", "regulatory", and / or other suitable labels. In this way, regulated questions may be more quickly and accurately paired with the appropriate person, shortening the process of providing a complete set of responses to the regulator and potentially shortening the regulatory approval process overall. This disclosure also describes a particular type of NLP model or architecture that is particularly suited to the task of classifying regulated documents. In some embodiments, a neural network employing at least one bidirectional layer (e.g., a long short-term memory (LSTM) neural network) performs the classification task. However, in other embodiments, the classification is performed by a neural network that is not typically intended for use in the field of text understanding or classification at all. In particular, in some embodiments, a deep feed-forward neural network classifies each question into the appropriate category. This approach has been determined to perform well despite its relative simplicity (i.e., lack of bidirectionality), and works well with a small number of layers (e.g., only one pooling layer and two dense layers). Because of the simplicity of deep feedforward neural networks, they can be trained and validated to perform classification much faster than other classification models. For example, deep feedforward neural networks can operate (during training, validation, and run-time) approximately 30 times faster (or more) than bidirectional neural networks.

[0007] In other embodiments, the systems and methods disclosed herein automatically identify one or more past / retrospective questions that are similar to the question under consideration. For example, the similarity unit may process / parse the question using an NLP model, retrieve similar questions from a retrospective database, and determine a confidence score indicating the degree of similarity of each. The user may then examine the most similar questions to better understand the question under consideration and / or see if answers / responses to the retrospective questions are useful for the current case. The similarity unit may perform pre-processing on the text of the regulatory question, e.g., as described above with respect to the classification unit.

[0008] In other embodiments, the systems and methods disclosed herein generate answers to the regulated question currently under consideration. For example, the answer generation unit can process / analyze the question using one or more NLP models to automatically generate one or more candidate answers. The answer generation unit can identify relevant retroactive answers by first identifying similar questions, e.g., by applying a similarity unit as described above. The user can then consider whether to incorporate any of the generated candidate answers (in whole or in part) into the submitted regulated response. The answer generation unit can apply pre-processing to the text of the regulated question, e.g., as described above for the classification unit.

[0009] In other embodiments, the systems and methods disclosed herein automatically summarize regulated questions. For example, the summarization unit may use one or more NLP models to process a relatively long regulated question (e.g., perhaps two or three paragraphs not structured as an explicit question) and output a more concise version of the question (e.g., one or two lines expressed as an explicit question). Summarizing the regulated questions in this manner may allow a user to more quickly understand and / or classify each question. The summarization unit may perform preprocessing on the regulated question text, for example as described above with respect to the classification unit.

[0010] In yet other embodiments, some or all of the above embodiments may be used together, for example in a pipeline, parallel, or hybrid pipeline / parallel architecture. For example, the systems and methods disclosed herein may input a question to a classification unit, and then input the same question to a similarity and answer generation unit specific to the classification output from the classification unit. The similarity unit may then identify similar retrospective questions, and the answer generation unit may suggest answers / replies to the question. In other embodiments and / or scenarios, the various units (classification, similarity, answer generation, or summarization) are used independently.

[0011] Those skilled in the art will appreciate that the drawings described herein are included for illustrative purposes and are not intended to limit the disclosure. The drawings are not necessarily to scale, with emphasis instead being placed on illustrating the principles of the disclosure. It should be understood that in some cases, various aspects of the described implementations may be shown in a simplified, exaggerated, or enlarged manner to facilitate understanding of the described implementations. In the drawings, similar reference numbers generally refer to functionally similar and / or structurally similar elements throughout the various views. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram of an example system in which the techniques described herein can be implemented. [Diagram 2] 1 illustrates an exemplary pipeline embodiment of the techniques described herein. [Diagram 3] 2 illustrates an exemplary process that may be implemented by the regulatory document response facilitation application of FIG. 1. [Figure 4] 2 illustrates an exemplary deep feed-forward neural network that may be implemented by the classification unit of the system of FIG. 1 . [Figure 5A] 5 shows a plot of the performance achieved by the deep feedforward neural network of FIG. 4. [Figure 5B]5 shows a plot of the performance achieved by the deep feedforward neural network of FIG. 4. [Figure 5C] 5 shows a plot of the performance achieved by the deep feedforward neural network of FIG. 4. [Figure 6] 2 illustrates an exemplary bidirectional neural network that may be implemented by the classification unit of the system of FIG. 1. [Figure 7A] 2 illustrates an exemplary user interface that may be displayed on a display device in the system of FIG. 1. [Figure 7B] 2 illustrates an exemplary user interface that may be displayed on a display device in the system of FIG. 1. [Figure 7C] 2 illustrates an exemplary user interface that may be displayed on a display device in the system of FIG. 1. [Figure 8] FIG. 2 is a flow diagram of an example method for classifying a regulated question. [Figure 9] FIG. 2 is a flow diagram of an example method for identifying similar documents to a regulated query. [Figure 10] FIG. 2 is a flow diagram of an example method for generating candidate answers to a regulated question. [Figure 11] FIG. 1 is a flow diagram of an exemplary method for summarizing a regulatory question. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The various concepts introduced above and discussed in more detail below may be implemented in any of numerous ways, and the concepts described are not limited to any particular implementation scheme. Examples of several implementation schemes are provided for illustrative purposes.

[0014] FIG. 1 is a block diagram of an exemplary system 100 in which the techniques described herein may be implemented. The system 100 includes a computing system 102 communicatively coupled to a client device 104 via a network 110. The computing system 102 (e.g., a server) is generally configured to train one or more machine learning models performing natural language processing (NLP) and to process regulated documents (e.g., specific regulated questions) using the NLP models for one or more purposes described in more detail below. The client device 104 is generally configured to enable users remote from the computing system 102 to utilize the regulated document processing capabilities of the computing system 102 and provide various interaction capabilities to the users, as described in more detail below. The network 110 may be a single communications network or may include multiple communications networks of one or more types (e.g., one or more wired and / or wireless local area networks (LANs) and / or one or more wired and / or wireless wide area networks (WANs), such as the Internet). Although only one client device 104 is shown in FIG. 1, other embodiments may include any number of different client devices communicatively coupled to the computing system 102 via the network 110. In particular, the client device 104 and a number of other client devices may utilize the regulated document / question processing capabilities of the computing system 102 as a "cloud" service. Alternatively, the computing system 102 may be a local server or collection of servers, or the client device 104 may include the elements and capabilities of the computing system 102 to perform the regulated document processing tasks itself. In the latter case, the computing system 102 and the network 110 may be excluded from the system 100. In yet other embodiments, one, some, or all of the NLP models are trained by another system or device not shown in FIG. 1 before being provided to the computing system 102 or the client device 104.

[0015] As seen in FIG. 1, computing system 102 includes processing hardware 120, network interface 122, and memory 124. However, in some embodiments, computing system 102 includes two or more computers, which may be co-located or remote from one another. In these distributed embodiments, the operations described herein with respect to processing hardware 120, network interface 122, and / or memory 124 may each be split across multiple processing units, network interfaces, and / or memories. Computing system 102 is communicatively coupled (directly or via one or more networks and / or computing devices / systems not shown in FIG. 1) to database 126. Database 126 may be one or more databases stored in one or more local or distributed memories. Database 126 generally includes an archive of past regulatory questions and their answers (e.g., answers manually created / generated by users with appropriate knowledge, experience, and responsibilities) as well as data that may be used to train machine learning models (e.g., NLP model 130, described below). However, in some embodiments, one or more NLP models 130 are trained using data outside of database 126, such as text data collected / scraped from websites, social media services, and / or one or more other sources.

[0016] Processing hardware 120 includes one or more processors, each of which may be a programmable microprocessor that executes software instructions stored in memory 124 to perform some or all of the functions of computing system 102 described herein. Processing hardware 120 may include, for example, one or more central processing units (CPUs) and / or one or more graphics processing units (GPUs). In some embodiments, some of the processors in processing hardware 120 may be other types of processors (e.g., application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc.).

[0017] Network interface 122 may include any suitable hardware (e.g., front-end transceiver hardware), firmware, and / or software configured to communicate with client device 104 (and with other client devices) over network 110 using one or more communications protocols. For example, network interface 122 may be or include an Ethernet interface that enables computing system 102 to communicate with client device 104 and other client devices, such as over the Internet or an intranet.

[0018] The memory 124 may include one or more volatile and / or non-volatile memories. It may include one or more of any suitable type of memory, such as read only memory (ROM), random access memory (RAM), flash memory, solid state drive (SSD), hard disk drive (HDD), etc. Collectively, the memory 124 may store one or more software applications, data received / used by those applications, and data output / generated by those applications. These applications include a Regulatory Document Response Facilitation (RDRF) application 128 that, when executed by the processing hardware 120, processes regulated documents / questions and outputs / displays information in a manner that facilitates the generation of responses to those documents / questions. For example, as described in more detail below, the RDRF application 128 may classify the regulated question under consideration, identify other documents (e.g., other regulated questions) similar to the question under consideration, generate an answer to the question under consideration, and / or summarize the question under consideration. The various software elements of the RDRF application 128 are described below using the term "units," which should be understood to refer to a particular type of software functionality. 1 may alternatively be distributed across two or more different software applications and / or the functionality of any single software unit may be split across two or more software applications. Further, two or more of the illustrated software units may be implemented by a single software module and / or two or more of the illustrated software units may share certain modules / libraries / resources, etc. Memory 124 also stores one or more NLP models 130 utilized by (possibly as part of) RDRF application 128.

[0019] In general, a pre-processing unit 140 of the RDRF application 128 performs one or more operations on the text data (e.g., a data file) that includes the regulated questions, such as parsing the data into different questions, filtering out words that are irrelevant for further processing, and / or other suitable operations. The RDRF application 128 also includes a number of software units that perform the primary processing tasks of the RDRF application 128, including (in the embodiment shown in FIG. 1 ) a classification unit 142A, a similarity unit 142B, an answer generation unit 142C, and a summarization unit 142D. In other embodiments, the RDRF application 128 includes only one, two, or three of the units 142A-D and / or includes other processing units not shown in FIG. 1 . In some embodiments, some or all of the functionality of the pre-processing unit 140 is specific to a particular one of the units 142A-D. For example, the similarity unit 142B may not require the same pre-processing steps as the summarization unit 142D.

[0020] The classification unit 142A typically applies one or more NLP models 130 to the text data (e.g., to the preprocessed text data) to determine an appropriate category for each regulated question represented by the text data. The RDRF application 128 stores, transmits, and / or displays (e.g., locally and / or on the client device 104) data indicative of the determined categories (e.g., to the client device 104 or another computing device or system not shown in FIG. 1). For example, the RDRF application 128 may store the determined categories locally (e.g., in memory 124) and then transmit the stored categories to the client device (e.g., the client device 104) to cause the client device to display the categories (or cause the client device to display the question in a manner that otherwise reflects the determined categories), or transmit the stored categories to a printer device to cause the printer device to print a representation of the categories. As another example, the RDRF application 128 may display the categories directly on the computing system 102.

[0021] The similarity unit 142B typically applies one or more NLP models 130 to the text data (or to the preprocessed text data) to identify one or more documents (e.g., other, historical / retrospective questions) that are most similar to the particular regulatory question represented by the text data. The similarity unit 142B may, for example, identify similar documents among documents contained in the database 126. The RDRF application 128 may store, transmit (e.g., to the client device 104 or to another computing device or system not shown in FIG. 1 ), and / or display (e.g., locally and / or at the client device 104) data indicative of the identified similar documents. For example, the RDRF application 128 may store data indicative of the identified documents locally (e.g., in memory 124) and then transmit the stored data to a client device (e.g., the client device 104) to cause the client device to display information about the documents (e.g., titles, excerpts, etc.) or transmit the stored data to a printer device to cause the printer device to print such information. As another example, the RDRF application 128 may display the data / information directly on the computing system 102 .

[0022] The answer generation unit 142C generally applies one or more NLP models 130 to the text data (or to the preprocessed text data) to generate one or more candidate answers to a particular regulated question represented by the text data. In some embodiments, the answer generation unit 142C utilizes the similarity unit 142B (or performs functions similar to the similarity unit 142B) to find documents in the database 126 that are similar to the particular regulated question, and then generates the candidate answers based at least in part on the text content of the similar documents. In these embodiments, the answer generation unit 142C may generate the candidate answers by identifying and extracting portions of the similar documents (e.g., portions of actual answers to past regulated questions identified by the similarity unit 142B), or may synthesize responses without relying on (or completely relying on) the verbatim text of the similar documents. The RDRF application 128 may store, transmit (e.g., to the client device 104 or another computing device or system not shown in FIG. 1), and / or display (e.g., locally and / or on the client device 104) data indicative of the generated answers (e.g., the answers themselves). For example, the RDRF application 128 may store the generated answers locally (e.g., in memory 124) and then send the stored answers to a client device (e.g., client device 104) to cause the client device to display the answers, or send the stored answers to a printer device to cause the printer device to print the answers. As another example, the RDRF application 128 may display the answers directly on the computing system 102.

[0023] The summarization unit 142D generally applies one or more NLP models 130 to the text data (or to the preprocessed text data) to generate a shorter summary of the particular regulated question represented by the text data. In some embodiments, the summarization unit 142D utilizes the similarity unit 142B (or performs a function similar to the similarity unit 142B) to find similar documents in the database 126 to the particular regulated question, and then generates a summary based at least in part on the text content of the similar documents. The RDRF application 128 stores, transmits (e.g., to the client device 104 or another computing device or system not shown in FIG. 1 ), and / or displays (e.g., locally and / or on the client device 104) data indicative of the generated summary (e.g., the summary itself). For example, the RDRF application 128 may store the generated summary locally (e.g., in the memory 124) and then transmit the stored summary to a client device (e.g., the client device 104) to cause the client device to display the summary or transmit the stored summary to a printer device to cause the printer device to print the summary. As another example, the RDRF application 128 may display the summary directly on the computing system 102 .

[0024] The operation of each of units 142A-D is described in further detail below. It will be appreciated that in some embodiments, one, some, or all of units 142A-D may each include two or more NLP models of NLP models 130. In one embodiment, for example, NLP models 130 include multiple NLP classification models each specialized to determine whether text data corresponding to a particular question should or should not be classified as belonging to a single respective category (e.g., one of NLP models 130 determines whether to classify as "safe" and another of NLP models 130 determines whether to classify as "labeling"), in which case classification unit 142A can utilize each of these class-specific NLP models to classify each question according to one or more classes / categories. As another example, the answer generation unit 142C may include a first model of the NLP models 130 to identify similar documents to a particular regulated question in the database 126, and a second model of the NLP models 130 to generate one or more candidate answers to the regulated question based on the textual content of the identified documents.

[0025] The RDRF application 128 can also collect data entered by a user via the client device's user interface and web browser application, and / or detect user activation of controls presented in the client device's user interface and web browser application, as described herein with specific reference to the client device 104. The client device 104 includes processing hardware 160, a network interface 162, a display device 164, a user input device 166, and a memory 168. The processing hardware 160 includes one or more processors, each of which may be a programmable microprocessor that executes software instructions stored in the memory 168 to perform some or all of the functions of the client device 104 described herein. The processing hardware 160 may include, for example, one or more CPUs and / or one or more GPUs. In some embodiments, some of the processors in the processing hardware 160 may be other types of processors (e.g., ASICs, FPGAs, etc.).

[0026] Network interface 162 may include any suitable hardware (e.g., front-end transceiver hardware), firmware, and / or software configured to communicate with computing system 102 over network 110 using one or more communications protocols. For example, network interface 162 may be or include an Ethernet interface that enables client device 104 to communicate with computing system 102 over the Internet, an intranet, or the like.

[0027] Memory 168 may include one or more volatile and / or non-volatile memories. It may include one or more of any suitable memory, such as ROM, RAM, flash memory, SSD, HDD, etc. Memory 168 may generally store one or more software applications, data received / used by those applications, and data output / generated by those applications. These applications include a web browser application 170 that, when executed by processing hardware 160, allows a user of client device 104 to access various websites and web services, including services provided by computing system 102 when executing RDRF application 128. In other embodiments not shown in FIG. 1 (e.g., certain embodiments that do not utilize web services), memory 168 stores and locally executes RDRF application 128 and NLP model 130.

[0028] The display device 164 of the client device 104 may implement any suitable display technology (e.g., LED, OLED, LCD, etc.) for presenting information to a user, and the user input device 166 of the client device 104 may include a keyboard, a microphone, a mouse, and / or any other suitable input device. In some embodiments, at least a portion of the display device 164 and at least a portion of the user input device 166 are integrated into a single device (e.g., a touch screen display). In general, the display device 164 and the user input device 166 together may enable a user to interact with a user interface that enables communication with the RDRF application 128 via web services (e.g., via a web browser application 170, the network interface 162, the network 110, and the network interface 122) or locally (if the RDRF application 128 and the NLP model 130 reside on the client device 104). For example, the user may interact with the user interface in a manner described below with reference to any one or more of FIGS. 7A-C.

[0029] FIG. 2 illustrates an exemplary embodiment in which the functionality of units 142A-D of RDRF application 128 is arranged as a pipeline 200. In pipeline 200, a particular regulated question is selected or obtained for consideration at stage 202. The regulated question may be a question entered by a user into a user interface (e.g., via display device 164 and user input device 166) or may be a question extracted automatically by preprocessing unit 140, for example, from a larger document. At stage 204 of pipeline 200, summarization unit 142D summarizes the regulated question. RDRF application 128 may display the summary to a user (e.g., via network 110 and display device 164). Also, in the illustrated embodiment, a summarized version of the regulated question is classified by classification unit 142A at stage 206. However, in other embodiments, classification unit 142A does not operate on a summary but rather operates directly on the regulated question (possibly after preprocessing by preprocessing unit 140). In either case, the RDRF application 128 may display the category / classification to the user (e.g., via the network 110 and the display device 164), for example, by generating / displaying a text label corresponding to the category / classification or by displaying a control question in a portion of the user interface reserved for the particular category.

[0030] In stage 208, the similarity unit 142B identifies one or more documents from the database 126 that are similar to the controlled query. In the illustrated embodiment, the classification from stage 206 is used in stage 208. For example, the RDRF application 128 may select and use a classification-specific NLP model in stage 208. However, in other embodiments, the similarity unit 142B does not use the classification from stage 206, but instead operates only on the controlled query itself (possibly after preprocessing by the preprocessing unit 140). In either case, the RDRF application 128 may display information related to the similar documents to the user (e.g., via the network 110 and the display device 164), for example, by generating / displaying the names and / or other identifiers (e.g., file names) of the documents and / or portions of the text from the documents (e.g., at least a portion of the particular text that caused the similarity unit 142B to identify the documents).

[0031] In stage 210, the answer generation unit 142D generates one or more candidate answers to the regulated question. In the illustrated embodiment, the similar documents from stage 208 are used in stage 210 to generate the answer. For example, the similarity unit 142B may use the first NLP model in stage 208 to identify similar documents in the database 126, after which the answer generation unit 142D may analyze the textual content of the identified documents in stage 210 to extract or synthesize one or more candidate answers. The RDRF application 128 may then display the candidate answers to the user (e.g., via the network 110 and the display device 164), possibly along with other information such as identifiers of the documents from which the candidate answers were derived (e.g., file names and / or other document identifiers) and / or portions of the text of the documents from which the candidate answers were derived (e.g., at least a portion of the particular text used by the answer generation unit 142D to generate the answer).

[0032] 3 illustrates a process 300 that reflects run-time operations of system 100, according to some embodiments. However, prior to the run-time operations reflected by process 300, computing system 102 (or another computing system not shown in FIG. 1) trains and validates NLP model 130 using data stored in database 126 and / or other data external to system 100. Some training data may be for unsupervised learning (e.g., for training a model that learns contextualized embeddings of words, as discussed further below), while other training data may include manually prepared labels for supervised learning (e.g., for training a classification model for classification unit 142A).

[0033] At stage 302 of process 300, RDRF application 128 retrieves a regulatory question (e.g., a question related to one or more regulatory documents, such as an HAQ, an RTQ, etc.). For example, RDRF application 128 can retrieve a regulatory document in a PDF or other electronic file format from a remote or local source, retrieve text data extracted from one or more larger regulatory documents, receive manually entered questions, etc.

[0034] At stage 304, preprocessing unit 140 parses the constituent questions from the text. Preprocessing unit 140 may parse the text into questions using known delimiters or fields in the data file containing the text, based on other formatting of the data file containing the text (e.g., based on the relative spacing / justification of text in a PDF file), or using any other suitable technique.

[0035] In stage 306, preprocessing unit 140 cleans the text of the question by filtering out words and / or characters that are (or would be) irrelevant to the task performed by one or more units of RDRF application 128 and one or more NLP models 130. This may include, for example, filtering out some or all conjunctions (e.g., "for," "and," "not," "but," "or," "because," "when," "while," etc.), some or all prepositions (e.g., "in," "under," "toward," "before," etc.), some or all special characters (e.g., semicolons, quotation marks, etc.), etc. In some embodiments, preprocessing unit 140 also filters out words that have substantial meaning in other contexts but are irrelevant or even hindering the performance of a particular task. For example, if stage 306 is used to prepare for classification by classification unit 142A, preprocessing unit 140 may filter out words that express numerical values ​​or words that express only degrees, such as "large" or "3%."

[0036] In stage 308, pre-processing unit 140 tokenizes the question text (e.g., parses each question into individual words or other linguistic units). In stage 310, pre-processing unit 140 converts the word strings in the questions (excluding the words removed in stage 306) into number strings by converting each token (e.g., each word) of the “cleaned” question into a numerical value. For example, the relatively short question “Provide detailed performance results showing viscosity greater than 10 cP” may be cleaned and parsed into the words / tokens “Provide”, “Detailed”, “Performance”, “Results”, “Show”, “Viscosity”, “Greater than”, and “cP”, which may be converted into the number string 125 453 067 012 363 284 138 421. To convert all questions into number strings of equal length (i.e., a predetermined fixed length suitable for one or more NLP models 130), pre-processing unit 140 pads each number string as necessary in stage 312. The fixed length may be, for example, slightly more than the number of tokens expected to be present in the longest question of the regulatory document (after any cleaning of the kind performed in stage 306).

[0037] At stage 314, one or more of units 142A-D apply one or more NLP models 130 to the (possibly padded) sequence to perform their respective tasks. For example, classification unit 142 may apply one of NLP models 130 to the (possibly padded) sequence to classify a regulated question corresponding to the sequence. At stage 316, RDRF application 128 stores, transmits, and / or displays data indicative of the output generated by NLP model 130 (e.g., data indicative of one or more classifications). For example, if classification unit 142A operates at stage 314, computing system 102 may transmit data to client device 104 to cause display device 164 of client device 104 to display the appropriate category next to each question, or to cause display device 164 to display only questions associated with a user-specified category (e.g., a category indicated by a user via user input device 166 when accessing a user interface via web browser application 170 or another application, etc.). As another example, computing system 102 may cause the data to be stored in memory (e.g., a flash device, a portion of memory 124, etc.) for later use (e.g., by computing system 102, client device 104, and / or another computing device or system), or may cause a printer device to print the data, etc.

[0038] 3 may differ from that shown, and / or fewer and / or different pre-processing stages may be included depending on the embodiment and units 142A-D operating in stage 314. In some embodiments, for example, pre-processing unit 140 parses questions (stage 304) only after cleaning the text of all questions to remove irrelevant words (stage 306). As another example, the sequence of stages 306, 308, 310, 312, 314, and 316 may be repeated for each question (e.g., as each question is parsed in stage 304 or after all questions have been parsed), or multi-threading may enable stages 306, 308, 310, 312, 314, and / or 316 to operate simultaneously on two or more questions.

[0039] Various embodiments of the particular NLP model 130 will now be discussed. Referring first to classification, the classification unit 142A performs classification tasks based on words or other tokens using an NLP model (of the NLP model 130) that is a neural network (or in other embodiments, a set of neural networks performing the respective classification tasks, as described above). In the embodiment shown in FIG. 4, the NLP model used by the classification unit 142A is or includes a deep feed-forward (DFF) neural network 400. Counterintuitively, the DFF neural network 400 can perform well despite a lack of bidirectionality that makes it less suitable for text understanding tasks such as classification. The performance of the DFF neural network 400 is described in more detail below with reference to FIGS. 5A-C.

[0040] In the DFF neural network 400, an embedding layer generates an embedding matrix 402 from the sequence generated in stage 310, where one dimension of the embedding matrix 402 is the length (after padding) of the sequence (e.g., 5,000, or 10,000, etc.) and the other dimension of the embedding matrix 402 is the input dimension of the global max pooling layer 404 of the DFF neural network 400 (e.g., 128, 256, or another suitable multiple of 2). In other embodiments, the embedding matrix 402 is three-dimensional. The DFF neural network 400 includes a first dense layer 406 after the global max pooling layer 404, and a second dense layer 408 after the first dense layer 406. In the illustrated embodiment, each node of the second dense layer 408 corresponds to a different classification / label / category 410. In this example, the set of available categories includes "CMC" (e.g., for manufacturing and control of drug substances and drug substances), "Clinical" (e.g., for patients, patient-directed formulations, or patient-directed devices), "Regulatory" (e.g., for regulatory or governmental spaces), "Labeling" (e.g., for product labeling, language, and compliance with legal requirements), and "Safety" (e.g., for patient safety). DFF neural network 400 may include one or more additional stages and / or layers not shown in FIG. 4. For example, DFF neural network 400 may include a dropout stage immediately after global max pooling layer 404, an activation layer (e.g., having a tanh or other suitable activation function) immediately after the first dense layer 406, and another dropout stage immediately after the activation layer. In alternative embodiments, DFF neural network 400 may include more or fewer dense and / or pooling layers than those shown in FIG. 4. However, the relatively low-complexity architecture of FIG. 4 (with only one pooling layer and two dense layers) can produce results that outperform other DFF neural networks with more or fewer pooling and / or dense layers.

[0041] The DFF neural network 400 calculates a value for each node of the second dense layer 408, and in some embodiments, the classification unit 142A determines the classification based on which node of the second dense layer 408 has the highest value. However, in other embodiments, the classification unit 142A does not make a strict decision regarding the appropriate classification, but instead outputs data indicative of a looser decision (e.g., by providing some or all of the values ​​calculated by the second dense layer 408 for a user to inspect / review).

[0042] DFF neural network 400 may be trained (prior to run-time operation) using manually labeled control questions from database 126 (and / or anywhere else), with the questions serving as inputs / features and the manual labels serving as training labels. The simplicity of DFF neural network 400 allows it to be trained and validated, and to perform classification, much faster (e.g., by an order of magnitude or more) than other classification models (e.g., bidirectional neural networks).

[0043] The performance of the DFF neural network 400 shown in FIG. 4 (i.e., with exactly one global max pooling layer and exactly two dense layers) is shown in FIGS. 5A-C, which show both training and validation. FIGS. 5A-C show both training and validation results, with the validation results being a better representation of expected run-time performance. As seen in plots 500, 520, and 540 in FIGS. 5A, 5B, and 5C, respectively, the DFF neural network 400 gave an accuracy of about 80%, a loss of about 0.62, and a recall of about 76%. It can be seen that the accuracy, loss, and recall measures of such a model do not have to be very close to the ideal measures, since incorrectly classified questions are eventually routed to a suitable person (e.g., after first being presented to a non-suitable person or after first being classified as "unknown"), albeit with some additional delay. As long as the measures are reasonably good, the classification can save the respondent quite a lot of time.

[0044] Figure 6 illustrates an alternative embodiment in which the NLP model used by classification unit 142A is or includes a bidirectional neural network 600. The example bidirectional neural network 600 (e.g., an LSTM neural network) of Figure 6 includes an input layer 602 that receives an input (e.g., the padded sequence output at stage 312 of Figure 3), an embedding layer 604 (e.g., for generating an embedding matrix similar to embedding matrix 402 from the padded sequence), a bidirectional layer 606 that performs inter-layer feedback within the neural network 600, a one-dimensional convolutional (Conv1D) layer 608, a one-dimensional average pooling layer 610, a one-dimensional max pooling layer 612, a concatenation layer 614, and a dense layer 616. In other embodiments, the bidirectional neural network 600 may include more or fewer layers and / or stages (e.g., more dense layers, more pooling layers, etc.). Although bidirectional neural network 600 may take more time to train, validate, and run than DFF neural network 400, bidirectional neural network 600 may provide better results in some cases (e.g., when many of the questions are relatively long) due to its ability to essentially read text both forward and backward.

[0045] The similarity unit 142B may use an NLP model (of NLP model 130) that is or includes a bidirectional neural network. Additionally, the NLP model used by the similarity unit 142B may be a contextualized embedding model (i.e., a model trained to learn word embeddings based on the context in which the words are used). For example, the similarity unit 142B may use a Bidirectional Encoder Representations from Transformers (BERT) model to identify similar documents.

[0046] The answer generation unit 142C can use the same NLP model (directly or by invoking the similarity unit 142B, etc.) to identify documents similar to the regulated question, and can use an additional NLP model (also of the NLP model 130) to generate one or more candidate answers to the regulated question based on the identified documents. The additional NLP model can be, for example, a transformer-based language model such as GPT-2, and can be trained using a large dataset such as SQuAD (Stanford Question Answering Dataset). In some embodiments, the NLP model is further trained / refinement (by the computing system 102 or another computing device / system) using a data source with text content that is more reflective of language likely to appear in the regulated question / document. If the regulated question is related to a drug (e.g., usage, risks, etc.), for example, the NLP model can be further trained using documents that are more likely to use drug-related terms, such as retrospective HAQ and RTQ, drug patents, etc. In this manner, the additional NLP models used by the answer generation unit 142C are better suited to understanding the terminology of controlled questions.

[0047] The summarization unit 142D can generate summaries of the regulatory questions using yet another NLP model (of the NLP models 130). The NLP model used by the summarization unit 142D can be or include a bidirectional neural network. Furthermore, the NLP model used by the summarization unit 142D can be a contextualized embedding model. For example, the summarization unit 142D can generate summaries using a BERT model.

[0048] The RDRF application 128 can use an Elastic Search Engine to search the database 126 (or at least the portion of the database 126 that contains retrospective regulatory documents and / or other documents). The Elastic Search Engine has proven to be particularly accurate and reliable with regulatory documents because of the sparseness of the data and because Elastic Search supports embeddings (which may be used by various NLP models, as discussed above).

[0049] Figures 7A-C illustrate exemplary user interfaces that can be provided by the system 100 of Figure 1. More specifically, a web browser application 170 on the client device 104 can present any or all of the user interfaces of Figures 7A-C to a user via the display device 164 using data provided to the client device 104 by the RDRF application 128 running on the computing system 102. Alternatively, the user interfaces of Figures 7A-C may all be generated on the client device 104 (e.g., in an embodiment in which the RDRF application 128 resides on the client device and the system 100 does not include a computing system 102).

[0050] Referring initially to FIG. 7A, an exemplary user interface 700 includes an area 702 in which the text of various questions from a regulatory document can be displayed along with associated information (i.e., the classification of the question, such as "Clinical" or "CMC" in this example). The user interface 700 also includes a series of controls 704 that provide the user with various filtering options. Based on the settings (default or user-configured) of the controls 704, the area 702 displays only those questions (from one or more relevant regulatory documents) that meet the specified filter criteria. The "Predicted Labels" control allows the user to filter according to any of the complete set of classifications of the question made by the classification unit 142A. The text search control allows the user to search for questions based on characters, terms, etc. contained within the text of the question.

[0051] Table 1 below provides a more extensive list of example questions with various classifications that may be included in area 702 (e.g., if a user scrolls down for a complete list of questions). However, it can be seen that the list in Table 1 is still quite short compared to the majority of real-world scenarios.

[0052] [Table 1]

[0053] The exemplary user interface 700 also includes a word distribution bar graph 706 showing counts of the most frequent words in the full set of questions (or, in some embodiments, counts of the most frequent words in the filtered set of questions), and a predicted label distribution bar graph 710 showing counts of the most frequent classifications / labels / categories for the full set of questions. The exemplary user interface 700 also includes a word cloud 712 to assist the user in visually approximating the frequency and number of different words. It will be appreciated that in other embodiments, the user interface 700 can display more information (e.g., all questions with their determined classifications), less information (e.g., without the word cloud 712), and / or different information, and / or can display the information in a different format (e.g., simple counts instead of bar graphs 706, 710).

[0054] FIG. 7B illustrates another exemplary user interface 720. In the user interface 720, an input field 722 allows the user to input (e.g., type or cut and paste) a regulated question of interest. A control unit 724 allows the user to select the type of model or function to apply to the question entered in the input field 722. In this example, if the user selects "DC", the RDRF application 128 processes the entered question through the classification unit 142A; if the user selects "SS", the RDRF application 128 processes the entered question through the similarity unit 142B; if the user selects "QA", the RDRF application 128 processes the entered question through the answer generation unit 142C (which also includes processing the question through the similarity unit 142B as described above); and if the user selects "SUM", the RDRF application 128 processes the entered question through the summarization unit 142D. FIG. 7B illustrates the scenario when the user selects "QA".

[0055] Another control 726 allows the user to set the level of complexity of the model (e.g., by selecting among five discrete complexity levels shown in FIG. 7B). Higher complexity may, for example, correspond to a more complex NLP model (e.g., more neural network layers) or may mean that a single NLP model is applied for a longer period of time. In general, higher complexity results in higher accuracy but also longer processing time.

[0056] Area 730 of user interface 720 shows similar documents identified by RDRF application 128. In some embodiments, similar questions are only displayed if they are questions identified by similarity unit 142B and / or if the user selects "SS" using control 724. Area 732 of user interface 720 shows candidate answers generated by answer generation unit 142C along with associated information. In this example, area 732 also shows, for each candidate answer, an associated confidence score generated by the GPT-2 or other NLP model used by answer generation unit 142C, an identifier of the source / document used by answer generation unit 142C to derive the illustrated answer, and a "context" indicating at least a portion of the particular text of the document used by answer generation unit 142C to derive the illustrated answer.

[0057] Controls 734 allow the user to indicate whether the displayed answer is useful / helpful or not useful / useless (in the illustrated example, by selecting a “thumbs up” icon or a “thumbs down” icon, respectively). The RDRF application 128, or other software stored on the computing system 120 or another system / device, can use the feedback data representing the user's selections or inputs via the controls 734 to further train / refine one or more NLP models 130 used by the answer generation unit 142C, e.g., by reinforcement learning. For example, the RDRF application 128 can use the feedback data to further train an NLP model (e.g., a BERT model) used to identify similar documents and / or to further train another NLP model (e.g., a GPT-2 model) used to generate answers based on similar documents.

[0058] Figure 7C illustrates yet another exemplary user interface 740. User interface 740 includes input fields 742 and controls 744, which may be the same as or similar to input fields 722 and controls 724 of Figure 7B. User interface 740 may be the same as user interface 720 shown in Figure 7B, for example, but for a different scenario in which the user selects "SS" instead of "QA."

[0059] Area 746 of user interface 740 shows the number of candidate categories / classifications determined by classification unit 142A along with their respective confidence scores. The confidence scores may be, for example, numerical values ​​output at different nodes of second dense layer 408 of DFF neural network 400 shown in FIG. 4. Area 752 of user interface 740 shows information about similar documents identified (within database 126) by similarity unit 142B. In this example, area 752 also shows, for each identified document, the document's identifier / name, the document's identifier ("ID"), and a "context" indicating at least a portion of the document's particular text that similarity unit 142B used as a basis for selecting / identifying the document as a "similar" document.

[0060] User interface 740 also includes controls 754 for providing user feedback, which may be similar to controls 734 of user interface 720. The RDRF application 128, or other software stored on computing system 120 or another system / device, can use the feedback data representing user selections or inputs via controls 754 to further train / refine, for example by reinforcement learning, one or more NLP models 130 used by similarity unit 142B. For example, RDRF application 128 can use the feedback data to further train a BERT model used by similarity unit 142B to identify similar documents.

[0061] 8-11 are flow diagrams of example methods for facilitating answering regulatory questions. The methods may be performed by processing hardware 120 of computing system 102, for example, when executing software instructions of RDRF application 128 stored in memory 124. In other embodiments, some or all of each method is performed by processing hardware 160 of client device 104, for example, when executing software instructions of an application stored in memory 168 (e.g., web browser application 170, or RDRF application 128 if resident on client device 104).

[0062] Referring initially to Figure 8, in block 802, text data representing a plurality of regulated questions (e.g., questions from one or more regulated documents) is obtained. Block 802 may be similar to stage 302 of process 300, for example. In block 804, one or more classifications of the plurality of regulated questions are generated, at least in part, by processing the text data obtained in block 802 with an NLP model. The NLP model may be, for example, one of the NLP models 130 of Figure 1. As a more specific example, the NLP model may be the DFF neural network 400 of Figure 4 or the bidirectional neural network 600 of Figure 6.

[0063] At block 806, data indicative of the classification is stored, transmitted, and / or displayed. The data may be data derived from the classification (e.g., a subset of questions corresponding to a particular one of the generated classifications) or may be the classification itself. In some embodiments, block 806 includes causing at least a subset of the plurality of regulated questions to be displayed (e.g., locally or on another computing device) in a manner indicative of the classification. For example, block 806 may include selectively displaying or not displaying each regulated question based on both the classification (of the classifications determined at block 804) corresponding to the regulated question and a filter setting selected by the user (e.g., a setting of a control similar to the "Predicted Label" control in user interface 700 of FIG. 7A). As another example, block 806 may include causing each question (and possibly all questions) of the subset of questions to be displayed in association with a corresponding classification (e.g., such that the classification generated at block 804 is displayed alongside the corresponding question in user interface 700 of FIG. 7A, or a similar user interface).

[0064] In some embodiments, method 800 includes one or more additional blocks not shown in Figure 8. For example, method 800 may include an additional block (e.g., occurring after block 802 and before block 804) in which the text data may be pre-processed to remove words and / or characters that are not used for classification by converting the word sequences of the control question into respective number sequences and / or by padding these sequences (e.g., any of the operations described above with reference to stages 304, 306, 308, 310, and / or 312 of process 300 in Figure 3).

[0065] 9, in block 902, text data representing a controlled question (e.g., a question from a controlled document) is obtained. Block 902 may be similar to, for example, part of stage 302 of process 300. In block 904, one or more documents similar to the controlled question are identified, at least in part, by processing the text data obtained in block 902 with an NLP model. The NLP model may be, for example, one of the NLP models 130 of FIG. 1. As a more specific example, the NLP model may be a BERT model or another bidirectional neural network that supports contextualized embeddings.

[0066] At block 906, data indicative of the documents is stored, transmitted, and / or displayed. The data may include names and / or other identifiers for each document and / or text from the documents that caused the NLP model to identify the documents as “similar” documents, for example, at block 904.

[0067] In some embodiments, method 900 includes one or more additional blocks not shown in Figure 9. For example, method 900 may include one or more additional blocks (e.g., occurring after block 902 and before block 904) that apply one or more preprocessing steps described above in connection with method 800 (e.g., filtering out irrelevant words and / or characters, converting word strings to number strings, and / or padding number strings).

[0068] 10 , in block 1002, text data representing a controlled question (e.g., a question from a controlled document) is obtained. Block 1002 may be similar to, for example, part of stage 302 of process 300. In block 1004, one or more documents similar to the controlled question are identified, at least in part, by processing the text data obtained in block 1002 with a first NLP model. Block 1004 may be similar to, for example, block 904 of method 900.

[0069] At block 1006, one or more candidate answers to the regulated question are generated, at least in part, by processing the documents identified at block 1004 with a second NLP model. The second NLP model may be, for example, a GPT-2 model or other suitable bidirectional neural network. At block 1008, data indicative of the candidate answers generated at block 1006 is stored, transmitted, and / or displayed. For each candidate answer, the data may include the candidate answer itself, an identifier for the document from which the candidate answer was derived, and / or a portion of the text of the document from which the candidate answer was derived.

[0070] In some embodiments, method 1000 includes one or more additional blocks not shown in FIG. 10. For example, method 1000 may include one or more additional blocks (e.g., occurring after block 1002 and before block 1004) that apply one or more pre-processing steps described above in connection with method 800 (e.g., filtering out irrelevant words and / or characters, converting a word string to a number string, and / or padding the number string). As another example, method 1000 may include a first additional block of determining a confidence score associated with each of one or more candidate answers to the regulated question, and a second additional block of storing, transmitting, and / or displaying data indicative of the confidence score associated with each of the one or more candidate answers to the regulated question. As yet another example, method 1000 may include a first additional block of receiving user feedback indicative of the usefulness of the one or more candidate answers, and a second additional block of training the first and / or second NLP models using the user feedback.

[0071] 11 , in block 1102, text data is obtained that represents a regulated question (e.g., a question from a regulated document). Block 1102 may be similar to, for example, a portion of stage 302 of process 300. In block 1104, a summary of the regulated question is generated, at least in part, by processing the text data obtained in block 1102 with an NLP model. The NLP model may be, for example, one of the NLP models 130 of FIG. 1. As a more specific example, the NLP model may be a BERT model or another bidirectional neural network that supports contextualized embeddings.

[0072] At block 1106, data indicative of the summary is stored, transmitted, and / or displayed. The data includes, for example, the summary itself and possibly associated information, such as names, identifiers, and / or portions of the one or more documents from which the summary was derived. In some embodiments, method 1100 includes one or more additional blocks not shown in FIG. 11. For example, method 1100 may include one or more additional blocks (e.g., occurring after block 1102 and before block 1104) that apply one or more preprocessing steps (e.g., filtering out irrelevant words and / or characters, converting word strings to number strings, and / or padding number strings) discussed above in connection with method 800.

[0073] The following list of examples reflects various embodiments expressly contemplated by this disclosure. EXAMPLES

[0074] Example 1. A method for processing regulated questions, the method including: obtaining, by one or more processors, text data representing a plurality of regulated questions; generating, by the one or more processors, one or more classifications of the plurality of regulated questions by at least partially processing the text data with a natural language processing model; and storing, transmitting, and / or displaying, by the one or more processors, data indicative of the one or more classifications.

[0075] Example 2. In the method of Example 1, the natural language processing model is a deep feedforward neural network.

[0076] Example 3. In the method of Example 2, the deep feedforward neural network includes exactly one global max pooling layer and multiple dense layers.

[0077] Example 4. In the method of Example 3, the deep feedforward neural network contains exactly two dense layers.

[0078] Example 5. In the method of example 1, the natural language processing model includes at least one bidirectional layer.

[0079] Example 6. In the method of Example 5, the natural language processing model is a long-term memory (LSTM) model.

[0080] Example 7. The method of any of Examples 1 to 6 further includes preprocessing the text data with one or more processors to remove words and / or characters that are not used for classification before processing the text data with the natural language processing model.

[0081] Example 8. The method of any of Examples 1-7, wherein the plurality of questions correspond to a plurality of respective word strings in the text data, and the method further includes preprocessing the text data by converting each word string into a respective number string with one or more processors prior to processing the text data with the natural language processing model.

[0082] Example 9. The method of example 8, further comprising preprocessing, by one or more processors, the text data prior to processing the text data with the natural language processing model by padding each word string such that all vectors representing each word string have equal sequence length.

[0083] Example 10. The method of any of Examples 1-9, the method including causing, by one or more processors, at least a subset of the plurality of questions to be displayed in a manner indicative of one or more classifications.

[0084] Example 11. In the method of example 10, displaying at least a subset of the plurality of questions in a manner indicative of one or more classifications includes selectively displaying or not displaying each question based on (i) a classification among the one or more classifications to which the question corresponds and (ii) a filter setting selected by a user.

[0085] Example 12. In the method of example 10, causing at least a subset of the plurality of questions to be displayed in a manner indicative of one or more classifications includes causing each question of the subset of the plurality of questions to be displayed in association with a corresponding classification from the one or more classifications.

[0086] Example 13. A system including one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the methods of Examples 1-12.

[0087] Example 14. A method for processing a regulated query, the method including obtaining, by one or more processors, text data representing the regulated query; identifying, by the one or more processors, one or more documents similar to the regulated query, at least in part, by processing the text data with a natural language processing model to identify one or more documents in a database; and storing, transmitting, and / or displaying, by the one or more processors, data indicative of the one or more documents.

[0088] Example 15. In the method of example 14, the natural language processing model is a neural network.

[0089] Example 16. The method of example 14 or 15, wherein the natural language processing model is bidirectional.

[0090] Example 17. The method of any of Examples 14-16, wherein the natural language processing model is a contextualized embedding model.

[0091] Example 18. The method of any of Examples 14-17, wherein processing the text data with a natural language processing model to identify one or more documents in the database includes searching the database using an elastic search engine.

[0092] Example 19. The method of any of Examples 14 to 18, further comprising preprocessing the text data by one or more processors to remove words and / or characters not used for classification prior to processing the text data by the natural language processing model.

[0093] Example 20. In any of the methods of Examples 14 to 19, the method further includes preprocessing the text data by converting word sequences of the text data into number sequences using one or more processors before processing the text data using the natural language processing model.

[0094] Example 21. The method of example 20, further comprising preprocessing the text data by one or more processors prior to processing the text data with the natural language processing model by padding word strings such that vectors representing the word strings have a predetermined sequence length.

[0095] Example 22. A system including one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the methods of Examples 14-21.

[0096] Example 23. A method for processing a regulated question, the method including: obtaining, by one or more processors, text data representing the regulated question; identifying, by the one or more processors, one or more documents similar to the regulated question by at least partially processing the text data with a first natural language processing model to identify one or more documents in a database; generating, by the one or more processors, one or more candidate answers to the regulated question by at least partially processing the identified one or more documents with a second natural language processing model; and storing, transmitting, and / or displaying, by the one or more processors, data indicative of the one or more candidate answers to the regulated question.

[0097] Example 24. In the method of example 23, the first natural language processing model and the second natural language processing model are neural networks.

[0098] Example 25. In the method of example 23 or 24, the first natural language processing model is bidirectional.

[0099] Example 26. In any of the methods of Examples 23-25, the second natural language processing model is a GPT-2 model.

[0100] Example 27. The method of any of Examples 23 to 26 further includes preprocessing the text data with one or more processors to remove words and / or characters not used for classification before processing the text data with the first natural language processing model.

[0101] Example 28. In any of the methods of Examples 23 to 27, the method further includes preprocessing the text data by converting word sequences of the text data into number sequences by one or more processors before processing the text data with the first natural language processing model.

[0102] Example 29. In the method of example 28, the method further includes preprocessing the text data by one or more processors prior to processing the text data with the first natural language processing model by padding word strings such that vectors representing the word strings have a predetermined sequence length.

[0103] Example 30. In any of the methods of Examples 23-29, the method further includes determining, by the one or more processors, a confidence score associated with each of the one or more candidate answers to the regulated question; and storing, transmitting, and / or displaying, by the one or more processors, data indicative of the confidence score associated with each of the one or more candidate answers to the regulated question.

[0104] Example 31. In any of the methods of Examples 23-30, the method further includes, for each of one or more candidate answers to the controlled question, displaying (i) the candidate answer, (ii) an identifier of the one or more documents from which the candidate answer was derived, and (iii) a portion of the text of the document from which the candidate answer was derived.

[0105] Example 32. In any of the methods of Examples 23-31, the method further includes receiving, by the one or more processors, user feedback indicating a usefulness of the one or more candidate answers, and training, by the one or more processors, the first and / or second natural language processing models using the user feedback.

[0106] Example 33. A system including one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the methods of Examples 23-32.

[0107] Example 34. A method of processing a regulated question, the method including obtaining, by one or more processors, text data representing the regulated question; generating, by the one or more processors, a summary of the regulated question, at least in part, by processing the text data with a natural language processing model; and storing, transmitting, and / or displaying, by the one or more processors, data indicative of the summary.

[0108] Example 35. The method of Example 34, wherein the natural language processing model is a neural network.

[0109] Example 36. The method of example 35, wherein the natural language processing model is bidirectional.

[0110] Example 37. A system including one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to perform any of the methods of Examples 34-36.

[0111] Certain embodiments of the present disclosure relate to non-transitory computer-readable storage media having computer code that performs various computer-implemented operations. Terms such as "computer-readable storage medium" may be used herein to include any medium capable of storing or encoding a set of instructions or computer code that performs the operations, methods, and techniques described herein. The medium and computer code may be specially designed and constructed for the purposes of the embodiments of the present disclosure, or may be of the kind known and available to those skilled in the computer software art. Examples of computer-readable storage media include, but are not limited to, magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs, holographic devices, magneto-optical media such as optical disks; hardware devices specially configured for storing and executing program code, such as ASICs, programmable logic devices ("PLDs"), and ROM and RAM devices.

[0112] Examples of computer code include machine code produced by a compiler and files containing high-level code executed by a computer using an interpreter or compiler. For example, embodiments of the present disclosure may be implemented using Java, C++, or other object-oriented programming languages ​​and development tools. Additional examples of computer code include encryption and compression code. Furthermore, embodiments of the present disclosure may be downloaded as a computer program product and transferred from a remote computer (e.g., a server computer) to a requesting computer (e.g., a client computer or another server computer) over a transmission channel. Other embodiments of the present disclosure may be implemented in hardwired circuitry in the alternative to or in combination with machine-executable software instructions.

[0113] As used herein, the singular terms "a," "an," and "the" may include plural referents unless the context clearly dictates otherwise.

[0114] As used herein, the terms "connect," "connected," and "connection" refer to (and are representative of) an operative coupling or association. Connected elements can be directly or indirectly coupled to one another, for example, through pairs of other elements.

[0115] The terms "nearly," "substantially," "substantial," and "about" are used herein to describe and explain slight differences. When used with events or circumstances, these terms may refer to the exact occurrence of the event or circumstances, as well as the approximation of the occurrence of the event or circumstances. For example, when used with a numerical value, these terms may refer to a variation range of ±10% or less of the numerical value, such as ±5% or less, ±4% or less, ±3% or less, ±2% or less, ±1% or less, ±0.5% or less, ±0.1% or less, or ±0.05% or less. For example, two numerical values ​​can be considered "substantially" identical if the difference between the two numerical values ​​is ±10% or less of the average of the numerical values, such as ±5% or less, ±4% or less, ±3% or less, ±2% or less, ±1% or less, ±0.5% or less, ±0.1% or less, or ±0.05% or less.

[0116] Additionally, amounts, ratios, and other numerical values ​​may be presented herein in a range format. It should be understood that such range formats are used for convenience and brevity and should be understood to be flexible and include not only the numerical values ​​explicitly stated as the limits of a range, but also all individual numerical values ​​or subranges contained within that range as if each numerical value and subrange were expressly stated.

[0117] Although the present disclosure has been described and illustrated with reference to specific embodiments, these descriptions and illustrations are not intended to limit the present disclosure. Those skilled in the art will recognize that various modifications may be made and equivalents may be substituted without departing from the true spirit and scope of the present disclosure as defined by the appended claims. The illustrations may not necessarily be drawn to scale. Differences between the artistic representations in the present disclosure and the actual devices may occur due to manufacturing processes, tolerances, and / or other reasons. There may be other embodiments of the present disclosure that are not specifically illustrated. The specification (other than the claims) and drawings should be considered illustrative rather than restrictive. Modifications may be made to adapt a particular situation, material, composition, technique, or process to the objective, spirit, and scope of the present disclosure. All such modifications are intended to be within the scope of the claims appended hereto. Although the techniques disclosed herein have been described with particular operations performed in a particular order, it will be understood that these operations may be combined, divided into parts, or reordered to form equivalent techniques without departing from the teachings of the present disclosure. Thus, unless specifically indicated herein, the order and grouping of the operations is not a constraint on the present disclosure.

Claims

1. 1. A method of processing regulatory queries, comprising: obtaining, by one or more processors, text data representing a plurality of control questions; generating, by the one or more processors, one or more classifications of the plurality of regulated questions by processing the text data at least in part through a natural language processing model; storing, transmitting, and / or displaying, by said one or more processors, data indicative of said one or more classifications; A method comprising:

2. The method of claim 1 , wherein the natural language processing model is a deep feedforward neural network.

3. 3. The method of claim 2, wherein the deep feedforward neural network includes exactly one global max pooling layer and multiple dense layers.

4. The method of claim 3 , wherein the deep feedforward neural network includes exactly two dense layers.

5. The method of claim 1 , wherein the natural language processing model includes at least one bidirectional layer.

6. The method of claim 5 , wherein the natural language processing model is a long short-term memory (LSTM) model.

7. 7. The method of claim 1, further comprising preprocessing the text data with one or more processors to remove words and / or characters that are not used for classification before processing the text data with the natural language processing model.

8. 7. The method according to claim 1, wherein the plurality of questions correspond to a plurality of respective word strings in the text data, and the method further comprises: and preprocessing the text data by the one or more processors by converting each of the word sequences into a respective number sequence before processing the text data with the natural language processing model.

9. and preprocessing the text data by the one or more processors prior to processing the text data with the natural language processing model by padding each word sequence so that all vectors representing each word sequence have equal sequence lengths. The method of claim 8 further comprising:

10. displaying, by the one or more processors, at least a subset of the plurality of questions in a manner indicative of the one or more classifications. The method according to any one of claims 1 to 6, comprising:

11. causing the at least one subset of the plurality of questions to be displayed in a manner indicative of the one or more classifications; Selectively displaying or not displaying each question based on (i) the category of the one or more categories to which the question corresponds, and (ii) a filter setting selected by the user. The method of claim 10, comprising:

12. causing the at least one subset of the plurality of questions to be displayed in a manner indicative of the one or more classifications; displaying each question of the subset of questions in association with a corresponding classification from the one or more classifications; The method of claim 10, comprising:

13. 1. A system comprising: one or more processors; one or more memories storing instructions that, when executed by said one or more processors, cause said one or more processors to perform the method of any one of claims 1 to 6; A system including:

Images