Classifying documents using domain-specific natural language processing models

Abstract

Provided herein are embodiments of systems, apparatus, devices, methods, and / or computer program products, and / or combinations and subcombinations thereof, for classifying documents using CNN and BiLSTM.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 123,336, filed December 9, 2020, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] background Entities such as businesses, government agencies, and educational institutions often receive thousands of documents that contain a combination of text, images, graphs, tables, and other forms of data / information / knowledge representation. These documents may be of different types including Microsoft Word, Microsoft Excel documents, png, tiff, jpg, raw, gif, PDF, emails, text files, handwritten notes, HTML, XML scanned documents, etc. Manually classifying and prioritizing such documents based on their content can be a tedious and error-prone task. Entities have attempted to automate this process using certain machine learning algorithms such as natural language processing (NLP). However, traditional NLP models often fall short in accurately classifying documents. For example, traditional NLP models cannot assign domain-specific labels to words or phrases to accurately classify documents.

[0003] Additionally, extracting the information manually or using highly intelligent third-party tools to extract the textual content of each PDF with acceptable accuracy (e.g., optical character recognition (OCR)) and correctly extract these data back into a machine-readable format is tedious, time-consuming, and error-prone. Furthermore, traditional methodologies implementing traditional machine learning models can face many obstacles when attempting to extract text from documents, such as optical clarity, alphanumeric characters, and orientation. Thus, traditional methods of classifying and prioritizing documents can be tedious, costly, and error-prone. Summary of the Invention [Problem to be solved by the invention]

[0004] overview Provided herein are embodiments of systems, apparatus, devices, methods, and / or computer program products, and / or combinations and sub-combinations thereof, for classifying documents using domain-specific NLP models. [Means for solving the problem]

[0005] In a given embodiment, a method for classifying documents includes receiving, by one or more computing devices, a set of documents and metadata for each document in the set of documents. The set of documents corresponds to a domain. The method further includes generating, by the one or more computing devices, a set of word embeddings for each document in the set of documents. Each word embedding includes one or more words from a respective document. The method further includes tokenizing, by the one or more computing devices, each word embedding in the set of word embeddings into a set of segments. Each segment includes a word from the word embedding. The method further includes training a learning model to classify each document of the set of documents of the domain by recursively performing the following steps: decomposing, by the one or more computing devices, each of the segments of the set of segments of each document of the set of documents into a set of features; assigning, by the one or more computing devices, a part-of-speech tag to each of the segments of the set of segments for each document of the set of documents based on a predetermined weight assigned to each feature of the set of features of the corresponding segment; assigning, by the one or more computing devices, a dependency tag to each of the segments of the set of segments of each document of the set of documents based on the part-of-speech tag assigned to the corresponding segment and the predetermined weight assigned to each feature of the set of features of the corresponding segment; assigning, by the one or more computing devices, a named entity recognition (NER) label from a set of predefined labels corresponding to the domain to each of the segments of the set of segments of each document of the set of documents based on the part-of-speech tag and the dependency tag assigned to the corresponding segment and the predetermined weight assigned to each feature of the set of features of the corresponding segment; and validating, by the one or more computing devices, the assigned NER label by comparing metadata of each document with the assigned NER label of the respective document.

[0006] In a given embodiment, a system for classifying documents includes a memory and a processor coupled to the memory. The processor is configured to receive a set of documents and metadata for each document in the set of documents. The set of documents corresponds to a domain. The processor is further configured to generate a set of word embeddings for each document in the set of documents. Each word embedding includes one or more words from a respective document. The processor is further configured to tokenize each word embedding in the set of word embeddings into a set of segments. Each segment includes a word from the word embedding. Further, the processor is further configured to train, with the one or more computing devices, a learning model to classify each document of the set of documents in the domain by recursively: decomposing each of the segments of the set of segments of each document of the set of documents into a set of features; assigning a part-of-speech tag to each of the segments of the set of segments for each document of the set of documents based on a predetermined weight assigned to each feature of the set of features of the corresponding segment; assigning a dependency tag to each of the segments of the set of segments of each document of the set of documents based on the part-of-speech tag assigned to the corresponding segment and the predetermined weight assigned to each feature of the set of features of the corresponding segment; assigning a named entity recognition (NER) label from a set of predefined labels corresponding to the domain to each of the segments of the set of segments of each document of the set of documents based on the part-of-speech tag and the dependency tag assigned to the corresponding segment and the predetermined weight assigned to each feature of the set of features of the corresponding segment; and validating the assigned NER label by comparing metadata of each document with the assigned NER label of the respective document.

[0007] In a given embodiment, a non-transitory computer-readable medium having instructions stored thereon, the execution of which by one or more processors of a device causes the one or more processors to perform operations including receiving a set of documents and metadata for each document in the set of documents. The set of documents corresponds to a domain. The operations further include generating a set of word embeddings for each document in the set of documents. Each word embedding includes one or more words from a respective document. The operations further include tokenizing each word embedding in the set of word embeddings into a set of segments. Each segment includes a word from the word embedding. The operations further include training, by the one or more computing devices, a learning model to classify each document of the set of documents in the domain by recursively performing the following steps: decomposing each of the segments of the set of segments for each document of the set of documents into a set of features; assigning a part-of-speech tag to each of the segments of the set of segments for each document of the set of documents based on a predetermined weight assigned to each feature of the set of features of the corresponding segment; assigning a dependency tag to each of the segments of the set of segments for each document of the set of documents based on the part-of-speech tag assigned to the corresponding segment and the predetermined weight assigned to each feature of the set of features of the corresponding segment; assigning a named entity recognition (NER) label from a set of predefined labels corresponding to the domain to each of the segments of the set of segments for each document of the set of documents based on the part-of-speech tag and the dependency tag assigned to the corresponding segment and the predetermined weight assigned to each feature of the set of features of the corresponding segment; and validating the assigned NER label by comparing metadata of each document with the assigned NER label of the respective document.

[0008] In a given embodiment, a method for classifying a document includes receiving, by one or more computing devices, a request to classify a document corresponding to a domain; generating, by the one or more computing devices, word embeddings that include one or more words of the document; and tokenizing, by the one or more computing devices, the word embeddings corresponding to the domain into a set of segments. The method further includes decomposing, by one or more computing devices, each of one or more strings of the document into a set of new features; assigning, by the one or more computing devices, a part-of-speech tag to each new segment of the set of segments of the new document based on a predefined weight assigned to each feature of the set of features of the corresponding segment using the trained learning model; assigning, by the one or more computing devices, a dependency label to each segment of the set of segments of the document based on the part-of-speech tag assigned to the corresponding segment and the predefined weight assigned to each feature of the set of features of the corresponding segment using the trained learning model; assigning, by the one or more computing devices, an NER label to each segment of the set of segments of the document from a set of predefined labels corresponding to the domain based on the part-of-speech tag and dependency tag assigned to the corresponding segment and the predefined weight assigned to each feature of the set of features of the corresponding segment using the trained learning model; and classifying, by the one or more computing devices, the document corresponding to the domain based on the assigned NER label using the trained learning model.

[0009] In a given embodiment, a method for training an NLP model includes receiving, by one or more computing devices, a set of documents and metadata for each document in the set of documents. The set of documents corresponds to pharmacovigilance. The method further includes generating, by the one or more computing devices, a set of word embeddings for each document in the set of documents. Each word embedding includes one or more words from a respective document. The method further includes tokenizing, by the one or more computing devices, each word embedding in the set of word embeddings into a set of segments. Each segment includes a word from the word embedding.The method further includes training a learning model to classify each document of the set of documents of the domain by recursively performing the following steps: decomposing, by the one or more computing devices, each of the segments of the set of segments of each document of the set of documents into a set of features; assigning, by the one or more computing devices, a part-of-speech tag to each of the segments of the set of segments for each document of the set of documents based on a predetermined weight assigned to each feature of the set of features of the corresponding segment; assigning, by the one or more computing devices, a dependency tag to each of the segments of the set of segments of each document of the set of documents based on the part-of-speech tag assigned to the corresponding segment and the predetermined weight assigned to each feature of the set of features of the corresponding segment; assigning, by the one or more computing devices, a named entity recognition (NER) label from a set of predefined labels corresponding to pharmacovigilance to each of the segments of the set of segments of each document of the set of documents based on the part-of-speech tag and the dependency tag assigned to the corresponding segment and the predetermined weight assigned to each feature of the set of features of the corresponding segment; and validating, by the one or more computing devices, the assigned NER label by comparing metadata of each document with the assigned NER label of the respective document. In response to fully training the learning model, the learning model is configured to classify the pharmacovigilance documents based on case validity, severity, fatality, and causality.

[0010] In a given embodiment, a method for classifying pharmacovigilance documents using a natural language processing (NLP) model includes receiving, by one or more computing devices, a request to classify the pharmacovigilance document; generating, by the one or more computing devices, an output including named entity recognition (NER) labels for one or more words in the pharmacovigilance document using a learning model configured to implement a combination of a convolutional neural network (CNN) and a bidirectional long-term short-term (BiLSTM) algorithm; and classifying, by the one or more computing devices, the pharmacovigilance document using the NER labels based on validity, severity, fatality, and causality of the cases.

[0011] BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and form a part of this specification, illustrate the present disclosure and, together with the description, further serve to explain the principles of the disclosure and to enable those skilled in the art to make and use the disclosure. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram of a system for classifying documents using domain-specific NLP models, according to an exemplary embodiment. [Diagram 2] FIG. 1 is a block diagram illustrating a process for training a learning model for classifying documents according to an example embodiment. [Diagram 3] FIG. 1 is a block diagram of an example of a learning model implementing a CNN algorithm, according to an exemplary embodiment. [Figure 4] 1 illustrates example documents used to train a learning model, according to an example embodiment. [Diagram 5] FIG. 1 is a block diagram illustrating a model design framework for a learning model in accordance with an illustrative embodiment. [Figure 6] FIG. 10 is a block diagram of a model approval flow in accordance with an illustrative embodiment; [Figure 7]1 is a graph showing F1 score in recognizing entities (e.g., words or phrases) in documents using a trained learning model, according to an example embodiment. [Figure 8] 1 is a graph illustrating a loss function of a learning model, in accordance with an example embodiment; [Figure 9] 1 illustrates a fully trained learning model for summarizing documents, according to an exemplary embodiment. [Figure 10] 11 is a flowchart illustrating a process for training in accordance with an illustrative embodiment. [Figure 11] 1 is a flowchart illustrating a process for classifying documents using a trained learning model in accordance with an example embodiment. [Figure 12] FIG. 2 is a block diagram of exemplary components of a device according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The drawing in which an element first appears is typically indicated by the leftmost digit or digits in the corresponding reference number. In the drawings, like reference numbers can indicate identical or functionally similar elements.

[0014] Detailed Description Provided herein are embodiments of systems, apparatus, devices, methods, and / or computer program products, and / or combinations and sub-combinations thereof, for classifying documents using image analysis.

[0015] As mentioned above, traditional methods for classifying and prioritizing documents can be cumbersome, costly, and error-prone. For example, in the field of pharmacovigilance (PV) operations, companies receive Individual Case Safety Reports (ICSRs) for various drugs. An ICSR is a written report of an adverse event experienced by a patient undergoing a particular treatment or taking a particular drug, which may potentially be related to that treatment or drug.

[0016] For an ICSR to be considered "valid", it must contain information on four elements: an identifiable patient, an identifiable reporter, the suspected drug, and the adverse event. If the ICSR is valid, it is determined whether the adverse event described is a "serious" adverse event. An adverse event is serious if it meets one of the following requirements: it is fatal or life-threatening, requires inpatient hospitalization or prolongs an existing hospitalization, results in persistent or significant disability or incapacity, results in congenital disability, or is otherwise medically significant because treatment and / or intervention is required to prevent one of the aforementioned requirements. Furthermore, when conducting clinical trials of a drug or other product, it can be determined whether the adverse effect presented in the form of an ICSR is a serious unexpected adverse reaction (SUSAR).

[0017] An ICSR may correspond to a specific case. Different regulatory bodies may require that actions be taken on cases that have a corresponding ICSR. Regulatory bodies may provide different timelines for different cases. For example, if a case involves a serious adverse effect that is documented in the ICSR, the case may be prioritized so that the company can address the case. Conversely, if a case involves a non-serious adverse effect in the ICSR, the case may be given a lower priority.

[0018] The ICSR may be provided in a variety of formats, such as Microsoft Word, Microsoft Excel documents, png, tiff, jpg, raw, gif, email, PDF, text file, handwritten notes, HTML, XML scanned documents, etc. The ICSR document may be a combination of formats. For example, the ICSR document may be in .doc format but may contain embedded JPEG images. In another example, part of the ICSR document may be an email message and another part may be in Microsoft Word or Microsoft Excel format.

[0019] ICSRs may come from various reporters such as pharmacies, clinicians, or patients. Furthermore, each of the documents may include the reported adverse effects of the drug along with other information about the drug. Based on the content of the document, the company may need to determine, for example, whether the document is a valid ICSR report, the severity of the adverse effects described in the ICSR document, and the severity, relevance, and predictability (SRE) of the adverse effects described in the ICSR document. Given the number of reports and the various types of formats of the reports, classifying the reports in such a manner may prove to be a challenging task. Thus, traditional methods may not be able to effectively and efficiently classify ICSR reports.

[0020] For example, traditional methods may involve a subject matter expert (SME) manually reviewing each ICSR document and making a decision. An individual may manually extract relevant information from the ICSR document and enter the information into a database, which is then reviewed by a medical expert to classify the ICSR document. However, a company may receive thousands of ICSR documents in a short period of time. Given the large number of ICSR documents that may be received by a company, manual review of ICSR documents may be a tedious task. Furthermore, many ICSR documents may be irrelevant because they may not be valid documents, may not have a significant effect, or may not have a significant, relevant, or expected effect. This may result in significant backlogs and delays in the processing of relevant and important ICSR documents.

[0021] Traditional methods can also include using machine learning algorithms that require converting documents to text (e.g., via optical character recognition (OCR)) prior to manipulation. However, given the complexity of OCR and the creation of normalized templates, traditional machine learning algorithms require significant time and human and monetary resources to train, implement, and update the algorithms. Thus, these machine learning algorithms can be operationally inefficient and costly to train and implement.

[0022] In a given embodiment, a server may receive a request to train a learning model to classify documents and identify entities within the documents that are specific to a domain. For example, the learning model may identify entities within the documents to automatically summarize the documents. The content of the documents may include one or more strings. Additionally, the documents may include corresponding metadata. The metadata may be annotations that label one or more strings within the documents. The annotations may be specific to a domain.

[0023] The server can train the learning model to classify documents specific to the domain by generating word embeddings for each document. The server can tokenize each word embedding into segments, including one or more words of each word embedding. The server can train the learning model by recursively decomposing each of the segments of each document into a set of features, assigning part-of-speech tags to one or more words corresponding to each respective segment of each respective document based on a predefined weight assigned to each feature of the set of features of the respective segment, and assigning dependency labels to one or more words corresponding to each respective segment of each respective document based on the part-of-speech tags assigned to each one or more words and the predefined weight assigned to each feature of the set of features of the respective segment. Training the learning model can further include recursively assigning named entity relation (NER) labels to one or more words corresponding to each respective segment of each respective document from a set of predefined labels corresponding to the domain based on the part-of-speech tags and dependency tags assigned to each one or more words and the predefined weight assigned to each feature of the set of features of the respective segment, and verifying the assigned labels by comparing metadata of each document to the assigned labels of the respective document.

[0024] The server may receive a request to classify a document corresponding to a domain using a trained learning model trained to classify documents specific to the domain. The server may tokenize the document into segments including one or more words of one or more strings of the document. The server may decompose each of the segments of the document into a set of features. The server may assign a part-of-speech tag to one or more words corresponding to each respective segment of the document based on a predefined weight assigned to each feature of the set of features of the respective segment. The server may assign a dependency label to one or more words corresponding to each respective segment based on the part-of-speech tag assigned to each one or more words and the predefined weight assigned to each feature of the set of features of the respective segment. Furthermore, the server may assign a named entity relation (NER) label to one or more words corresponding to each respective segment from a set of predefined labels corresponding to the domain based on the part-of-speech tag and dependency tag assigned to each one or more words and the predefined weight assigned to each feature of the set of features of the respective segment. The server may classify the document based on each NER label assigned to the words in the document.

[0025] The above configuration allows for processing and classification of multiple document formats and languages ​​without transcribing and retrieving data from source documents. The above configuration reduces data input for case processing and enables inferential analysis and search signal management. Thus, the above configuration avoids text processing, including but not limited to transcription and translation, by leveraging domain-specific NLP that implements a convolutional neural network (CNN) along with a bidirectional long short-term memory (BiLSTM) model. This methodology enhances the speed at which models can be trained to understand domain concepts within the PV domain. Additionally, the above configuration minimizes the training and maintenance efforts of traditional NLP models.

[0026] Furthermore, the above configuration allows for labeling one or more strings in a document using a domain-specific NLP model so that the document can be more accurately classified. For example, the NLP model can be specific to PV. Thus, the NLP model can be successfully used across PV systems.

[0027] FIG. 1 is a block diagram of a system for classifying documents using a domain-specific NLP model. The system may include a server 100, a client device 110, and a database 120. The devices of the system may be connected via a network. For example, the devices of the system may be connected via wired connections, wireless connections, or a combination of wired and wireless connections. In an exemplary embodiment, one or more portions of the network may be an ad-hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the public switched telephone network (PSTN), a cellular network, a wireless network, a WiFi network, a WiMax network, any other type of network, or a combination of two or more such networks. Alternatively, the server 100, the client device 110, and the database 120 may be located on a single physical or virtual machine.

[0028] In some embodiments, server 100 and database 120 can reside in a cloud computing environment. In other embodiments, server 100 may reside in a cloud computing environment and database 120 resides outside of the cloud computing environment. Additionally, in other embodiments, server 100 may reside outside of a cloud computing environment and database 120 resides in a cloud computing environment.

[0029] The client device 110 may be a device operated by an individual associated with an administrator (e.g., a programmer, a user, etc.) of the server 100. The client device 110 may include a training application 112 and a classification application 114. A cloud computing environment may also host the training application 112 and the classification application 114. Alternatively, one or both of the training application 112 and the classification application 114 may be installed on the client device 110.

[0030] The training application 112 and the classification application 114 may be executable applications configured to interface with the server 100. The training application 112 may send a request to the server 100 to train a learning model to classify documents using image analysis. The classification application 114 may be configured to send a request to the server 100 to classify documents using the learning model. The classification application 114 may also be installed and executed on a third-party user device. In this regard, an authorized third party may send a request to classify a document using the server 100. The documents may be stored in the database 120. The database 120 may be one or more data storage devices configured to store documents of various types and formats.

[0031] The learning engine 102 may include a learning model 104. The learning model 104 may implement a natural language processing (NLP) framework, which is configured to recursively implement deep machine learning algorithms, such as convolutional neural networks (CNNs) and BiLSTMs, to classify and prioritize documents. The learning model 104 may be a domain-specific learning model configured to classify domain-specific documents. The learning model 104 may assign multiple classifications to a given document. Additionally, the learning model 104 may be configured to summarize a given document. Each of the classifications is described in more detail below. In some embodiments, documents may be classified using fewer or additional learning modules.

[0032] FIG. 2 is a block diagram illustrating a process of training a learning model for classifying documents, according to an example embodiment. FIG. 2 is described in conjunction with FIG. 1. In a given embodiment, a client device 110 may receive a request to train a learning model 104 to classify documents corresponding to a domain. The learning model 104 may be an NLP framework configured to implement CNN and Bidirectional Long Term Short Term (BiLSTM) algorithms to classify documents.

[0033] The training application 112 can build a statistical NER model. The statistical NER model can be used to implement a rule-based recognition system. For example, the statistical NER model can provide domain-specific rules on how to tag strings in a document. Furthermore, the statistical NER model can be a dictionary or ontology used by the learning model 104 to recognize words or phrases in a document. The statistical NER model can be specifically tied to a particular domain. For example, a statistical NER model built using MedDRA can include terms or expressions that specifically correspond to concepts in the PV domain. The training application 112 can load the statistical NER model into the learning model 104. The statistical NER model can be used in combination with a standard language (e.g., English, Spanish, French, etc.).

[0034] The request may include training data 200. The training data 200 may include documents (and concepts) corresponding to a domain. The documents may include text 202 (e.g., one or more strings) and labels 204 assigned to the text 202. The labels 204 may be from a predefined set of labels corresponding to the domain. Furthermore, each label of the labels 204 may be assigned to one or more strings (e.g., words or phrases) of the text 202. The labels assigned to one or more strings may define the strings. For example, the labels 204 may correspond to entities or fields of a particular domain. Thus, a given label of the labels 204 assigned to a given string indicates that the given string corresponds to a given entity or field of the particular domain. The labels 204 may be included in the metadata of each document.

[0035] The training application 112 can send the training data 200, the labels (e.g., metadata) 204 corresponding to the training data 200, and the parameters to the learning engine 102 for training learning models 104. The learning engine 102 can receive the training data 200 and the labels 204.

[0036] The learning model 104 may generate word embeddings for each of the documents in the training data 200. Word embeddings may be vector representations of the words of a document. A vector may be an n-dimensional vector space in which words that share common context and semantics are located close to each other in the vector space. The learning model 104 may use Bloom embeddings for each of the documents in the training data 200. Bloom embeddings are compact vector representations of the words of a document. Word embeddings or Bloom embeddings may be generated using a statistical NER model.

[0037] The learning model 104 may tokenize the word embeddings (or Bloom embeddings) into segments such as words, characters, and punctuation. Tokenization segments each document based on rules specific to the language and the particular domain. The learning model 104 may also use statistical models to segment each document. For example, if a document contains the phrase "I live in the USA," the learning application 112 may determine that the first period after "USA" corresponds to the abbreviation for "USA" and the second period corresponds to the end of the sentence. Thus, the tokenization of the phrase may be segmented as follows: [I][live][in][the][USA][.]. Each segment may include a single word, a partial word, or two or more words.

[0038] The learning model 104 may implement a CNN algorithm to decompose each segment into a set of features and generate a vector (e.g., a one-dimensional vector) corresponding to each of the segments using the set of features of each respective segment. The learning model 104 may assign a weight to each of the sets of features. The CNN algorithm is described in further detail with reference to FIG. 3.

[0039] The learning model 104 may apply weights to the vectors to generate a resultant vector. The weights may be included in the parameters received from the training application 112. The learning model 104 may assign part-of-speech tags to words in a segment corresponding to the vector based on the resulting vector and the statistical NER model. The part-of-speech tags may indicate whether the word is a noun, verb, adjective, etc. The learning model 104 may predict the part of speech of a word in a segment given a context. For example, the learning model 104 may determine that the word following the word "the" must be a noun based on the rules of the English language. The learning model 104 may use predefined rules to make inferences about words and phrases in a document and identify relationships between words in the document. Additionally, the learning model 104 may use word embeddings to identify relationships between words. Additionally, the learning model 104 may use a statistical NER model, including domain-specific dictionaries and ontologies, to understand the vocabulary used in the document.

[0040] The learning model 104 may also assign dependency tags to words in each segment of each respective document based on the resulting vectors corresponding to each segment, the statistical NER model, and part-of-speech tags assigned to the words in each segment. The dependency tags may define a relationship between two or more words. For example, in the phrase "lazy dog," the learning engine 104 may determine that the word "lazy" modifies "dog." This dependency may be represented by a tag (e.g., an amod tag). The learning model 104 may use predefined rules to make inferences about words and phrases in the document and identify relationships between words in the document. Additionally, the learning model 104 may use word embeddings to identify relationships between words. Additionally, the learning model 104 may use a statistical NER model, including domain-specific dictionaries and ontologies, to understand the vocabulary used in the document.

[0041] The learning model 104 can assign NER labels to words in each segment of each respective document based on the resulting vectors corresponding to each segment, the statistical NER model, and part-of-speech and dependency tags assigned to each word in each segment. The NER labels can be selected from a predefined set of labels corresponding to a domain. The NER labels indicate that the words correspond to fields or entities in the domain. The learning model 104 can use predefined rules to make inferences about words and phrases in the document and identify relationships between words in the document. Additionally, the learning model 104 can use word embeddings to identify relationships between words. Additionally, the learning model 104 can use the statistical NER model, including domain-specific dictionaries and ontologies, to understand the vocabulary used in the document.

[0042] The learning model 104 can verify the NER labels assigned to the words of each document based on the respective labels 204 corresponding to each document. Based on the verification results and the gradients 208, the learning model 104 can modify the weights assigned to each feature, tokenize each document to generate new segments for each document, generate new vectors based on the new segments and the new weights, assign part-of-speech tags to the words of the new segments based on the new vectors, assign dependency tags to the words of the new segments based on the part-of-speech tags assigned to the words and the new vectors, assign NER labels to the words of the new segments based on the part-of-speech tags and dependency tags assigned to the words and the new vectors, and verify the NER labels based on the labels 204. The learning model 104 can recursively modify the weights and perform these steps until the learning model 104 assigns NER labels with a desired accuracy. In some embodiments, the part-of-speech tags and dependency tags can also be verified.

[0043] When the learning model 104 is assigning NER labels with a desired accuracy, the learning model 104 can become a fully trained learning model 210. The fully trained learning model 210 is shown as a distinct component of the learning model 104. The fully trained learning model 210 illustrates the process of training the learning model 104. However, it should be understood that the learning model 104 can remain the same component in the system even after it is fully trained.

[0044] The client device 110 can receive a request to classify a document using the fully trained learning model 210. The request can include the document. The classification application 114 can send the document and parameters to the fully trained learning model 210. The fully trained learning model 210 can generate word embeddings (or Bloom embeddings) for the document.

[0045] The fully trained learning model 210 can tokenize the word embeddings (or Bloom embeddings) to generate segments of the document, generate vectors based on the segments and weights included in the parameters, assign part-of-speech tags to words of the segment based on the statistical NER model and vectors, assign dependency tags to words of the segment based on the part-of-speech tags assigned to the words, the statistical NER model, and vectors, and assign NER labels to words of the segment based on the part-of-speech tags and dependency tags assigned to the words, the statistical NER model, and vectors. The fully trained learning model 210 can generate an output 212 in response to the assignment of NER labels. Additionally, the fully trained learning model 210 can classify the document based on the NER labels.

[0046] In some embodiments, the fully trained learning model 210 can extract words and phrases from documents and their respective NER labels. The fully trained learning model 210 can use the words and phrases extracted from documents and their respective NER labels, along with words and phrases extracted from other documents and their respective NER labels to build a knowledge base. The knowledge base can be a graph-based structure including nodes connected using edges. The nodes can include the extracted words and phrases and their respective NER labels. The fully trained learning model 210 can connect the nodes using edges based on identifying relationships between the nodes. The fully trained learning model 210 can determine relationships between nodes that store words or phrases based on the NER labels of each word or phrase. The knowledge base can be stored in the database 120.

[0047] As a non-limiting example, the above-mentioned system for classifying documents using image analysis can be used to classify ICSR documents. ICSR documents can also include literature and clinical reports. As mentioned above, ICSR documents include information about the patient, geography, adverse effects, quality and compliance characteristics of the ICSR, benefit-risk characteristics, product details, study details, and consumer complaints, legal concepts, or other medical concepts related to the use of FDA-regulated products. Companies in the pharmaceutical industry may need to process ICSR documents to determine whether any action is required for a particular product.

[0048] The ICSR workflow may include three process blocks: case acceptance, case processing, and case reporting. At acceptance, the PV department receives ICSRs globally from different sources in various formats and languages. Reporting is done from various reporters, healthcare professionals, and non-healthcare professionals, and through various mediums such as email, fax, mail, and telephone. At the time of case acceptance, several key assessments are made, which are crucial in routing the case considering its severity to meet predefined regulatory guidelines.

[0049] Compliance with regulatory authorities is determined based on country-specific regulatory reporting needs within their respective specified timelines. Therefore, ex ante prioritization must be precise to limit the propagation of work efforts made on less urgent reports. The evaluation for prioritization can include the following key characteristics: case validity (valid or invalid), case severity (major or non-major), relevance (related or not related to the suspected product), and SRE of the adverse effect (labeled or unlabeled). Case validity can indicate whether the ICSR document is a valid document. Case severity can indicate whether the adverse effect described in the ICSR document is a major or non-major. SRE can indicate whether the adverse effect is a major, relevant, and expected (e.g., labeled on the product) effect.

[0050] A company may need to take action on a particular product if the adverse effects described in a valid ICSR document are significant and unexpected. As a result, the learning model 104 may be trained to classify the validity, severity, fatality, and causality of a given ICSR document case. The learning model 104 may also be trained to identify adverse effects in a Structured Product Label (SPL) of an FDA approved drug for predictiveness and to identify possible off-label product use. Additionally, the learning model 104 may be trained to identify entities within the document. The learning model 104 may be used to generate a summary of the document based on the identified entities. The learning model 104 may be trained to understand the context of the document such that an accurate summary of the document can be generated.

[0051] For example, the client device 110 may receive a request to train the learning model 104 to classify ICSR documents corresponding to the PV domain. The learning model 104 may be an NLP framework configured to implement CNN and BiLSTM algorithms to classify the documents.

[0052] As a non-limiting example, the learning model 104 can implement spaCy, spaCy(v2.0), or MedSpaCy. SpaCy(v2.0) is an open source software library for advanced NLP that utilizes state-of-the-art convolutional neural network (CNN) models with residual connections and layer-regularized maxout nonlinearity. SpaCy offers much better efficiency than standard BiLSTM solutions for tagging, parsing, named entity recognition, and deep learning integration. In addition, spaCy has GloVe (global vector) support in the English language model. The maximum size of the vector is 2.2 million for the GloVe.840B.300d common crawl (840B tokens, 2.2M vocab, examples, 300d vectors). To support effective handling of the rather large MedDRA vocabulary, an internally implemented Bloom embedding strategy using subword features was used.

[0053] The training application 112 can build a statistical NER model. For example, the training application 112 can embed 81,900 MedDRA entities to create a statistical NER model. The training application 112 can also build a statistical NER model using the Unified Medical Language System (UMLS). The statistical NER model may be incorporated as a spaCy EntityRuler search pattern attribute. The statistical NER model may be a dictionary or ontology used by the learning model 104 to recognize words or phrases in ICSR documents. The training application 112 can load the statistical NER model into the learning model 104. The statistical NER model can be used in combination with standard languages ​​(e.g., English, Spanish, French, etc.).

[0054] The request may include training data 200. The training data 200 may include an ICSR document. The document may include text 202 (e.g., one or more strings) and labels 204 assigned to the text 202. The labels 204 may be from a predefined set of labels corresponding to a PV domain. Furthermore, each label of the labels 204 may be assigned to one or more strings (e.g., words or phrases) of the text 202. The labels assigned to one or more strings may define the strings. For example, the labels 204 may correspond to an entity or field of the PV domain. Thus, in one embodiment, a given label of the labels 204 assigned to a given string indicates that the given string corresponds to a given entity or field of the PV domain. The labels 204 may be included in the metadata of each document.

[0055] As an example, the label 204 may include the following entities:

[0056]

number

[0057] The labels 204 can correspond to 109 PV entities. The F1 score indicates the level of accuracy with which the fully trained learning model 210 can identify a given entity. The F1 score is described in more detail below.

[0058] The training application 112 can send the training data 200, the labels (e.g., metadata) 204 corresponding to the training data 200, and the parameters to the learning engine 102 for training learning model 104. The learning engine 102 can receive the training data 200 and the labels 204. In a non-limiting example, the training data 200 included 20,000 machine-readable ICSRs. These ICSRs were received by Celgene's Drug Safety Division over a two-year period (January 2015-December 2016). Further details regarding the characteristics and sampling of the ICSRs have been published previously (Abatemarco et al., 2018; Mockute et al., 2019). For the hyperparameters, the best-practice values ​​available were used by spaCy.

[0059] The learning model 104 may generate word embeddings (or Bloom embeddings) for each of the documents in the training data 200. The learning model 104 may tokenize the word embeddings (or Bloom embeddings) for each of the documents in the training data 200. More specifically, the learning model 104 may tokenize each document into segments, such as words, characters, punctuation, etc. The tokenization segments each document based on language and domain-specific rules. The learning model 104 may also segment each document using a statistical model.

[0060] The learning model 104 may implement a CNN algorithm to decompose each segment into a set of features and generate a vector (e.g., a one-dimensional vector) corresponding to each of the segments using the set of features of each respective segment.

[0061] The learning model 104 can apply the weights assigned to each feature in the set of features to the vector to generate a resulting vector. The weights may be included in the parameters received from the training application 112. The learning model 104 can assign part-of-speech tags to words in the segment corresponding to the vector based on the resulting vector and the statistical NER model.

[0062] The learning model 104 can also assign dependency tags to the words in each segment of each respective document based on the resulting vectors corresponding to each segment, the statistical NER model, and the part-of-speech tags assigned to the words in each segment.

[0063] The learning model 104 can assign NER labels to words in each segment of each respective document based on the resulting vectors corresponding to each segment, the statistical NER model, and part-of-speech and dependency tags assigned to each word in each segment. The NER labels can be selected from a predefined set of labels corresponding to a domain. The NER labels indicate that the words correspond to a field or entity in the domain. For example, NER labels may be assigned to words or phrases that are important for PV event detection to accurately classify documents.

[0064] The learning model 104 can verify the NER labels assigned to the words of each document based on the respective labels 204 corresponding to each document. Based on the verification results and the gradients 208, the learning model 104 can modify the weights assigned to each feature, tokenize the word (or Bloom) embeddings for each document to generate new segments for each document, generate new vectors based on the new segments and the new weights, assign part-of-speech tags to the words of the new segments based on the new vectors, assign dependency tags to the words of the new segments based on the part-of-speech tags assigned to the words and the new vectors, assign NER labels to the words of the new segments based on the part-of-speech tags and dependency tags assigned to the words and the new vectors, and verify the NER labels based on the labels 204. The learning model 104 can recursively modify the weights and perform these steps until the learning model 104 assigns NER labels with a desired accuracy. The gradients 208 can be a gradient loss function configured to optimize the weights to train the learning model 104.

[0065] In a non-limiting example, a threshold F1 score of 75% was used to determine whether the learning model 104 was fully trained based on training the learning model 104 using 20,000 annotated ICSR documents. The following formula was used to generate the F1 score:

[0066] Accuracy = true positive / (true positive + false positive) Recall rate = true positive / (true positive + false negative) F1=2×(precision×recall) / (precision+recall) If the learning model 104 assigns NER labels with a desired accuracy, the learning model 104 may be a fully trained learning model 210. The client device 110 may receive a request to classify an ICSR document using the fully trained learning model 210. The request may include the ICSR document. The classification application 114 may send the ICSR document and parameters to the fully trained learning model 210. The fully trained learning model 210 may generate word (Bloom) embeddings for the ICSR document. The fully trained learning model 210 may tokenize the word embeddings to generate segments of the document, generate vectors based on the segments and weights included in the parameters, assign part-of-speech tags to words of the segment based on the statistical NER model and vectors, assign dependency tags to words of the segment based on the part-of-speech tags assigned to the words, the statistical NER model, and vectors, and assign NER labels to words of the segment based on the part-of-speech tags and dependency tags assigned to the words, the statistical NER model, and vectors. The fully trained learning model 210 may generate an output 212 in response to the assignment of NER labels.

[0067] The output 212 may include NER labels of important words or phrases required to classify the ICSR document. For example, the output 212 may include the NER label for the word "hospitalization" as "ReportersSeriousHospitalization" and the word "melanoma" may include the NER label as AE10053571.

[0068] The fully trained learning model 210 can classify the ICSR documents based on the NER labels. For example, the fully trained learning model 210 can classify the ICSR documents for case efficacy, severity, fatality, and causality, identify adverse effects present in the Structured Product Label (SPL) for predictiveness of FDA-approved drugs, and identify potential off-label product use.

[0069] The fully trained learning model 210 can generate an output that includes the uncovered entities along with any detected relationships between the product instruction(s) and the associated AEs. As an example, Streamlit can be used to generate a web interface for rendering the output. The fully trained learning model 210 can cause the output to be displayed on the client device 110.

[0070] In some embodiments, the fully trained learning model 210 can extract words and phrases from the ICSR documents and their respective NER labels. The fully trained learning model 210 can use the words and phrases extracted from the ICSR documents and their respective NER labels, along with extracted words and phrases from other documents (ICSR or otherwise) and their respective NER labels to build a knowledge base. The knowledge base can relate to drugs, types of drugs, diseases, patient demographics, etc.

[0071] The knowledge base may be a graph-based structure including nodes connected using edges. The nodes may include extracted words and phrases and their respective NER labels. The fully trained learning model 210 may connect the nodes using edges based on identifying relationships between the nodes. The fully trained learning model 210 may determine relationships between nodes that store words or phrases based on the NER labels of each word or phrase. The knowledge base may be stored in the database 120.

[0072] The knowledge base may be used across different industries to determine different aspects about drugs, types of drugs, diseases, patient demographics, etc. Industries may include technology, regulatory, medical, etc. For example, a regulated industry such as the FDA may use the knowledge base to identify various effects of drugs. In another example, a manufacturer of a drug may use the knowledge base to identify ingredients in a drug that are causing adverse effects. In yet another example, a medical professional or researcher may use the knowledge base to identify drugs or therapies for a disease.

[0073] 3 is a block diagram of an example of a learning model implementing a CNN algorithm according to an exemplary embodiment. FIG. 3 will be described with reference to FIG. 1-2.

[0074] As described above, the learning model 104 can implement a supervised deep learning algorithm, such as CNN, to classify documents based on their corresponding image files. The CNN algorithm 300 can be used to generate vectors that represent each tokenized segment of a document, and the vectors can be used to assign part-of-speech tags, dependency tags, and NER labels.

[0075] The CNN algorithm 300 can be trained in two phases: forward and reverse. The forward phase includes convolutional layers 304 and 308, pooling layers 306 and 310, and fully connected layers 312 and 314. The convolutional layers 304 and 308 can apply filters to the input segment 302 to generate a feature map. The pooling layers 306 and 310 can generate a reduced feature map. The fully connected layers 312 and 314 can then classify the image features using the weights and biases to generate an output 316. The output 316 can be an assigned part-of-speech tag, a dependency tag, and an NER label. The values ​​of the filters, weights, and biases can be parameters received by the CNN algorithm 300 (e.g., from the client device 110).

[0076] In the reverse phase, the CNN algorithm 300 can use backpropagation to determine whether the CNN algorithm can correctly assign part-of-speech tags, dependency tags, and NER labels. The CNN algorithm can update the values ​​of the filters, weights, and biases using a gradient descent algorithm and re-run the forward phase on the input segment 302.

[0077] As an example, the CNN algorithm 300 may be trained using a tokenized segment of a word embedding of a document that includes an input segment 302. The input segment 302 may correspond to one or more words in the document. As a non-limiting example, the input segment 302 may have a size of 160×160 pixels, resulting in a 160×160 dimensional array. The 160×160 dimensional array may be flattened into a 25600 dimensional vector. Each component of the vector may include a value between 0 and 1. The value may describe the intensity of the pixel. The input segment 302 may include a unique JPEG “fingerprint” in grayscale format.

[0078] In the convolutional layer 304, the CNN algorithm 300 can perform feature extraction on the input segment 302. The features can include portions of the input segment 302. For example, the features can be different edges or shapes of the input segment 302. The CNN algorithm can extract different types of features to generate different kinds of feature maps. For example, the CNN algorithm 300 can apply an array of numbers (e.g., a kernel) over different portions of the input segment 302. The kernels are sometimes referred to as filters. As described above, different types of filters can be applied to the input segment 302 to generate different feature maps. For example, a filter for identifying shapes in the input segment 302 can be different than a filter for edge detection. Thus, different kernels can be applied to identify shapes in the input segment 302 compared to edge detection. Each kernel can include a different array of numbers. The values ​​of the filters or kernels can be randomly assigned and optimized over time (e.g., using a gradient descent algorithm). The kernels can be applied as sliding windows over different portions of the input segment 302. The kernels may be summed with a given portion of the input segment 302 to generate an output value. The output value may be included in a feature map. The feature map may include output values ​​from different kernels applied to each portion of the input segment 302. The generated feature map may be a two-dimensional array.

[0079] In the pooling layer 306, the CNN algorithm 300 may reduce the dimensionality of each feature map generated in the convolutional layer 304. In particular, the CNN algorithm 300 may extract portions of a given feature map and discard the rest. Pooling images preserves important features. For example, a feature map may include activated and deactivated regions. Activated regions may include detected features, and deactivated regions may indicate that portions of the segment did not include the feature. Pooling may remove deactivated regions. This reduces the size of the image. The CNN algorithm 300 may perform these operations using max pooling or average pooling in the pooling layer. Max pooling retains higher values ​​of portions of the feature map while discarding the remaining values. Average pooling retains the average value of different portions of the feature map. The CNN algorithm 300 may then generate a reduced feature map for each of the feature maps generated in the convolutional layer 304.

[0080] In the convolutional layer 308, the CNN algorithm 300 can generate additional feature maps based on the reduced feature maps generated in the pooling layer 306. Additionally, in the pooling layer 310, the CNN algorithm 300 can generate further reduced feature maps based on the feature maps generated in the convolutional layer 308. Multiple different convolutional and pooling layers can be added to the CNN algorithm 300.

[0081] The convolutional layers 304 and 308 may also apply a rectified linear unit (ReLU) function to the input segment 302. The ReLU function is applied to the input segment 302 to remove linearity from the input segment 302. For example, the ReLU function may remove all black components from the input segment 302 and retain only gray and white colors. This results in more abrupt changes in color in the input segment 302 and removes linearity from the input segment 302.

[0082] The convolutional layers 304 and 308 and the pooling layers 306 and 310 can be used for feature learning. Feature learning allows the CNN algorithm 300 to identify desired features in the input segment 302 and therefore accurately classify the input segment 302. Thus, by optimizing the convolutional layers 304 and 308 and the pooling layers 306 and 310, the CNN algorithm 300 can apply the correct filters to the input segment 302 to extract the features necessary to classify the input segment 302.

[0083] In the fully connected layer 312, the CNN algorithm 300 may flatten the reduced feature map generated in the pooling layer 310 into a one-dimensional array (or vector). The fully connected layer is a neural network. The CNN algorithm 300 may perform a linear transformation on the one-dimensional array in the fully connected layer 312. The CNN algorithm 300 may perform the linear transformation by applying weights and biases to the one-dimensional array to generate the fully connected layer 314. Initially, the weights and biases may be randomly initialized and optimized over time.

[0084] In the fully connected layer 314, the CNN algorithm 300 may perform nonlinear transformations such as activation layer functions (e.g., softmax or sigmoid) to assign part-of-speech tags, dependency tags, and NER labels to each word in the input segment 302.

[0085] The CNN algorithm 300 can use backpropagation to validate its classification of the input segment 302. As described in FIG. 2, in training mode, the CNN algorithm 300 can receive a document and corresponding document metadata. The metadata can include labels 204 assigned to different words or phrases of the document. The CNN algorithm 300 can compare the NER labels assigned to the words or phrases of the document with the labels 204 corresponding to the document. If the NER labels assigned to the document by the CNN algorithm 300 match the labels 204 corresponding to the document, the CNN algorithm 300 can determine that the NER labels were correctly assigned. Alternatively, if the classification assigned to the document by the CNN algorithm 300 does not match the classification included in the document metadata, the CNN algorithm 300 can determine that the NER labels were incorrectly assigned. In some embodiments, part-of-speech tags and dependency tags can also be validated.

[0086] Backpropagation may also include optimizing input parameters such that part-of-speech and dependency tags and NER labels are more accurately assigned to words or phrases of documents. The input parameters may include kernel values, weights, biases, etc. Gradient descent (e.g., gradient 208) may be used to optimize the parameters.

[0087] Gradient descent is an iterative process for optimizing the CNN algorithm 300. Gradient descent can update the parameters of the CNN algorithm 300 and cause the learning model 104 to assign part-of-speech and dependency tags and NER labels to words or phrases in each document and verify the part-of-speech and dependency tags and NER labels. After each iteration of running the CNN algorithm 300, the parameters may be further updated (or optimized). Each iteration of the CNN algorithm 300 may be referred to as an epoch. The CNN algorithm 300 may be considered fully trained after running a predetermined number of epochs. The predetermined number of epochs may be selected based on a desired accuracy of assigning part-of-speech and dependency tags and NER labels.

[0088] As a non-limiting example, the CNN algorithm 300 may be implemented by the learning model 104 to classify ICSR documents. The learning model 104 may use the CNN algorithm 300 to recognize words or phrases in the ICSR documents, enabling the learning model 104 to determine the validity, severity, fatality, and causality of cases, identify predictiveness present in structured product labels (SPL) of FDA-approved drugs, and identify potential off-label product use.

[0089] As described above, the CNN algorithm 300 may be considered fully trained by running a predetermined number of epochs. A user (e.g., a developer) may determine that the CNN algorithm is fully trained based on the F-1 score and the area under the curve (AUC) score of the CNN algorithm 300. The F-1 score and the area under the curve (AUC) curve may indicate the accuracy of assigning part-of-speech and dependency tags and NER labels to words or phrases of each document.

[0090] FIG. 4 illustrates an exemplary document used to train a learning model, according to an exemplary embodiment. The document 400 may be an ICSR document and may include important words or phrases 402 necessary to classify an ICSR document. The document 400 may be used to train a learning model (e.g., the learning model 104 as shown in FIG. 1) for classifying an ICSR document. The document 400 may include annotations 404 (e.g., the labels 204 as shown in FIG. 2). The annotations 404 may be labels assigned to important words or phrases 402 necessary to classify an ICSR document. For example, the word "Name" may be assigned the label "ReporterTypeHCP". The annotations 204 may be used to verify the results of assigning NER labels to the document 400.

[0091] 5 is a block diagram illustrating a model design framework for a learning model, according to an exemplary embodiment. The model design framework 500 can be implemented by a learning model (e.g., the learning model 104 or the fully trained learning model 210). The model design framework 500 can be used to train the learning model using the training data 200. A dictionary or ontology can be loaded into the model design framework 500. For example, the dictionary or ontology can be MedRNA or UMLS.

[0092] Once the learning model is trained, as described above, the model design framework 500 can build a vocabulary using a custom statistical NER model and an existing linguistic statistical model. The model design framework 500 can receive a document for classification. The model design framework 500 can tokenize the document and implement a CNN algorithm to recognize words or phrases in the document using the vocabulary. The model design framework 500 can assign part-of-speech and dependency tags to words or phrases in the document and assign NER labels to the words or phrases in the document. The model design framework 500 can generate an output 212 that includes NER labels for the words or phrases in the document that are required to classify the document. The model design framework 500 can understand the context of the document when assigning NER labels. For example, even though gastritis is included in the text in the output 212, the complete phrase reads as "gastritis excluded...". This indicates that gastritis does not need to be labeled because it is not important to classify the document.

[0093] FIG. 6 is a block diagram of a model validation flow, according to an example embodiment. Model validation flow 600 can be used to determine whether a learning model assigns part-of-speech and dependency tags and NER labels with an acceptable level of accuracy. At operation 602, a training corpus (e.g., training data) can be created. More specifically, subject matter experts (SMEs) can annotate documents with labels assigned to words or phrases in the documents that are necessary to classify the documents. The SMEs can perform a quality check of the annotations. The annotated documents can be part of the training corpus, which can be stored in a training data repository (e.g., database 120 as shown in FIG. 1).

[0094] In operation 604, a learning model (e.g., learning model 104 as shown in FIG. 1) may be trained using a training corpus stored in the training data repository. A developer or SME may submit a request to train the learning model using the training corpus. The learning model may be trained with the training corpus as described above. Based on the validation of the assigned part-of-speech and dependency tags and NER labels, the learning model may determine whether the accuracy of the learning model's assignment of the assigned part-of-speech and dependency tags and NER labels meets a threshold. The threshold may be pre-programmed or may be provided in the request to train the learning model. If the threshold accuracy of the learning model is not met, a request may be submitted to retrain the learning model with the same set of documents or a different set of documents.

[0095] If the classification accuracy of the learned model meets the threshold, the model approval flow 600 may proceed to operation 606. At operation 606, an SME may manually review the learned model's assignment of assigned part-of-speech and dependency tags and NER labels. In some embodiments, the SME may verify the classification of the same subset of documents as the learned model. Alternatively, the SME may verify the classification of a different subset of documents. The subset of documents may be 10% of the set of documents.

[0096] The SME may verify whether the learning model accurately classified each document of the subset of documents based on the document metadata. Additionally, the SME may also verify whether the metadata is accurate. In response to verifying the assigned part-of-speech and dependency tags and the learning model's assignment of NER labels for the subset of documents, the SME may determine whether an acceptable quality limit (AQL) has been achieved. The AQL may be a value associated with a desired level of accuracy for the learning model. In some embodiments, the AQL may be the same as a threshold. In other embodiments, the AQL may be different from the threshold. If the SME determines that the AQL has not been met, a request may be sent to retrain the learning model. In response to the SME determining that the AQL has been met, the SME may approve the learning model. The learning model may be considered fully trained and ready to classify documents.

[0097] FIG. 7 is a graph showing F1 scores in recognizing entities (e.g., words or phrases) in documents using a trained learning model, according to an exemplary embodiment. Graph 700 shows that a trained learning model (e.g., a fully trained learning model 210 as shown in FIG. 2) can accurately identify 54.13% of the 109 entities with an F1 score between 0 and 50% accuracy. Furthermore, the trained learning model can accurately identify 22.02% of the 109 entities with an F1 score between 51 and 74% accuracy. Furthermore, the trained learning model can accurately identify 23.85% of the 109 entities with an F1 score above 75% accuracy.

[0098] FIG. 8 is a graph 800 illustrating a loss function of a learning model, according to an exemplary embodiment. The y-axis of the graph 800 represents the difference between predicted and actual values. The x-axis represents the number of iterations that a learning model (e.g., learning model 104) has been trained for. Each iteration can be referred to as an epoch. The graph 800 can show 100 epochs spanning the x-axis. The graph 800 shows that the more iterations of training the learning model, the smaller the difference between predicted and actual values. Thus, the error between the predicted and labeled entities was minimized over the 100 epochs. The loss function of the graph 800 shows a smooth decrease in loss, resulting in a loss decrease of about 74% over the course of training.

[0099] FIG. 9 illustrates a fully trained learning model for summarizing documents according to an exemplary embodiment. An in-house summarizer implements the fully trained learning model described herein (e.g., learning model 104 or fully trained learning model 210). NLTK implements a conventional NLP model. The NLTK summarizer can generate a summary 902 of the document 900, and the in-house summarizer can generate a summary 904 of the document 900. As shown in the summary 904, the in-house summarizer broadens the scope of the document 900 in the summary 904 compared to the summary 902. The NLTK summarizer focuses on a limited portion of the document 900 when generating the summary 902. Thus, an in-house summarizer implementing the fully trained learning model described herein provides an advantage over a conventional summarizer implementing a conventional NLP model by providing a broader scope of the document when generating a summary.

[0100] FIG. 10 is a flow chart illustrating a process for training a learning model according to an embodiment. Method 1000 can be performed by processing logic that can include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps are required to carry out the disclosure provided herein. Furthermore, as will be understood by those skilled in the art, some of the steps may be performed simultaneously or in a different order than that shown in FIG. 10.

[0101] The method 1000 is described with reference to Figure 1. However, the method 1000 is not limited to that example embodiment.

[0102] At operation 1002, the learning engine 102 receives a request to train a learning model 104 to classify documents in a domain. The request includes a set of documents and metadata for each of the documents. The metadata may be annotations that label specific words or phrases in each document. The learning model 104 may be an NLP framework that uses CNN and BiLSTM.

[0103] At operation 1004, the learning model 104 generates a set of word embeddings for each document of the set of documents. The word embeddings may be vector representations of each word of the document. The words may be arranged in a vector space. The words may be arranged in proximity to other words that are closer in context and meaning. The word embeddings may include words from each respective document. The word embeddings may be Bloom embeddings.

[0104] At operation 1006, the learning model 104 tokenizes each word embedding in the set of word embeddings into a set of segments. Each segment includes a word from the word embedding. The learning model 104 can use a statistical NER model and general linguistic rules to segment the words from the word embeddings.

[0105] In operation 1008, the learning model 104 is trained to classify each document in the set of documents in a domain by decomposing each of the segments in the set of segments for each document in the set of documents into a set of features. A CNN algorithm can be used to generate the set of features.

[0106] In operation 1010, the learning model 104 assigns a part-of-speech tag to each of the set of segments for each document in the set of documents based on a predefined weight assigned to each feature in the set of features for the corresponding segment. The part-of-speech tag defines whether the word is a noun, verb, adjective, etc.

[0107] In operation 1012, the learning model 104 assigns a dependency tag to each of the segments of the set of segments for each document of the set of documents based on the part-of-speech tag assigned to the corresponding segment and a predefined weight assigned to each feature of the set of corresponding string features. A dependency tag indicates a relationship between a word and other words in a document. For example, the phrase "slothful dog" indicates that the word "slothful" modifies the noun "dog."

[0108] In operation 1014, the learning model 104 assigns a named entity recognition (NER) label from a set of predefined labels corresponding to the domain to each of the segments of the set of segments in each document of the set of documents based on the part-of-speech and dependency tags assigned to the corresponding segment and a predefined weight assigned to each feature of the set of features of the corresponding segment. The learning model 104 can generate an output including a NER tag along with each respective word. The learning model 104 can generate NER tags for words needed to classify the documents.

[0109] At operation 1016, the learning model 104 validates the assigned NER labels by comparing the metadata of each document with the assigned NER label of the respective document. Operations 1008-1016 may be performed recursively to train the learning model. For each training iteration, the weights assigned to the features may be optimized based on a gradient loss function.

[0110] FIG. 11 is a flow chart illustrating a process for classifying documents using a trained learning model, according to one embodiment. Method 1100 can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps are required to perform the disclosure provided herein. Furthermore, as will be appreciated by those skilled in the art, some of the steps may be performed simultaneously or in a different order than that shown in FIG. 11.

[0111] The method 1100 is described with reference to Figure 1. However, the method 1100 is not limited to that example embodiment.

[0112] In operation 1102, the learning engine 102 receives a request to classify a new document using a trained learning model. The trained learning model can be configured to classify documents corresponding to a domain of documents. The trained learning model can implement an NLP framework using CNN and BiLSTM. The learning model 104 may be a fully trained learning model.

[0113] At operation 1104, the learning model 104 generates word embeddings for the document. The word embeddings may include words from the document. The word embeddings may be Bloom embeddings.

[0114] At operation 1106, the learning model 104 tokenizes the word embeddings into a set of segments. Each segment contains a word from the word embedding. The learning model 104 can use a statistical NER model and general linguistic rules to segment the words from the word embeddings.

[0115] In operation 1108, the learning model 104 decomposes each of the set of document segments into a set of features. A CNN algorithm can be used to generate the set of features.

[0116] At operation 1110, the learning model 104 assigns a part-of-speech tag to each of the segments in the set of segments based on a predefined weight assigned to each feature in the set of features for the corresponding segment. The part-of-speech tag defines whether the word is a noun, verb, adjective, etc.

[0117] In operation 1112, the learning model 104 assigns a dependency tag to each of the segments in the set of segments based on the part-of-speech tag assigned to the corresponding segment and a predefined weight assigned to each feature in the set of features of the corresponding string. The dependency tag indicates a relationship between the word and other words in the document.

[0118] In operation 1114, the learning model 104 assigns a named entity recognition (NER) label to each of the segments of the set of segments from a set of predefined labels corresponding to the domain based on the part-of-speech and dependency tags assigned to the corresponding segment and the predefined weights assigned to each feature of the set of features of the corresponding segment. The learning model 104 can generate an output that includes a NER tag along with each respective word. The learning model 104 can generate NER tags for words needed to classify the document.

[0119] In operation 1116, the learning model 104 uses the trained learning model to classify new documents corresponding to the domain based on the assigned NER labels. In one example, the learning model is configured to classify pharmacovigilance documents based on case validity, severity, fatality, and causality. The learning model may be further configured to identify adverse effects in structured product labels (SPL) of FDA-approved drugs for predictiveness and to identify potential off-label product use.

[0120] 12 is a block diagram of example components of a computer system 1200. For example, one or more computer systems 1200 can be used to implement any of the embodiments described herein, as well as combinations and subcombinations thereof. The computer system 1200 can include one or more processors (also referred to as central processing units or CPUs), such as processor 1204. The processor 1204 can be connected to a communication infrastructure or bus 1206.

[0121] The computer system 1200 may also include a user input / output interface 1202 , such as a monitor, keyboard, pointing device, etc., that may communicate with a communications infrastructure 1206 via user input / output devices 1203 .

[0122] One or more of the processors 1204 may be a graphics processing unit (GPU). In one embodiment, a GPU may be a processor that is a dedicated electronic circuit designed to handle mathematically intensive applications. GPUs may have a parallel structure that is efficient for parallel processing of large blocks of data, such as the mathematically intensive data common in computer graphics applications, images, videos, etc.

[0123] The computer system 1200 may also include a main or primary memory 1208, such as a random access memory (RAM). The main memory 1208 may include one or more levels of cache. The main memory 1208 may store control logic (i.e., computer software) and / or data.

[0124] The computer system 1200 may also include one or more secondary storage devices or memories 1210. The secondary memory 1210 may include, for example, a hard disk drive 1212 and / or a removable storage drive 1214.

[0125] The removable storage drive 1214 can interact with a removable storage unit 1218. The removable storage unit 1218 can include a computer usable or readable storage device storing computer software (control logic) and / or data. The removable storage unit 1218 can be a program cartridge and cartridge interface (such as those found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick and USB port, a memory card and associated memory card slot, and / or any other removable storage unit and associated interface. The removable storage drive 1214 can read from and write to the removable storage unit 1218.

[0126] The secondary memory 1210 may include other means, devices, components, means, or other techniques for allowing computer programs and / or other instructions and / or data to be accessed by the computer system 1200. Such means, devices, components, means, or other techniques may include, for example, a removable storage unit 1222 and an interface 1220. Examples of a removable storage unit 1222 and an interface 1220 may be a program cartridge and a cartridge interface (such as those found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick and a USB port, a memory card and associated memory card slot, and / or any other removable storage unit and associated interface.

[0127] Computer system 1200 may further include a communication or network interface 1224. Communication interface 1224 may enable computer system 1200 to communicate and interact with any combination of external devices, external networks, external entities, etc. (individually and collectively referred to by reference number 1228). For example, communication interface 1224 may enable computer system 1200 to communicate with external or remote devices 1228 via communication path 1226, which may be wired and / or wireless (or a combination thereof) and may include any combination of a LAN, a WAN, the Internet, etc. Control logic and / or data may be transmitted to and from computer system 1200 via communication path 1226.

[0128] The computer system 1200 may also be a personal digital assistant (PDA), a desktop workstation, a laptop or notebook computer, a netbook, a tablet, a smartphone, a smart watch or other wearable, a device, part of the Internet of Things, and / or an embedded system, or any combination thereof, to name a few non-limiting examples.

[0129] Computer system 1200 may be a client or server accessing or hosting any application and / or data via any delivery paradigm, including, but not limited to, remote or distributed cloud computing solutions, local or on-premise software ("on-premise" cloud-based solutions), an "as a service" model (e.g., Content as a Service (CaaS), Digital Content as a Service (DCaaS), Software as a Service (SaaS), Management Software as a Service (MSaaS), Platform as a Service (PaaS), Desktop as a Service (DaaS), Framework as a Service (FaaS), Backend as a Service (BaaS), Mobile Backend as a Service (MBaaS), Infrastructure as a Service (IaaS), etc.), and / or a hybrid model including any combination of the foregoing examples or other service or delivery paradigms.

[0130] Any applicable data structures, file formats, and schemas in computer system 1200 may be derived from standards including, but not limited to, Java Script Object Notation (JSON), Extensible Markup Language (XML), Yet Another Markup Language (YAML), Extensible Hypertext Markup Language (XHTML), Wireless Markup Language (WML), MessagePack, XML User Interface Language (XUL), or any other functionally similar representations, alone or in combination. Alternatively, proprietary data structures, formats, or schemas may be used exclusively or in combination with known or open standards.

[0131] In some embodiments, a tangible, non-transitory device or article of manufacture comprising a tangible, non-transitory computer usable or readable medium having stored thereon control logic (software) may be referred to herein as a computer program product or program storage device, including, but not limited to, a tangible article of manufacture embodying computer system 1200, main memory 1208, secondary memory 1210, and removable storage units 1218 and 1222, and any combination thereof. Such control logic, when executed by one or more data processing devices (such as computer system 1200), can cause such data processing devices to operate as described herein.

[0132] The embodiments of the present disclosure are described above with functional building blocks illustrating implementation of specified functions and relationships thereof. The boundaries of these functional building blocks are arbitrarily defined herein for convenience of description. Alternative boundaries may be defined so long as the specified functions and relationships thereof are appropriately performed.

[0133] The foregoing description of the specific embodiments sufficiently reveals the general nature of the present disclosure so that others can easily modify and / or adapt such specific embodiments to various applications by applying knowledge within the skill of those of ordinary skill in the art without departing from the general concept of the present disclosure and without undue experimentation. Such adaptations and modifications are therefore intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. It should be understood that the expressions or terms in this specification are intended to be descriptive rather than limiting, as the terms or terms in this specification would be interpreted by one of ordinary skill in the art in light of the teaching and guidance.

[0134] The breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

[0135] [Table 1]

Claims

1. 1. A method comprising: receiving, by one or more computing devices, a set of documents and metadata for each document in the set of documents, the set of documents corresponding to a domain; generating, by the one or more computing devices, a set of word embeddings for each document of the set of documents, each word embedding including one or more words from a respective document; tokenizing, by the one or more computing devices, each word embedding in the set of word embeddings into a set of segments, each segment comprising a word from the word embedding; decomposing, by the one or more computing devices, each of the segments of the set of segments for each document of the set of documents into a set of features; assigning, by the one or more computing devices, a part-of-speech tag to each of the segments of the set of segments for each document of the set of documents based on a predetermined weight assigned to each feature of the set of features for a corresponding segment; assigning, by the one or more computing devices, a dependency tag to each of the segments of the set of segments for each document of the set of documents based on the part-of-speech tag assigned to the corresponding segment and the predetermined weight assigned to each feature of the set of features for the corresponding segment; assigning, by the one or more computing devices, a named entity recognition (NER) label from a set of predefined labels corresponding to the domain to each of the segments of the set of segments for each document of the set of documents based on the part-of-speech and dependency tags assigned to the corresponding segments and the predetermined weights assigned to each feature of the set of features for the corresponding segments; training, by the one or more computing devices, a learning model to classify each document of the set of documents in the domain by recursively performing verifying, by the one or more computing devices, the assigned NER label by comparing the metadata of each document with the assigned NER label of the respective document; A method comprising:

2. receiving, by the one or more computing devices, a request to classify a new document corresponding to the domain; generating, by the one or more computing devices, new word embeddings that include one or more words of the new document; tokenizing, by the one or more computing devices, the new word embeddings corresponding to the domain into a set of new segments; decomposing, by the one or more computing devices, each of the segments of the new document into a set of new features; assigning, by the one or more computing devices, a new part-of-speech tag to each new segment of the set of new segments of the new document based on a predetermined weight assigned to each feature of the set of new features of the corresponding new segment using the trained learning model; assigning, by the one or more computing devices, a new dependency tag to each of the new segments of the set of new segments of the new document based on the part-of-speech tag assigned to the corresponding new segment and the predetermined weight assigned to each feature of the set of new features of the corresponding new segment using the trained learning model; using the trained learning model, assigning, by the one or more computing devices, a new NER label from the set of predefined labels corresponding to the domain to each of the segments of the set of new segments of the new document based on the part-of-speech and dependency tags assigned to the new corresponding segments and the predetermined weights assigned to each feature of the set of new features of the corresponding new segment; classifying, by the one or more computing devices, the new document corresponding to the domain based on the assigned NER label using the trained learning model; The method of claim 1 further comprising:

3. extracting, by the one or more computing devices, the set of new segments and the assigned NER labels; generating, by the one or more computing devices, a knowledge base; the knowledge base includes a plurality of nodes and a plurality of edges; an edge connecting each node of the plurality of nodes to at least one other node of the plurality of nodes based on a relationship; each node of the plurality of nodes including at least one segment of the set of new segments and a corresponding NER label of the assigned NER labels; The method of claim 2 , further comprising:

4. 4. The method of claim 3, further comprising: identifying, by the one or more computing devices, the relationship between the node of the plurality of nodes and the different node of the plurality of nodes based on a first NER label of the assigned NER labels in a node and a second NER label of the assigned NER labels stored in a different node.

5. 2. The method of claim 1 , further comprising iteratively revising the weights assigned to the features in the set of features for each document in the set of documents after each iteration of training the learning model.

6. 2. The method of claim 1, wherein the learning model uses a statistical model for the part-of-speech tag for each of the segments for each document of the set of documents, the dependency tag for each of the segments of the set of segments for each document of the set of documents, and the NER label from a set of predefined labels corresponding to the domain for each of the segments of the set of segments for each document of the set of documents.

7. The method of claim 1 , wherein the trained learning model implements a supervised learning algorithm.

8. generating, by the one or more computing devices, a feature map for each document of the set of documents based on the set of features for each corresponding segment of each corresponding document; generating, by the one or more computing devices, a reduced feature map for each document of the set of documents by reducing a dimensional characteristic of the feature map for each corresponding segment of each corresponding document; converting, by the one or more computing devices, the reduced feature map for each corresponding segment of each corresponding document into a vector; applying, by the one or more computing devices, the predetermined weight assigned to each feature of the set of features to each corresponding vector; The method of claim 1 further comprising:

9. 1. A system comprising: Memory, A processor coupled to a memory, the processor comprising: receiving a set of documents and metadata for each document in the set of documents, the set of documents corresponding to a domain; generating a set of word embeddings for each document of the set of documents, each word embedding including one or more words from a respective document; tokenizing each word embedding in the set of word embeddings into a set of segments, each segment containing a word from the word embedding; decomposing the segments of the set of segments for each document of the set of documents into a set of features; assigning, for each document of the set of documents, a part-of-speech tag to each of the segments of the set of segments based on a predetermined weight assigned to each feature of the set of features for the corresponding segment; assigning a dependency tag to each of the segments of the set of segments for each document of the set of documents based on the part-of-speech tag assigned to the corresponding segment and the predetermined weight assigned to each feature of the set of features for the corresponding segment; assigning a named entity recognition (NER) label from a set of predefined labels corresponding to the domain to each of the segments of the set of segments in each document of the set of documents based on the part-of-speech and dependency tags assigned to the corresponding segments and the predetermined weights assigned to each feature of the set of features for the corresponding segments. training a learning model to classify each document in the set of documents in the domain by recursively a processor configured to verify the assigned NER label by comparing the metadata of each document with the assigned NER label of the respective document; A system comprising:

10. The processor, receiving a request to classify new documents corresponding to the domain; generating new word embeddings that include one or more words of the new document; tokenizing the new word embeddings corresponding to the domain into a new set of segments; decomposing each of the segments of the new document into a new set of features; assigning a new part-of-speech tag to each new segment of the set of new segments of the new document based on a predetermined weight assigned to each feature of the set of new features of the corresponding new segment using the trained learning model; using the trained learning model, assigning new dependency tags to each of the new segments of the set of new segments of the new document based on the part-of-speech tag assigned to the corresponding new segment and the predetermined weights assigned to each feature of the set of new features of the corresponding new segment; using the trained learning model, assigning a new NER label to each of the segments of the new set of segments of the new document from the set of predefined labels corresponding to the domain based on the part-of-speech and dependency tags assigned to the new corresponding segments and the predetermined weights assigned to each feature of the new set of features of the corresponding new segments; 10. The system of claim 9, further configured to use the trained learning model to classify the new documents corresponding to the domain based on the assigned NER labels.

11. The processor, Extracting the set of new segments and the assigned NER labels; Generate a knowledge base, the knowledge base includes a plurality of nodes and a plurality of edges; an edge connecting each node of the plurality of nodes to at least one other node of the plurality of nodes based on a relationship; The system of claim 10 , wherein each node of the plurality of nodes is further configured to include at least one segment of the set of new segments and a corresponding NER label of the assigned NER labels.

12. 12. The system of claim 11, wherein the processor is further configured to identify the relationship between the node of the plurality of nodes and the different node of the plurality of nodes based on a first NER label of the assigned NER labels in a node and a second NER label of the assigned NER labels stored in a different node.

13. 10. The system of claim 9, wherein the processor is further configured to iteratively modify the weights assigned to the features in the set of features for each document in the set of documents after each iteration of training the learning model.

14. 11. The system of claim 10, wherein the learning model uses a statistical model for the part-of-speech tag for each of the segments for each document of the set of documents, the dependency tag for each of the segments of the set of segments for each document of the set of documents, and the NER label from the set of predefined labels corresponding to the field of the segment of the set of segments for each document of the set of documents.

15. The system of claim 9 , wherein the trained learning model implements a supervised learning algorithm.

16. The processor, generating a feature map for each document of the set of documents based on the set of features for each corresponding segment of each corresponding document; generating a reduced feature map for each document of the set of documents by reducing a dimensional characteristic of the feature map for each corresponding segment of each corresponding document; converting the reduced feature map for each corresponding segment of each corresponding document into a vector; The system of claim 9 , further configured to apply the predetermined weight assigned to each feature of the set of features to each corresponding vector.

17. A non-transitory computer-readable medium having instructions stored thereon, the execution of which by one or more processors of a device causes the one or more processors to: receiving a set of documents and metadata for each document in the set of documents, the set of documents corresponding to a domain; generating, for each document of the set of documents, a set of word embeddings, each word embedding including one or more words from a respective document; tokenizing each word embedding in the set of word embeddings into a set of segments, each segment containing a word from the word embedding; decomposing the segments of the set of segments for each document of the set of documents into a set of features; assigning, for each document of the set of documents, a part-of-speech tag to each of the segments of the set of segments based on a predetermined weight assigned to each feature of the set of features for the corresponding segment; assigning a dependency tag to each of the segments of the set of segments for each document of the set of documents based on the part-of-speech tag assigned to the corresponding segment and the predetermined weight assigned to each feature of the set of features for the corresponding segment; assigning a named entity recognition (NER) label to each of the segments of the set of segments for each document of the set of documents from a set of predefined labels corresponding to the domain based on the part-of-speech and dependency tags assigned to the corresponding segments and the predetermined weights assigned to each feature of the set of features for the corresponding segments; training a learning model to classify each document in the set of documents in the domain by recursively performing verifying the assigned NER label by comparing the metadata of each document with the assigned NER label of the respective document; A non-transitory computer-readable medium for causing a computer to perform operations including:

18. The operation includes: receiving a request to classify a new document corresponding to the domain; generating new word embeddings that include one or more words of the new document; and tokenizing the new word embeddings corresponding to the domain into a set of new segments; decomposing each of the one or more segments of the new document into a set of new features; assigning a new part-of-speech tag to each new segment of the set of new segments of the new document based on a predefined weight assigned to each feature of the set of new features of the corresponding new segment using the trained learning model; assigning a new dependency tag to each of the new segments of the set of new segments of the new document based on the part-of-speech tag assigned to the corresponding new segment and the predetermined weight assigned to each feature of the set of new features of the corresponding new segment using the trained learning model; using the trained learning model, assigning a new NER label to each of the segments of the new set of segments of the new document from the set of predefined labels corresponding to the domain based on the part-of-speech and dependency tags assigned to the new corresponding segments and the predetermined weights assigned to each feature of the new set of features of the corresponding new segments; classifying the new documents corresponding to the domain based on the assigned NER labels using the trained learning model; 20. The non-transitory computer-readable medium of claim 17, further comprising:

19. 20. The non-transitory computer-readable medium of claim 17, wherein the operations further include iteratively revising the weights assigned to the features in the set of features for each document in the set of documents after each iteration of training the learning model.

20. 20. The non-transitory computer-readable medium of claim 17, wherein the learning model uses a statistical model for the part-of-speech tag for each of the one or more segments for each document of the set of documents, the dependency tag for each of the segments of the set of segments for each document of the set of documents, and the NER label from the set of predefined labels corresponding to the field of the segment of the set of segments for each document of the set of documents.

21. 20. The non-transitory computer-readable medium of claim 17, wherein the trained learning model implements a supervised learning algorithm.

22. The operation includes: generating a feature map for each document of the set of documents based on the set of features for each corresponding segment of each corresponding document; generating a reduced feature map for each document of the set of documents by reducing a dimensional characteristic of the feature map for each corresponding segment of each corresponding document; converting the reduced feature map for each corresponding segment of each corresponding document into a vector; applying the predetermined weight assigned to each feature of the set of features to each corresponding vector; 20. The non-transitory computer-readable medium of claim 17, further comprising:

23. 1. A method for classifying documents, comprising: receiving, by one or more computing devices, a request to classify a document corresponding to a domain; generating, by the one or more computing devices, a word embedding comprising one or more words of the document; tokenizing, by the one or more computing devices, the word embeddings corresponding to the domain into a set of segments; decomposing, by the one or more computing devices, each of the segments of the document into a set of new features; assigning, by the one or more computing devices, a part-of-speech tag to each new segment of the set of segments of the new document based on a pre-defined weight assigned to each feature of the set of features of the corresponding segment using a trained learning model; assigning, by the one or more computing devices, a dependency tag to each of the segments of the set of segments of the document using the trained learning model based on the part-of-speech tag assigned to the corresponding segment and the predetermined weight assigned to each feature of the set of features of the corresponding segment; assigning, by the one or more computing devices, to each of the segments of the set of segments of the document, a NER label from a set of predefined labels corresponding to the domain based on the part-of-speech and dependency tags assigned to the corresponding segment and the predetermined weights assigned to each feature of the set of features of the corresponding segment, using the trained learning model; classifying, by the one or more computing devices, the documents corresponding to the domain based on the assigned NER labels using the trained learning model; A method comprising:

24. 1. A method for training a natural language processing (NLP) model to classify pharmacovigilance documents, comprising: receiving, by one or more computing devices, a set of documents and metadata for each document in the set of documents, the set of documents corresponding to pharmacovigilance; generating, by the one or more computing devices, a set of word embeddings for each document of the set of documents, each word embedding including one or more words from a respective document; tokenizing, by the one or more computing devices, each word embedding in the set of word embeddings into a set of segments, each segment comprising a word from the word embedding; decomposing, by the one or more computing devices, each of the segments of the set of segments for each document of the set of documents into a set of features; assigning, by the one or more computing devices, a part-of-speech tag to each of the segments of the set of segments for each document of the set of documents based on a predetermined weight assigned to each feature of the set of features for a corresponding segment; assigning, by the one or more computing devices, a dependency tag to each of the segments of the set of segments for each document of the set of documents based on the part-of-speech tag assigned to the corresponding segment and the predetermined weight assigned to each feature of the set of features for the corresponding segment; assigning, by the one or more computing devices, a named entity recognition (NER) label from a set of predefined labels corresponding to pharmacovigilance to each of the segments of the set of segments for each document of the set of documents based on the part-of-speech and dependency tags assigned to the corresponding segments and the predetermined weights assigned to each feature of the set of features for the corresponding segments; training, by the one or more computing devices, a learning model to classify each document of the set of documents by recursively performing and verifying, by the one or more computing devices, the assigned NER label by comparing the metadata of each document with the assigned NER label of the respective document; In response to fully training the learning model, the learning model is configured to classify pharmacovigilance documents based on case effectiveness, severity, fatality, and causality, identify adverse effects in a structured product label (SPL) of an FDA-approved drug for predictiveness, and identify potential off-label product use. method.

25. 1. A method for classifying pharmacovigilance documents using a natural language processing (NLP) model, comprising: receiving, by one or more computing devices, a request to classify a pharmacovigilance document; generating, by the one or more computing devices, an output comprising named entity recognition (NER) labels for one or more words in the pharmacovigilance document using a learning model configured to implement a combination of a convolutional neural network (CNN) and a bidirectional long-term short-term (BiLSTM) algorithm; classifying, by the one or more computing devices, the pharmacovigilance documents using the NER labels based on case validity, severity, fatality, and causality; A method comprising:

26. identifying, by the one or more computing devices, adverse effects in a Structured Product Label (SPL) of an FDA-approved drug for predictability using the NER label; identifying, by the one or more computing devices, potential off-label product use; 26. The method of claim 25, further comprising:

Images