Classifying documents using domain-specific natural language processing models

Abstract

This document provides embodiments of systems, apparatuses, devices, methods, and / or computer program products for classifying documents using CNN and BiLSTM, and / or combinations and sub-combinations thereof.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 123,336, filed December 9, 2020, the contents of which are incorporated herein by reference in their entirety. Background Technology

[0003] Entities such as companies, government agencies, and educational institutions often receive thousands of documents, including combinations of text, images, charts, tables, and other forms of data / information / knowledge representation. These documents can be of various types, including Microsoft Word, Microsoft Excel documents, png, tiff, jpg, raw, gif, PDF, emails, txt files, handwritten annotations, HTML, XML scanned documents, etc. Manually classifying and prioritizing these documents based on their content can be a tedious and error-prone task. Entities have attempted to automate this process using machine learning algorithms such as Natural Language Processing (NLP). However, traditional NLP models often lack accurate document classification. For example, traditional NLP models cannot accurately classify documents by assigning domain-specific labels to words or phrases.

[0004] Furthermore, manually extracting information or using highly intelligent third-party tools to extract the text content of each PDF with acceptable accuracy (e.g., Optical Character Recognition (OCR)), correctly extracting this data in a machine-readable format, and then piecing it back together is cumbersome, time-consuming, and error-prone. Moreover, traditional methods implementing conventional machine learning models may face numerous obstacles when attempting to extract text from documents, such as optical sharpness, alphanumeric characters, and orientation. Therefore, traditional methods for classifying and prioritizing documents can be cumbersome, expensive, and error-prone. Summary of the Invention

[0005] This document provides embodiments of systems, apparatuses, devices, methods, and / or computer program products for classifying documents using domain-specific NLP models, and / or combinations and sub-combinations thereof.

[0006] In a given embodiment, a method for classifying documents includes the following steps: receiving a set of documents and metadata for each document in the set of documents by one or more computing devices. The set of documents corresponds to a domain. The method further includes generating a set of word embeddings for each document in the set of documents by one or more computing devices. Each word embedding includes one or more words from the corresponding document. The method further includes tokenizing each word embedding in the set of word embeddings into a set of segments by the one or more computing devices. Each segment includes a word from the word embedding. Furthermore, the method includes training a learning model by the one or more computing devices to classify each document in the set of documents of the domain by recursively performing the following steps: decomposing each segment of the set of segments of each document in the set of documents into a set of features by the one or more computing devices; assigning part-of-speech tagging to each segment of the set of segments of each document in the set of documents based on predetermined weights assigned to each feature of the set of features to the corresponding segment by the one or more computing devices; and assigning part-of-speech tagging to each segment of the set of features to the corresponding segment based on predetermined weights assigned to each feature of the set of features to the corresponding segment. The predetermined weights are used to assign dependency tags to each segment of the set of segments in each document in the set of documents; the one or more computing devices assign named entity recognition (NER) tags from a set of predetermined tags corresponding to the domain to each segment of the set of segments in each document in the set of documents, based on the part-of-speech tag and the dependency tag assigned to the corresponding segment, and the predetermined weights assigned to each feature of the set of features assigned to the corresponding segment; and the one or more computing devices verify the assigned NER tags by comparing the metadata of each document with the assigned NER tags of the corresponding document.

[0007] In a given embodiment, the system for classifying documents includes a memory and a processor coupled to the memory. The processor is configured to receive a set of documents and metadata for each document in the set. The set of documents corresponds to a domain. The processor is further configured to generate a set of word embeddings for each document in the set. Each word embedding includes one or more words from the corresponding document. The processor is further configured to tokenize each word embedding in the set of word embeddings into a set of segments. Each segment includes a word from the word embedding. Furthermore, the processor is configured to train a learning model to classify each document in the set of documents of the domain by recursively performing the following steps: decomposing each segment of each document in the set of documents into a set of features; assigning part-of-speech tags to each segment of each document in the set of documents based on predetermined weights assigned to each feature of the corresponding segment; assigning dependency tags to each segment of each document in the set of documents based on the part-of-speech tags assigned to the corresponding segment and the predetermined weights assigned to each feature of the corresponding string; assigning named entity recognition (NER) tags from a set of predetermined tags corresponding to the domain to each segment of each document in the set of documents based on the part-of-speech tags and the dependency tags assigned to the corresponding segment and the predetermined weights assigned to each feature of the corresponding segment; and verifying the assigned NER tags by comparing the metadata of each document with the assigned NER tags of the corresponding document.

[0008] In a given embodiment, a non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors of a device, cause the one or more processors to perform the following operations: receiving a set of documents and metadata for each document in the set of documents. The set of documents corresponds to a domain. The operation further includes generating a set of word embeddings for each document in the set of documents. Each word embedding includes one or more words from the corresponding document. The operation further includes tokenizing each word embedding in the set of word embeddings into a set of segments. Each segment includes words from the word embeddings. Furthermore, the operation includes training a learning model to classify each document in the set of documents of the domain by recursively performing the following steps: decomposing each segment of each segment in the set of documents of the domain into a set of features; assigning part-of-speech tags to each segment of each segment in the set of documents of the domain based on predetermined weights assigned to each feature of the set of features to the corresponding segment; assigning dependency tags to each segment of each segment in the set of documents of the domain based on the part-of-speech tags assigned to the corresponding segment and the predetermined weights assigned to each feature of the set of features to the corresponding string; assigning named entity recognition (NER) tags from a set of predetermined tags corresponding to the domain to each segment of each document of the domain based on the part-of-speech tags and the dependency tags assigned to the corresponding segment and the predetermined weights assigned to each feature of the set of features to the corresponding segment; and verifying the assigned NER tags by comparing the metadata of each document with the assigned NER tags of the corresponding document.

[0009] In a given embodiment, a method for classifying documents includes: receiving a request from one or more computing devices to classify a document corresponding to a domain; generating word embeddings of one or more words of the document from the one or more computing devices; tokenizing the word embeddings corresponding to the domain into a set of segments from the one or more computing devices; decomposing each string in one or more strings of the document into a new set of features from the one or more computing devices; assigning part-of-speech tagging to each new segment in the set of segments of the new document using a trained learning model, based on predetermined weights assigned to each feature in the set of features corresponding to the segment; and using the trained learning model to... The trained learning model assigns dependency tags to each segment of the document's set of segments based on the part-of-speech tag assigned to the corresponding segment and the predetermined weights assigned to each feature in the set of features assigned to the corresponding segment; the one or more computing devices use the trained learning model to assign NER tags from a set of predetermined tags corresponding to the domain to each segment of the document's set of segments based on the part-of-speech tag and the dependency tag assigned to the corresponding segment, and the predetermined weights assigned to each feature in the set of features assigned to the corresponding segment; and the one or more computing devices use the trained learning model to classify the document corresponding to the domain based on the assigned NER tags.

[0010] In a given embodiment, a method for training an NLP model includes the following steps: receiving a set of documents and metadata for each document in the set of documents by one or more computing devices. The set of documents corresponds to pharmacovigilance. The method further includes generating a set of word embeddings for each document in the set of documents by one or more computing devices. Each word embedding includes one or more words from the corresponding document. The method further includes tokenizing each word embedding in the set of word embeddings into a set of segments by the one or more computing devices. Each segment includes a word from the word embedding. Furthermore, the method includes training a learning model by the one or more computing devices to classify each document in the set of documents by recursively performing the following steps: decomposing each segment of the set of segments of each document in the set of documents into a set of features by the one or more computing devices; assigning part-of-speech tagging to each segment of the set of segments of each document in the set of documents based on predetermined weights assigned to each feature of the set of features to the corresponding segment by the one or more computing devices; and assigning part-of-speech tagging to each segment of the set of segments of each document in the set of documents based on the part-of-speech tagging assigned to the corresponding segment and each feature of the set of features assigned to the corresponding string. The system assigns predetermined weights to dependency tags and assigns them to each segment of the set of segments in each document within the set of documents. The one or more computing devices assign Named Entity Recognition (NER) tags from a set of predetermined tags corresponding to the domain to each segment of the set of segments in each document within the set of documents, based on the part-of-speech tagging and dependency tagging assigned to the corresponding segment, and the predetermined weights assigned to each feature of the set of features assigned to the corresponding segment. The one or more computing devices verify the assigned NER tags by comparing the metadata of each document with the assigned NER tags of the corresponding document. In response to fully training the learning model, the learning model is configured to classify pharmacovigilance documents based on case effectiveness, severity, mortality, and causality.

[0011] In a given embodiment, a method for classifying pharmacovigilance documents using a natural language processing (NLP) model includes: receiving a request to classify the pharmacovigilance documents by one or more computing devices; generating output by the one or more computing devices using a learning model configured to implement a combination of convolutional neural networks (CNN) and bidirectional long short-term (BiLSTM) algorithms, the output including named entity recognition (NER) tags for one or more words in the pharmacovigilance documents; and classifying the pharmacovigilance documents by the one or more computing devices using the NER tags based on case effectiveness, severity, mortality, and causality. Attached Figure Description

[0012] The accompanying drawings, which are incorporated herein and form part of this specification, illustrate the present disclosure and, together with the specification, further serve to explain the principles of the present disclosure and enable those skilled in the art to make and use the present disclosure.

[0013] Figure 1 This is a block diagram of a system for classifying documents using a domain-specific NLP model, according to an exemplary embodiment.

[0014] Figure 2 This is a block diagram illustrating the process of training a learning model to classify documents according to an exemplary embodiment.

[0015] Figure 3 This is a block diagram illustrating an example of a learning model implementing a CNN algorithm according to an exemplary embodiment.

[0016] Figure 4 Example documentation for training a learning model, based on an example embodiment, is shown.

[0017] Figure 5 This is a block diagram illustrating a model design framework for a learning model according to an exemplary embodiment.

[0018] Figure 6 This is a block diagram of a model approval process according to an exemplary embodiment.

[0019] Figure 7 This is a graph illustrating the F1 score of an entity (e.g., a word or phrase) in a document using a trained learning model according to an exemplary embodiment.

[0020] Figure 8 This is a graph illustrating the loss function of the learning model according to an exemplary embodiment.

[0021] Figure 9 A fully trained learning model of a summary document according to an example embodiment is shown.

[0022] Figure 10 This is a flowchart illustrating a training process according to an exemplary embodiment.

[0023] Figure 11 This is a flowchart illustrating the process of classifying documents using a trained learning model according to an exemplary embodiment.

[0024] Figure 12 This is a block diagram of exemplary components of a device according to an embodiment.

[0025] The first appearance of an element in a drawing is usually indicated by one or more numerals on the leftmost side of the corresponding reference numeral. In the drawings, the same reference numerals may denote the same or similarly functional elements. Detailed Implementation

[0026] This document provides embodiments of systems, apparatuses, devices, methods, and / or computer programs for classifying documents using image analysis, and / or combinations and sub-combinations thereof.

[0027] As mentioned above, traditional methods for classifying and prioritizing documents can be cumbersome, expensive, and error-prone. For example, in the field of pharmacovigilance (PV) operations, companies receive Individual Case Safety Reports (ICSRs) for various medications. An ICSR is a written report of an adverse event experienced by a patient undergoing a specific treatment or taking a specific medication, which may potentially be related to that treatment or medication.

[0028] For an ICSR to be considered “effective,” it must contain information relating to four elements: an identifiable patient, an identifiable reporter, the suspected drug, and the adverse event. If the ICSR is effective, it is determined whether the described adverse event is a “serious” adverse event. An adverse event is considered serious if it meets one of the following criteria: causes death or is life-threatening; requires hospitalization or prolongs an existing hospitalization; causes persistent or significant disability or incapacity; causes a congenital disability; or is otherwise medically important because treatment and / or intervention are required to prevent one of the aforementioned criteria. Furthermore, when conducting clinical trials of a drug or other product, it can be determined whether the side effect indicated in the ICSR form is a serious unintended consequence adverse reaction (SUSAR).

[0029] ICSRs can be tailored to specific situations. Different governing bodies may need to act on cases with corresponding ICSRs. Governing bodies can provide different timelines for different cases. For example, if a case includes serious side effects listed in the ICSR, it can be prioritized so that the company can act on it. Conversely, if a case includes non-serious side effects in the ICSR, it can be given lower priority.

[0030] ICSRs can be provided in various formats, such as Microsoft Word, Microsoft Excel documents, png, tiff, jpg, raw, gif, email, PDF, txt files, handwritten notes, HTML, XML scanned documents, etc. ICSR documents can also be a combination of multiple formats. For example, an ICSR document can be in .doc format; however, it can also include embedded JPEG images. In another example, one part of an ICSR document could be an email message, while another part could be in MS Word or MS Excel format.

[0031] ICSRs can come from a variety of reporters, such as pharmacies, clinicians, or patients. Furthermore, each document may include reported side effects of the drug along with other information about the drug. For example, a company may need to determine whether a document is a valid ICSR report based on its content, the severity of the side effects listed in the ICSR document, and the severity, relevance, and predictability (SRE) of the side effects listed in the ICSR document. Given the sheer volume and variety of report types and formats, classifying reports in this way can prove to be a challenging task. Therefore, traditional methods may not be effective and efficient in classifying ICSR reports.

[0032] For example, a standard approach might involve subject matter experts (SMEs) manually reviewing each ICSR document and making a determination. Individuals could manually extract relevant information from ICSR documents and input that information into a database, which would then be reviewed by medical professionals to categorize the ICSR documents. However, companies can receive thousands of ICSR documents in a short period. Given the sheer volume of ICSR documents a company might receive, manually reviewing them can be an arduous task. Furthermore, many ICSR documents may be irrelevant, as they may not be valid documents, may not indicate serious effects, or may not indicate serious, relevant, or anticipated effects. This can create a large backlog and delays when processing relevant and important ICSR documents.

[0033] Conventional approaches may also include the use of machine learning algorithms, which require converting documents to text before processing (e.g., via optical character recognition (OCR)). However, given the complexity of OCR and the creation of standardized templates, traditional machine learning algorithms require significant time, human, and financial resources to train, implement, and update them. Thus, these machine learning algorithms are inefficient in operation and costly to train and implement.

[0034] In a given embodiment, the server may receive a request to train a learning model to classify documents and identify domain-specific entities within the documents. For example, the learning model may identify entities within a document to automatically summarize the documents. The content of the document may include one or more strings. Furthermore, the document may include corresponding metadata. Metadata may be annotations that mark one or more strings in the document. The annotations may be domain-specific.

[0035] The server can train a learning model to classify domain-specific documents by generating word embeddings for each document. The server can tokenize each word embedding into segments, including one or more words from each word embedding. The server can train the learning model by recursively performing the following steps: decomposing each segment of each document into a set of features; assigning part-of-speech tags to one or more words corresponding to each corresponding segment of each corresponding document based on predetermined weights assigned to each feature in the set of features assigned to the corresponding segment; and assigning dependency tags to one or more words corresponding to each corresponding segment of each corresponding document based on the part-of-speech tags assigned to the corresponding one or more words and predetermined weights assigned to each feature in the set of features assigned to the corresponding segment. Training the learning model may further include recursively assigning Name Entity Relation (NER) tags from a set of predetermined labels corresponding to the domain to one or more words corresponding to each corresponding segment of each corresponding document based on the part-of-speech tags and dependency tags assigned to the corresponding one or more words and predetermined weights assigned to each feature in the set of features assigned to the corresponding segment; and validating the assigned tags by comparing the metadata of each document with the assigned tags of the corresponding document.

[0036] The server can receive requests to classify domain-specific documents using a trained learning model. The server can tokenize documents into segments, each containing one or more words from one or more strings within the document. The server can decompose each segment of the document into a set of features. The server can assign part-of-speech tags to one or more words corresponding to each segment of the document based on predetermined weights for each feature in the set of features assigned to the corresponding segment. The server can assign dependency tags to one or more words corresponding to each segment based on the part-of-speech tags assigned to the corresponding one or more words and predetermined weights for each feature in the set of features assigned to the corresponding segment. Furthermore, the server can assign Name Entity Relationship (NER) tags from a predetermined set of tags corresponding to the domain to one or more words corresponding to the corresponding one or more words, based on the part-of-speech tags and dependency tags assigned to the corresponding one or more words, and predetermined weights for each feature in the set of features assigned to the corresponding segment. The server can classify the document based on each NER tag assigned to a word in the document.

[0037] The above configuration allows for the processing and classification of multiple document formats and languages ​​without requiring transcription and data retrieval from source documents. It reduces the data input required for case processing and enables inference analysis and search signal management. Therefore, this configuration bypasses text processing, including but not limited to transcription and translation, by leveraging domain-specific NLP that implements a convolutional neural network (CNN) combined with a bidirectional long short-term memory (BiLSTM) model. This approach improves the speed of training models to understand domain concepts within the PV domain. Furthermore, this configuration minimizes the training and maintenance workload of traditional NLP models.

[0038] Furthermore, the above configuration allows the use of domain-specific NLP models to tag one or more strings in a document, resulting in more accurate document classification. For example, the NLP model can be PV-specific. Therefore, the NLP model can be successfully used across PV systems.

[0039] Figure 1 This is a block diagram of a system for classifying documents using a domain-specific NLP model. The system may include a server 100, client devices 110, and a database 120. The devices in the system can be connected via a network. For example, the devices can be connected via wired connections, wireless connections, or a combination of wired and wireless connections. In an example embodiment, one or more portions of the network may be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless local area network (WLAN), wide area network (WAN), wireless wide area network (WWAN), metropolitan area network (MAN), a portion of the Internet, a portion of the public switched telephone network (PSTN), a cellular telephone network, a wireless network, a WiFi network, a WiMax network, any other type of network, or a combination of two or more such networks. Optionally, server 100, client devices 110, and database 120 may reside on a single physical machine or virtual machine.

[0040] In some embodiments, server 100 and database 120 may reside in a cloud computing environment. In other embodiments, server 100 may reside in a cloud computing environment, while database 120 resides outside of the cloud computing environment. Furthermore, in other embodiments, server 100 may reside outside of the cloud computing environment, while database 120 resides in the cloud computing environment.

[0041] Client device 110 can be a device operated by an individual (e.g., a programmer, user, etc.) associated with the administrator of server 100. Client device 110 can include training application 112 and classification application 114. A cloud computing environment can also host training application 112 and classification application 114. Alternatively, one or both of training application 112 and classification application 114 can be installed on client device 110.

[0042] Training application 112 and classification application 114 can be executable applications configured to interface with server 100. Training application 112 can send requests to server 100 to train a learning model, thereby classifying documents using image analysis. Classification application 114 can be configured to send requests to server 100 to classify documents using the learned model. Classification application 114 can also be installed on and executed by a third-party user device. In this case, an authorized third party can send requests to classify documents using server 100. Documents can be stored in database 120. Database 120 can be one or more data storage devices configured to store documents of various types and formats.

[0043] Learning engine 102 may include learning model 104. Learning model 104 may implement a natural language processing (NLP) framework configured to recursively implement deep machine learning algorithms, such as convolutional neural networks (CNNs) and BiLSTMs, for document classification and prioritization. Learning model 104 may be a domain-specific learning model configured to classify domain-specific documents. Learning model 104 may assign multiple categories to a given document. Furthermore, learning model 104 may be configured to generalize a given document. Each category will be explained in more detail below. In some embodiments, fewer or additional learning modules may be used to classify documents.

[0044] Figure 2 This is a block diagram illustrating the process of training a learning model to classify documents according to an exemplary embodiment. (Refer to...) Figure 1 describe Figure 2 In a given embodiment, client device 110 may receive a request to train learning model 104 to classify documents corresponding to a domain. Learning model 104 may be an NLP framework configured to implement CNN and Bidirectional Long Short-Term (BiLSTM) algorithms for document classification.

[0045] Training application 112 can build a statistical NER model. The statistical NER model can be used to implement a rule-based recognition system. For example, the statistical NER model can provide domain-specific rules on how to tag strings in a document. Furthermore, the statistical NER model can be a dictionary or ontology used by learning model 104 to recognize words or phrases in a document. The statistical NER model can be specifically associated with a particular domain. For example, a statistical NER model built using MedDRA can include terms or phrases that specifically correspond to concepts in the PV domain. Training application 112 can load the statistical NER model into learning model 104. The statistical NER model can be used in conjunction with standard languages ​​(e.g., English, Spanish, French, etc.).

[0046] The request may include training data 200. Training data 200 may include documents (and concepts) corresponding to the domain. Documents may include text 202 (e.g., one or more strings) and tags 204 assigned to the text 202. Tags 204 may come from a predefined set of tags corresponding to the domain. Furthermore, each tag in tags 204 may be assigned to one or more strings (e.g., words or phrases) of text 202. The tags assigned to one or more strings may define the string. For example, tags 204 may correspond to an entity or field of a specific domain. Thus, a given tag in tags 204 assigned to a given string indicates that the given string corresponds to a given entity or field of a specific domain. Tags 204 may be included in the metadata of each document.

[0047] The training application 112 can send training data 200, labels (e.g., metadata) 204 corresponding to the training data 200, and parameters to the learning engine 102 for training the learning model 104. The learning engine 102 can receive the training data 200 and the labels 204.

[0048] Learning model 104 can generate word embeddings for each document in the training data 200. A word embedding can be a vector representation of the words in a document. The vectors can be an n-dimensional vector space where words sharing common context and semantics are located close to each other. Learning model 104 can use Bloom embeddings for each document in the training data 200. A Bloom embedding is a compact vector representation of the words in a document. Word embeddings or Bloom embeddings can be generated using a statistical NER model.

[0049] Learning model 104 can tokenize word embeddings (or Bloom embeddings) into segments of words, letters, punctuation marks, etc. Tokenization segments each document according to language- and domain-specific rules. Furthermore, learning model 104 can use statistical models to segment each document. For example, if a given document contains the phrase "I live in the USA.", learning application 112 can determine that the first period after "USA" corresponds to the abbreviation "USA," and the second period corresponds to the end of the sentence. Therefore, the tokenization of the phrase can be segmented as follows: [I][live][in][the][USA][.] Each segment can include a single word, a partial word, or more than one word.

[0050] Learning model 104 can implement a CNN algorithm to decompose each segment into a set of features and use the set of features for each corresponding segment to generate a vector (e.g., a one-dimensional vector) corresponding to each segment. Learning model 104 can assign weights to each of the set of features. (Refer to...) Figure 3 A more detailed description of the CNN algorithm.

[0051] Learning model 104 can apply weights to a vector to generate a result vector. The weights can be included in the parameters received from training application 112. Learning model 104 can assign part-of-speech tags to words in the segment corresponding to the vector based on the result vector and a statistical NER model. Part-of-speech tags can indicate whether a word is a noun, verb, adjective, etc. Learning model 104 can consider the context and predict the part of speech of words in the segment. For example, learning model 104 can determine that the word following "the" must be a noun based on English language rules. Learning model 104 can use predefined rules to make inferences about words and phrases in the document and identify relationships between words in the document. Furthermore, learning model 104 can use word embeddings to identify relationships between words. Additionally, learning model 104 can use a statistical NER model that includes a domain-specific dictionary and ontology to understand the vocabulary used in the document.

[0052] Learning model 104 can also assign dependency tags to words in each segment of each corresponding document based on the result vector corresponding to each segment, a statistical NER model, and part-of-speech tagging assigned to words in each segment. Dependency tags can define relationships between more than two words. For example, in the phrase "lazy dog," learning engine 104 can determine that the word "lazy" modifies "dog." This dependency can be represented by tags (e.g., Amod tags). Learning model 104 can use predefined rules to make inferences about words and phrases in a document and identify relationships between words in the document. Furthermore, learning model 104 can use word embeddings to identify relationships between words. Additionally, learning model 104 can use a statistical NER model that includes a domain-specific dictionary and ontology to understand the vocabulary used in the document.

[0053] Learning model 104 can assign NER tags to words in each segment of each document based on the result vector corresponding to each segment, a statistical NER model, and part-of-speech and dependency annotations assigned to the corresponding words in each segment. NER tags can be selected from a predefined set of tags corresponding to a domain. NER tags indicate that the word corresponds to a field or entity within that domain. Learning model 104 can use predefined rules to make inferences about words and phrases in the document and identify relationships between words in the document. Furthermore, learning model 104 can use word embeddings to identify relationships between words. Additionally, learning model 104 can use a statistical NER model that includes a domain-specific dictionary and ontology to understand the vocabulary used in the document.

[0054] The learning model 104 can validate the NER labels assigned to the words of each document based on the corresponding labels 204 for each document. Based on the validation results and gradients 208, the learning model 104 can modify the weights assigned to each feature, tokenize each document to generate a new segment for each document, generate a new vector based on the new segment and new weights, assign part-of-speech tags to the words of the new segment based on the new vectors, assign dependency tags to the words of the new segment based on the part-of-speech tags assigned to the words and the new vectors, assign NER labels to the words of the new segment based on the part-of-speech tags and dependency tags assigned to the words and the new vectors, and validate the NER labels based on the labels 204. The learning model 104 can recursively modify the weights and perform these steps until the learning model 104 assigns NER labels with the desired accuracy. In some embodiments, part-of-speech tags and dependency tags can also be validated.

[0055] Once the learning model 104 assigns NER labels with the desired accuracy, it can become a fully trained learning model 210. The fully trained learning model 210 is shown as different components of the learning model 104. The fully trained learning model 210 illustrates the process of training the learning model 104. However, it should be understood that the learning model 104 can retain the same components in the system even after being fully trained.

[0056] Client device 110 can receive a request to classify a document using a fully trained learning model 210. This request may include the document. Classification application 114 can send the document and parameters to the fully trained learning model 210. The fully trained learning model 210 can generate word embeddings (or Bloom embeddings) for the document.

[0057] The fully trained learning model 210 can tokenize word embeddings (or Bloom embeddings) to generate segments of a document, generate vectors based on the segments and weights included in the parameters, assign part-of-speech tags to the words of the segments based on a statistical NER model and the vectors, assign dependency tags to the words of the segments based on the part-of-speech tags assigned to the words, the statistical NER model, and the vectors, and assign NER tags to the words of the segments based on the part-of-speech tags and dependency tags assigned to the words, the statistical NER model, and the vectors. The fully trained learning model 210 can generate output 212 in response to the assignment of NER tags. Furthermore, the fully trained learning model 210 can classify documents based on NER tags.

[0058] In some embodiments, a fully trained learning model 210 can extract words and phrases and their corresponding NER tags from a document. The fully trained learning model 210 can construct a knowledge base using the words and phrases extracted from the document and their corresponding NER tags, as well as words and phrases extracted from other documents and their corresponding NER tags. The knowledge base can be a graph-based structure, including nodes connected using edges. Nodes can contain the extracted words and phrases and their corresponding NER tags. The fully trained learning model 210 can connect nodes using edges based on identifying relationships between nodes. The fully trained learning model 210 can determine the relationships between nodes storing words or phrases based on the NER tags of the corresponding words or phrases. The knowledge base can be stored in a database 120.

[0059] As a non-limiting example, the system described above for classifying documents using image analysis can be used to classify ICSR documents. ICSR documents may also include literature and clinical reports. As mentioned above, ICSR documents include information about patients, geographic locations, side effects, ICSR quality and compliance characteristics, benefit-risk profiles, product details, study details and consumer complaints, legal concepts, or other medical concepts related to the use of products subject to FDA regulations. Companies in the pharmaceutical space may need to process ICSR documents to determine whether any action is required for a particular product.

[0060] The ICSR workflow can be comprised of three processing blocks: case acceptance, case processing, and case reporting. Upon acceptance, the PV department receives ICSRs globally from various sources in diverse formats and languages. Reports come from different reporters, healthcare professionals and non-healthcare professionals, and via various media such as email, fax, mail, and telephone. Several important assessments are conducted upon case acceptance, which, given the severity of the case, are crucial for routing the case to meet predefined regulatory guidelines.

[0061] Compliance with regulatory authorities is determined based on the ability to report to the relevant national regulatory authorities within their respective prescribed timelines. Therefore, prioritization should be precise to limit the spread of workload on less urgent reports. Assessments used for prioritization may include the following key characteristics: case effectiveness (effective or ineffective), case severity (serious or non-serious), relevance (related to or unrelated to the suspected product), and SRE (labeled or unlabeled) of the side effect. Case effectiveness can indicate whether the ICSR document is effective. Case severity can indicate whether the side effect listed in the ICSR document is serious or non-serious. SRE can indicate whether the side effect is serious, relevant, and anticipated (e.g., labeled on the product) action.

[0062] If the side effects listed in a valid ICSR document are serious and unexpected, the company may need to take action on the specific product. As a result, a learning model 104 can be trained to classify case validity, severity, mortality, and causality for a given ICSR document. The learning model 104 can also be trained to identify side effects in the structured product label (SPL) of FDA-approved prospective drugs and to identify potential off-label product use. Furthermore, the learning model 104 can be trained to identify entities within the document. The learning model 104 can be used to generate a summary of the document based on the identified entities. The learning model 104 can be trained to understand the context of the document, thereby generating an accurate summary of the document.

[0063] For example, client device 110 can receive a request to train learning model 104 to classify ICSR documents corresponding to the PV domain. Learning model 104 can be an NLP framework configured to implement CNN and BiLSTM algorithms to classify documents.

[0064] As a non-limiting example, learning model 104 can implement spaCy, spaCy (v2.0), or MedSpaCy. SpaCy (v2.0) is an open-source software library for advanced NLP that utilizes convolutional neural network (CNN) models with residual connections and layer normalized maxout nonlinearities. SpaCy offers better efficiency than standard BiLSTM solutions for annotation, analysis, named entity recognition, and deep learning ensembles. Furthermore, spaCy features GloVe (global vector) support in English models. For GloVe.840B.300d Common Crawl (840B tokens, 2.2M vocab, cased, 300d vectors), the maximum vector size is 2.2 million. A Bloom embedding strategy, internally implemented using sub-word features, is used to support efficient processing of a large MedDRA vocabulary.

[0065] Training application 112 can build a statistical NER model. For example, training application 112 can embed 81,900 MedDRA entities to create a statistical NER model. Training application 112 can also use the Unified Medical Language System (UMLS) to build a statistical NER model. The statistical NER model can be incorporated into the spaCyEntityRuler search pattern attribute. The statistical NER model can be a dictionary or ontology used by learning model 104 to identify words or phrases in ICSR documents. Training application 112 can load the statistical NER model into learning model 104. The statistical NER model can be used in conjunction with standard languages ​​(e.g., English, Spanish, French, etc.).

[0066] The request may include training data 200. Training data 200 may include ICSR documents. A document may include text 202 (e.g., one or more strings) and tags 204 assigned to the text 202. Tags 204 may come from a predefined set of tags corresponding to a PV domain. Furthermore, each tag in tags 204 may be assigned to one or more strings (e.g., words or phrases) in the text 202. The tags assigned to one or more strings may define the string. For example, tags 204 may correspond to an entity or field of a PV domain. Thus, in an embodiment, a given tag in tags 204 assigned to a given string indicates that the given string corresponds to a given entity or field of a PV domain. Tags 204 may be included in the metadata of each document.

[0067] As an example, label 204 can include the following entities:

[0068]

[0069] Label 204 can correspond to 109 PV entities. The F1 score represents the level of accuracy with which the fully trained learning model 210 can identify a given entity. The F1 score will be described in more detail below.

[0070] Training application 112 can send training data 200, labels (e.g., metadata) 204 corresponding to the training data 200, and parameters to learning engine 102 for training learning model 104. Learning engine 102 can receive training data 200 and labels 204. In a non-limiting example implementation, training data 200 includes 20,000 machine-readable ICSRs. These ICSRs were received by Celgene (Celgene Corporation) drug safety over a two-year period (January 2015–December 2016). Further details regarding the characterization and sampling of ICSRs have been previously disclosed (Abatemarco et al., 2018; Mockute et al., 2019). For hyperparameters, best practice values ​​are available and used by spaCy.

[0071] Learning model 104 can generate word embeddings (or Bloom embeddings) for each document in the training data 200. Learning model 104 can tokenize word embeddings (or Bloom embeddings) for each document in the training data 200. More specifically, learning model 104 can tokenize each document into segments of words, letters, punctuation, etc. Tokenization segments each document based on language-specific and domain-specific rules. Furthermore, learning model 104 can use a statistical model to segment each document.

[0072] The learning model 104 can implement the CNN algorithm to decompose each segment into a set of features and use that set of features for each corresponding segment to generate a vector (e.g., a one-dimensional vector) corresponding to each segment.

[0073] The learning model 104 can apply the weights assigned to each feature in a set of features to the vector to generate a result vector. The weights can be included in the parameters received from the training application 112. The learning model 104 can then assign part-of-speech tags to words in the segments corresponding to the vectors based on the result vectors and a statistical NER model.

[0074] Learning model 104 can also assign dependency tags to words in each segment of each corresponding document based on the result vector corresponding to each segment, the statistical NER model, and the part-of-speech tags assigned to words in each segment.

[0075] The learning model 104 can assign NER tags to words in each segment of each corresponding document based on the result vector corresponding to each segment, the statistical NER model, and the part-of-speech and dependency annotations assigned to the corresponding words in each segment. NER tags can be selected from a predefined set of tags corresponding to the domain. NER tags indicate which field or entity a word corresponds to within that domain. For example, NER tags can be assigned to words or phrases crucial for PV event detection to accurately classify documents.

[0076] The learning model 104 can validate the NER labels assigned to the words in each document based on the corresponding labels 204 for each document. Based on the validation results and gradients 208, the learning model 104 can modify the weights assigned to each feature, tokenize word (or Bloom) embeddings for each document to generate new segments for each document, generate new vectors based on the new segments and new weights, assign part-of-speech tags to the words in the new segments based on the new vectors, assign dependency tags to the words in the new segments based on the part-of-speech tags assigned to the words and the new vectors, assign NER labels to the words in the new segments based on the part-of-speech tags assigned to the words and the dependency tags assigned to the words and the new vectors, and validate the NER labels based on labels 204. The learning model 104 can recursively modify the weights and perform these steps until the learning model 104 assigns NER labels with the desired accuracy. Gradients 208 can be gradient loss functions configured to optimize the weights used to train the learning model 104.

[0077] In the non-restricted example implementation, learning model 104 is trained on ICSR documents with 20,000 annotations, and a 75% threshold F1 score is used to determine whether learning model 104 is fully trained. The F1 score is generated using the following equation:

[0078] Precision = True / (True + False Precision)

[0079] Recall = True / (True + False Negative)

[0080] F1=2×(Precision×Recall) / (Precision+Recall)

[0081] Once the learning model 104 assigns NER labels with the desired accuracy, the learning model 104 can be a fully trained learning model 210. The client device 110 can receive a request to classify an ICSR document using the fully trained learning model 210. This request may include the ICSR document. The classification application 114 can send the ICSR document and parameters to the fully trained learning model 210. The fully trained learning model 210 can generate word (Bloom) embeddings for the ICSR document. The fully trained learning model 210 can tokenize the word embeddings to generate segments of the document, generate vectors based on the segments and weights included in the parameters, assign part-of-speech tags to the words of the segments based on a statistical NER model and the vectors, assign dependency tags to the words of the segments based on the part-of-speech tags assigned to the words, the statistical NER model, and the vectors, and assign NER labels to the words of the segments based on the part-of-speech tags assigned to the words, the statistical NER model, and the vectors. The fully trained learning model 210 can generate output 212 in response to the assignment of NER labels.

[0082] Output 212 may include NER tags for keywords or phrases necessary for classifying ICSR documents. For example, output 212 may include NER tags for the word “admitted”, such as “ReportersSeriousHospitalization”, and the word “Melanoma” may include NER tags such as AE10053571.

[0083] The fully trained learning model 210 can classify ICSR documents based on NER tags. For example, the fully trained learning model 210 can classify ICSR documents based on case effectiveness, severity, mortality, and causality, identify side effects in the structured product label (SPL) of FDA-approved prospective drugs, and identify potential off-label product use.

[0084] The fully trained learning model 210 can generate outputs that include uncovered entities, as well as any detected relationships between product indications and associated AEs. As an example, Streamlit can be used to generate a web interface to render the output. The fully trained learning model 210 can result in the output being displayed on a client device 110.

[0085] In some embodiments, a fully trained learning model 210 can extract words and phrases and their corresponding NER tags from ICSR documents. The fully trained learning model 210 can use the words and phrases extracted from ICSR documents and their respective NER tags, as well as words and phrases extracted from other documents (ICSR or others) and their respective NER tags, to construct a knowledge base. The knowledge base may relate to drugs, drug types, diseases, patient demographics, etc.

[0086] The knowledge base can be a graph-based structure, including nodes connected using edges. Nodes can contain extracted words and phrases along with their respective NER labels. A fully trained learning model 210 can connect nodes using edges based on identifying relationships between them. The fully trained learning model 210 can determine the relationships between nodes storing words or phrases based on the NER labels of the corresponding words or phrases. The knowledge base can be stored in a database 120.

[0087] Knowledge bases can be used across different industries to identify various aspects of drugs, drug types, diseases, patient demographics, and more. Industries can include technology, management, and healthcare. For example, regulatory industries such as the FDA can use knowledge bases to identify the various effects of drugs. In another example, drug manufacturers can use knowledge bases to identify the components of drugs that cause side effects. In yet another example, medical professionals or researchers can use knowledge bases to identify treatments for drugs or diseases.

[0088] Figure 3 This is a block diagram illustrating an example of a learning model implementing a CNN algorithm according to an exemplary embodiment. (Refer to...) Figure 1-2 describe Figure 3 .

[0089] As described above, learning model 104 can implement supervised deep learning algorithms, such as CNNs, to classify documents based on their corresponding image files. CNN algorithm 300 can be used to generate vectors representing each tokenized segment of a document, such that these vectors can be used to assign part-of-speech tags, dependency tags, and NER labels.

[0090] The CNN algorithm 300 can be trained in two stages: a forward stage and a backward stage. The forward stage includes convolutional layers 304 and 308, pooling layers 306 and 310, and fully connected layers 312 and 314. Convolutional layers 304 and 308 apply filters to the input segment 302 to generate feature maps. Pooling layers 306 and 310 generate simplified feature maps. Then, fully connected layers 312 and 314 classify the features of the image using weights and biases to generate output 316. Output 316 can be the assigned part-of-speech tag, dependency tag, and NER label. The values ​​of the filters, weights, and biases can be parameters received by the CNN algorithm 300 (e.g., from the client device 110).

[0091] In the backpropagation phase, CNN algorithm 300 can use backpropagation to determine whether the CNN algorithm can correctly assign part-of-speech tags, dependency tags, and NER labels. The CNN algorithm can use gradient descent to update the values ​​of filters, weights, and biases, and re-execute the forward phase on input segment 302.

[0092] As an example, a tokenized segment of word embeddings from a document (including input segment 302) can be used to train a CNN algorithm 300. Input segment 302 may correspond to one or more words in the document. As a non-limiting instance, input segment 302 may have a size of 160×160 pixels, making it a 160×160-dimensional array. The 160×160-dimensional array can be flattened into a 25600-dimensional vector. Each component of the vector may include a value between 0 and 1. This value can describe the intensity of the pixel. Input segment 302 may include a unique JPEG "fingerprint" in grayscale format.

[0093] In convolutional layer 304, CNN algorithm 300 can perform feature extraction on input segment 302. Features may include portions of input segment 302. For example, features may be different edges or shapes of input segment 302. CNN algorithm can extract different types of features to generate different types of feature maps. For example, CNN algorithm 300 can apply arrays of numbers (e.g., kernels) to different portions of input segment 302. Kernels can also be referred to as filters. As described above, different types of filters can be applied to input segment 302 to generate different feature maps. For example, the filter used to identify shapes in input segment 302 may be different from the filter used for edge detection. Therefore, different kernels can be applied to identify shapes in input segment 302 compared to edge detection. Each kernel may include a different array of numbers. The values ​​of filters or kernels can be randomly assigned and optimized over time (e.g., using a gradient descent algorithm). Kernels can be applied as sliding windows across different portions of input segment 302. Kernels can be added to a given portion of input segment 302 to generate an output value. The output value can be included in the feature map. The feature map may include the output values ​​from different kernels applied to each portion of input segment 302. The generated feature map can be a two-dimensional array.

[0094] In pooling layer 306, CNN algorithm 300 can reduce the dimensionality of each feature map generated in convolutional layer 304. Specifically, CNN algorithm 300 can extract a portion of a given feature map and discard the rest. Pooling the image preserves important features. For example, a feature map can include active and inactive regions. Active regions can include detected features, while inactive regions can indicate that parts of a segment do not contain features. Pooling can remove inactive regions. This reduces the image size. CNN algorithm 300 can perform these operations using max pooling or average pooling in the pooling layer. Max pooling preserves higher values ​​for parts of the feature map while discarding the remaining values. Average pooling preserves the average value for different parts of the feature map. Therefore, CNN algorithm 300 can generate a simplified feature map for each feature map generated in convolutional layer 304.

[0095] In convolutional layer 308, CNN algorithm 300 can generate additional feature maps based on the simplified feature maps generated in pooling layer 306. Furthermore, in pooling layer 310, CNN algorithm 300 can generate further simplified feature maps based on the feature maps generated in convolutional layer 308. Multiple different convolutional and pooling layers can be added to CNN algorithm 300.

[0096] Convolutional layers 304 and 308 can also apply a Corrected Linear Unit (ReLU) function to input segment 302. The ReLU function is applied to input segment 302 to remove linearity from it. For example, the ReLU function can remove all black elements from input segment 302, leaving only gray and white. This makes the color changes in input segment 302 more abrupt, thus removing linearity from it.

[0097] Convolutional layers 304 and 308, and pooling layers 306 and 310, can be used for feature learning. Feature learning allows the CNN algorithm 300 to identify desired features in the input segment 302 and thus accurately classify the input segment 302. Therefore, by optimizing convolutional layers 304 and 308 and pooling layers 306 and 310, the CNN algorithm 300 can apply the correct filters to the input segment 302 to extract the necessary features required for classifying the input segment 302.

[0098] In fully connected layer 312, CNN algorithm 300 can flatten the simplified feature map generated in pooling layer 310 into a one-dimensional array (or vector). A fully connected layer is a neural network. CNN algorithm 300 can perform a linear transformation on the one-dimensional array in fully connected layer 312. CNN algorithm 300 can perform a linear transformation by applying weights and biases to the one-dimensional array to generate fully connected layer 314. Initially, the weights and biases are randomly initialized and can be optimized over time.

[0099] In the fully connected layer 314, the CNN algorithm 300 can perform non-linear transformations, such as activation layer functions (e.g., softmax or sigmoid), to assign part-of-speech tags, dependency tags, and NER labels to the corresponding words in the input segment 302.

[0100] The CNN algorithm 300 can be used to verify its classification of the input segment 302 using backpropagation. For example... Figure 2 As shown, in training mode, the CNN algorithm 300 can receive a document and its corresponding metadata. The metadata may include tags 204 assigned to different words or phrases in the document. The CNN algorithm 300 can compare the NER tags assigned to the words or phrases in the document with the tags 204 corresponding to that document. If the NER tags assigned to the document by the CNN algorithm 300 match the tags 204 corresponding to the document, the CNN algorithm 300 can determine that the NER tags are correctly assigned. Alternatively, if the classification assigned to the document by the CNN algorithm 300 does not match the classification included in the document's metadata, the CNN algorithm 300 can determine that the NER tags are incorrectly assigned. In some embodiments, part-of-speech tagging and dependency tagging can also be verified.

[0101] Backpropagation can also involve optimizing the input parameters so that part-of-speech tagging and dependency labeling, as well as NER tags, are more accurately assigned to words or phrases in the document. Input parameters can include values ​​for kernels, weights, offsets, etc. Gradient descent (e.g., gradient 208) can be used to optimize these parameters.

[0102] Gradient descent is an iterative process for optimizing CNN algorithm 300. Gradient descent updates the parameters of CNN algorithm 300 and enables the learning model 104 to assign part-of-speech and dependency annotations and NER labels to words or phrases in each document, and to validate the part-of-speech and dependency annotations and NER labels. After each iteration of CNN algorithm 300, the parameters can be further updated (or optimized). Each iteration of CNN algorithm 300 can be referred to as an epoch. CNN algorithm 300 can be executed for a predetermined number of epochs that are considered fully trained. The predetermined number of epochs can be selected based on the expected accuracy of assigning part-of-speech and dependency annotations and NER labels.

[0103] As a non-limiting example, the CNN algorithm 300 can be implemented by the learning model 104 to classify ICSR documents. The learning model 104 can use the CNN algorithm 300 to identify words or phrases in the ICSR documents, enabling the learning model 104 to determine case effectiveness, severity, mortality, and causality, identify side effects present in the structured product label (SPL) of FDA-approved prospective drugs, and identify potential off-label product use.

[0104] As described above, the CNN algorithm 300 can be performed for a predetermined number of periods that are considered fully trained. Users (e.g., developers) can determine whether the CNN algorithm is fully trained based on its F-1 score and area under the curve (AUC) score. The F-1 score and AUC curve can indicate the accuracy of assigning part-of-speech tagging and dependency tagging, as well as NER tags, to words or phrases in the corresponding document.

[0105] Figure 4 An example document for training a learning model is shown according to an example embodiment. Document 400 may be an ICSR document and may include keywords or phrases 402 required for classifying ICSR documents. Document 400 can be used to train a learning model (e.g., learning model 104, such as...). Figure 1 (As shown) to categorize ICSR documents. Document 400 may include annotations 404 (e.g., tag 204, such as...). Figure 2 (As shown). Note 404 can be a tag for the keywords or phrases 402 required to classify ICSR documents. For example, the word "Name" could be assigned the tag "ReporterTypeHCP". Note 204 can be used to verify the results of assigning NER tags to documents 400.

[0106] Figure 5 This is a block diagram illustrating a model design framework for a learning model according to an exemplary embodiment. The model design framework 500 can be implemented by a learning model (e.g., learning model 104 or a fully trained learning model 210). The model design framework 500 can be used to train the learning model using training data 200. A dictionary or ontology can be loaded onto the model design framework 500. For example, the dictionary or ontology could be MedRNA or UMLS.

[0107] As described above, once the learning model is trained, the model design framework 500 can construct a vocabulary using a custom statistical NER model and existing language statistical models. The model design framework 500 can receive documents for classification. It can tokenize the documents and implement a CNN algorithm to identify words or phrases in the documents using the vocabulary. The model design framework 500 can assign part-of-speech and dependency tags to the words or phrases in the documents and assign NER labels to them. The model design framework 500 can generate output 212, which includes the NER labels for the words or phrases in the documents necessary for classification. The model design framework 500 is able to understand the context of the document when assigning NER labels. For example, even though gastritis is included in the text of output 212, the complete phrase reads "We exclude gastritis...". This indicates that gastritis does not need to be labeled because it is not important in the classification document.

[0108] Figure 6 This is a block diagram of a model approval process according to an exemplary embodiment. Model approval process 600 can be used to determine whether a learning model assigns part-of-speech and dependency tags and NER labels with an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) can be created. More specifically, a Subject Matter Expert (SME) can annotate documents with tags assigned to them, such that the tags are necessary for classifying the documents. The SME can perform quality checks on the annotations. The annotated documents can be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, such as...). Figure 1 As shown in the figure.

[0109] In operation 604, a training corpus stored in the training data repository can be used to train the learning model (e.g., ...). Figure 1(See learning model 104). Developers or SMEs can send a request to train the learning model using a training corpus. The training corpus described above can be used to train the learning model. Based on the validation of the assigned part-of-speech and dependency tags and NER labels, the learning model can determine whether the accuracy of the learning model in assigning the assigned part-of-speech and dependency tags and NER labels meets a threshold. The threshold can be pre-programmed or can be provided in the request to train the learning model. If the accuracy does not meet the threshold of the learning model, a request can be sent to retrain the learning model with a set of the same documents or a different set of documents.

[0110] If the classification accuracy of the learning model meets the threshold, the model approval process 600 can proceed to operation 606. In operation 606, the SME can manually verify the learning model's assignment of part-of-speech and dependency tags, as well as NER labels. In some embodiments, the SME can verify the classification of the same subset of documents as the learning model. Alternatively, the SME can verify the classification of different subsets of documents. A document subset can be 10% of a set of documents.

[0111] The SME can verify whether the learning model accurately classifies each document in a document subset based on the document's metadata. Furthermore, the SME can verify the accuracy of the metadata. In response to verifying the learning model's assignment of part-of-speech and dependency tagging, as well as NER tags, to the assigned document subset, the SME can determine whether an acceptable quality limit (AQL) has been met. The AQL can be a value associated with the expected level of accuracy of the learning model. In some embodiments, the AQL can be the same as a threshold. In other embodiments, the AQL can be different from a threshold. If the SME determines that the AQL has not been met, it can send a request to retrain the learning model. In response to the SME determining that the AQL has been met, the SME can approve the learning model. The learning model can be considered fully trained and ready to classify documents.

[0112] Figure 7 This is a graph illustrating the F1 score of using a trained learning model to identify entities (e.g., words or phrases) in a document according to an exemplary embodiment. Figure 700 indicates the trained learning model (e.g., such as...). Figure 2 The fully trained learning model 210 shown can accurately identify 54.13% of 109 entities with an F1 score of 0-50%. Furthermore, the trained learning model can accurately identify 22.02% of 109 entities with an F1 score of 51-74%. Additionally, the trained learning model can accurately identify 23.85% of 109 entities with an F1 score greater than 75%.

[0113] Figure 8Figure 800 illustrates the loss function of a learning model according to an exemplary embodiment. The y-axis of Figure 800 represents the difference between the predicted and actual values; the x-axis represents the number of iterations the learning model (e.g., learning model 104) has been trained on. Each iteration can be referred to as an epoch. Figure 800 may show 100 epochs on the x-axis. Figure 800 shows that as the number of iterations of training the model increases, the difference between the predicted and actual values ​​decreases. Thus, the error between the predicted entity and the labeled entity is minimized over 100 epochs. The loss function in Figure 800 illustrates the smooth reduction in loss, resulting in a loss reduction of approximately 74% during training.

[0114] Figure 9 A fully trained learning model for summarizing documents according to an example embodiment is shown. An internal aggregator implements the fully trained learning model described herein (e.g., learning model 104 or fully trained learning model 210). NLTK implements a conventional NLP model. The NLTK aggregator can generate a summary 902 of document 900, and the internal aggregator can generate a summary 904 of document 900. As shown in summary 904, the internal aggregator provides a broader coverage of document 900 in summary 904 compared to summary 902. The NLTK aggregator focuses on a limited portion of document 900 when generating summary 902. Thus, the internal aggregator implementing the fully trained learning model described herein provides an advantage by providing a broader coverage of the document when generating the summary, compared to a conventional aggregator implementing a conventional NLP model.

[0115] Figure 10 This is a flowchart illustrating a process for training a learning model according to an embodiment. Method 1000 may be performed by processing logic that may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps need to perform the disclosure provided herein. Furthermore, as those skilled in the art will understand, steps may be performed simultaneously or in combination with... Figure 10 The different sequences shown are used to perform some steps.

[0116] Method 1000 will refer to Figure 1 The following description is provided. However, method 1000 is not limited to this example embodiment.

[0117] In operation 1002, learning engine 102 receives a request to train learning model 104 to classify documents in a domain. This request includes a set of documents and metadata for each document. The metadata may be annotations that mark certain words or phrases in each document. Learning model 104 may be an NLP framework using CNN and BiLSTM.

[0118] In operation 1004, learning model 104 generates a set of word embeddings for each document in a set of documents. A word embedding can be a vector representation of a word in each document. Words can reside in a vector space. Words can be located near other words that are more contextually and semantically close. Word embeddings can include words from each corresponding document. Word embeddings can be Bloom embeddings.

[0119] In operation 1006, the learning model 104 tokenizes each word embedding in a set of word embeddings into a set of segments. Each segment includes a word from the word embedding. The learning model 104 can use a statistical NER model and general language rules to segment words from the word embeddings.

[0120] In operation 1008, model 104 is trained to classify each document in a set of documents in a domain by decomposing each segment of a set of segments in each document into a set of features. A CNN algorithm can be used to generate this set of features.

[0121] In operation 1010, learning model 104 assigns part-of-speech tags to each segment in a set of segments for each document in a set of documents, based on predetermined weights for each feature in a set of features assigned to the corresponding segment. Part-of-speech tags define whether a word is a noun, verb, adjective, etc.

[0122] In operation 1012, learning model 104 assigns dependency tags to each segment of a set of segments in each document within a set of documents, based on part-of-speech tags assigned to the corresponding segments and predetermined weights for each feature in a set of features assigned to the corresponding strings. Dependency tags indicate the relationship between a word and other words in the document. For example, the phrase "lazy dog" indicates that the word "lazy" modifies the noun "dog".

[0123] In operation 1014, learning model 104 assigns Named Entity Recognition (NER) tags from a set of predefined tags corresponding to a domain to each segment of a set of documents, based on part-of-speech and dependency tags assigned to the corresponding segment, and predetermined weights for each feature in a set of features assigned to the corresponding segment. Learning model 104 can generate output including NER tags next to each corresponding word. Learning model 104 can generate NER tags for words necessary for classifying documents.

[0124] In operation 1016, the learning model 104 validates the assigned NER labels by comparing the metadata of each document with the assigned NER labels for the corresponding document. Operations 1008-1016 can be performed recursively to train the learning model. For each training iteration, the weights assigned to the features can be optimized based on the gradient loss function.

[0125] Figure 11 This is a flowchart illustrating a process for classifying documents using a trained learning model according to an embodiment. Method 1100 can be performed by processing logic that may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps need to perform the disclosure provided herein. Furthermore, as those skilled in the art will understand, steps can be performed simultaneously or in combination with... Figure 11 The different sequences shown are used to perform some steps.

[0126] Reference Figure 1 Method 1100 is described. However, method 1100 is not limited to this example embodiment.

[0127] In operation 1102, learning engine 102 receives a request to classify a new document using a trained learning model. The trained learning model can be configured to classify documents corresponding to domains of the document. The trained learning model can implement an NLP framework using CNN and BiLSTM. Learning model 104 can be a fully trained learning model.

[0128] In operation 1104, learning model 104 generates word embeddings for the document. Word embeddings can include words from the document. Word embeddings can be Bloom embeddings.

[0129] In operation 1106, the learning model 104 tokenizes the word embeddings into a set of segments. Each segment includes a word from the word embedding. The learning model 104 can use a statistical NER model and general language rules to segment words from the word embeddings.

[0130] In operation 1108, learning model 104 decomposes each segment of a set of segments in a document into a set of features. A CNN algorithm can be used to generate this set of features.

[0131] In operation 1110, learning model 104 assigns part-of-speech tags to each segment in a set of features based on predetermined weights assigned to each feature in a set of features to the corresponding segment. Part-of-speech tags define whether a word is a noun, verb, adjective, etc.

[0132] In operation 1112, learning model 104 assigns dependency tags to each segment in a set of segments based on part-of-speech tags assigned to the corresponding segments and predetermined weights for each feature in a set of features assigned to the corresponding string. Dependency tags indicate the relationship between a word and other words in the document.

[0133] In operation 1114, learning model 104 assigns Named Entity Recognition (NER) tags from a set of predetermined labels corresponding to a domain to each segment based on part-of-speech and dependency tags assigned to the corresponding segment, and predetermined weights for each feature in a set of features assigned to the corresponding segment. Learning model 104 can generate output including NER tags next to each corresponding word. Learning model 104 can generate NER tags for words necessary for classifying the document.

[0134] In operation 1116, learning model 104 uses a trained learning model to classify new documents corresponding to the domain based on assigned NER labels. In one example, the learning model is configured to classify pharmacovigilance documents based on case effectiveness, severity, mortality, and causality. The learning model can also be configured to identify side effects in the structured product labels (SPLs) of FDA-approved prospective drugs and identify potential off-label product use.

[0135] Figure 12 This is a block diagram of exemplary components of computer system 1200. For example, one or more computer systems 1200 may be used to implement any of the embodiments discussed herein, as well as combinations and sub-combinations thereof. Computer system 1200 may include one or more processors (also referred to as a central processing unit or CPU), such as processor 1204. Processor 1204 may be connected to communication infrastructure or bus 1206.

[0136] The computer system 1200 may also include a user input / output interface 1202, such as a monitor, keyboard, pointing device, etc., which can communicate with the communication infrastructure 1206 through the user input / output device 1203.

[0137] One or more processors 1204 may be graphics processing units (GPUs). In one embodiment, a GPU may be a processor with dedicated electronic circuitry designed to process mathematically intensive applications. The GPU may have a parallel architecture that is efficient for parallel processing of large blocks of data, such as common mathematically intensive data in computer graphics applications, images, videos, etc.

[0138] The computer system 1200 may also include main memory or first-level memory 1208, such as random access memory (RAM). Main memory 1208 may include one or more levels of cache memory. Main memory 1208 may store control logic (i.e., computer software) and / or data therein.

[0139] The computer system 1200 may also include one or more auxiliary storage devices or memories 1210. Auxiliary memory 1210 may include, for example, a hard disk drive 1212 and / or a removable storage drive 1214.

[0140] The removable storage drive 1214 can interact with the removable storage unit 1218. The removable storage unit 1218 may include a computer-usable or readable storage device on which computer software (control logic) and / or data are stored. The removable storage unit 1218 may be a program box and box interface (e.g., found in video game devices), a removable storage chip (e.g., EPROM or PROM) and associated receptacle, a memory stick and USB port, a memory card and associated memory card slot, and / or any other removable storage unit and associated interface. The removable storage drive 1214 can read from and / or write to the removable storage unit 1218.

[0141] Auxiliary storage 1210 may include other means, devices, components, tools, or other methods that allow computer programs and / or other instructions and / or data to be accessed by computer system 1200. Such means, devices, components, tools, or other methods may include, for example, removable storage unit 1222 and interface 1220. Examples of removable storage unit 1222 and interface 1220 may include a program box and box interface (e.g., found in video game devices), a removable storage chip (e.g., EPROM or PROM) and associated socket, a memory stick and USB port, a memory card and associated memory card slot, and / or any other removable storage unit and associated interface.

[0142] Computer system 1200 may also include a communication or network interface 1224. Communication interface 1224 enables computer system 1200 to communicate and interact with any combination of external devices, external networks, external entities, etc. (individually and collectively indicated by reference numeral 1228). For example, communication interface 1224 may allow computer system 1200 to communicate with external or remote devices 1228 via communication path 1226, which may be wired and / or wireless (or a combination thereof) and may include any combination of LAN, WAN, Internet, etc. Control logic and / or data may be sent to and from computer system 1200 via communication path 1226.

[0143] Computer system 1200 may also be any of a personal digital assistant (PDA), desktop workstation, laptop or notebook computer, webbook, writing tablet, smartphone, smart writing tablet or other web, appliance, part of the Internet of Things, and / or embedded system, to name just a few non-limiting examples, or any combination thereof.

[0144] Computer system 1200 may be a client or server that accesses or hosts any application and / or data through any delivery paradigm, including but not limited to remote or distributed cloud computing solutions; on-premises or on-premises software (“on-premises” cloud-based solutions); “models” as a service (e.g., Content as a Service (CaaS), Digital Content as a Service (DCaaS), Software as a Service (SaaS), Managed Software as a Service (MSaaS), Platform as a Service (PaaS), Desktop as a Service (DaaS), Framework as a Service (FAAS), Backend as a Service (BaaS), Mobile Backend as a Service (mbaaS), Infrastructure as a Service (IaaS), etc.); and / or hybrid models that include any combination of the foregoing examples or other services or delivery paradigms.

[0145] Any applicable data structures, file formats, and schemas in the computer system 1200 can be derived from, but are not limited to, the following standards: JavaScript Object Notation (JSON), Extensible Markup Language (XML), Yet Ather Markup Language (YAML), Extensible Hypertext Markup Language (XHTML), Wireless Markup Language (WML), MessagePack, XML User Interface Language (XUL), or any other functionally similar single or combined representation. Alternatively, proprietary data structures, formats, or schemas can be used, either exclusively or in combination with known or open standards.

[0146] In some embodiments, a tangible, non-transitory device or article of art, including a tangible, non-transitory computer-usable or readable medium on which control logic (software) is stored, may also be referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system 1200, main memory 1208, secondary memory 1210 and removable storage units 1218 and 1222, and tangible articles embodying any combination thereof. When executed by one or more data processing devices (e.g., computer system 1200), such control logic can cause such data processing devices to operate as described herein.

[0147] Embodiments of this disclosure have been described above using functional building blocks, which illustrate the implementation of specific functions and their relationships. For ease of description, the boundaries of these functional components are arbitrarily defined herein. Alternative boundaries can be defined as long as the specified functions and their relationships are properly performed.

[0148] The foregoing description of specific embodiments will so fully reveal the general nature of this disclosure that others can readily modify and / or adapt it to various applications by applying knowledge within the art, without requiring excessive experimentation and without departing from the general concept of this disclosure. Therefore, based on the teachings and guidance given herein, such adjustments and modifications are intended to fall within the meaning and scope of equivalents of the disclosed embodiments. It should be understood that the wording or terminology herein is for descriptive purposes and not for limiting purposes, and is intended for those skilled in the art to interpret the terminology or terminology of this specification in accordance with the teachings and guidance.

[0149] The breadth and scope of this disclosure should not be limited by any of the exemplary embodiments described above, but should be defined solely by the appended claims and their equivalents.

[0150] References:

[0151] Abatemarco D, Perera S, Bao SH, Desai S, Assuncao B, Tetarenko N, Danysz K, Mockute R, Widdowson M, Fornarotto N, Beauchamp S, Cicirello S, Mingle E. Training Augmented Intelligent Capabilities for Pharmacovigilance: Applying Deep-learning Approaches to Individual Case Safety Report Processing.PharmaceutMed.2018;32(6):391-401.

[0152] Danysz K, Cicirello S, Mingle E, Assuncao B, Tetarenko N, Mockute R, Abatemarco D, Widdowson M, Desai S. Artificial Intelligence and the Future of the Drug Safety Professional. Drug Saf. 2019 Apr;42(4):491-497.

[0153] Devlin J,Chang M-W,Lee K,Toutanova K.Bert,:pre-training of deepbidirectional transformers for language understanding.In:Proceedings of the2019 Conference of the North American Chapter of the Association forComputational Linguistics:Human Language Technologies,Volume 1(Long and ShortPapers),Minneapolis,MN,USA.pp.4171–4186.Association for ComputationalLinguistics.https: / / www.aclweb.org / anthology / N19-1423。

[0154] FDA Adverse Event Reporting System(FAERS)Public Dashboard.US Food andDrug Administration website.Updated June 30,2020.Accessed October 22,2020.https: / / fis.fda.gov / sense / app / d10be6bb-494e-4cd2-82e4-0135608ddc13 / sheet / 7a47a261-d58b-4203-a8aa-6d3021737452 / state / analysis。

[0155] Lee J,Yoon W,Kim S,Kim D,Kim S,So CH,Kang J.BioBERT:a pre-trainedbiomedical language representation model for biomedical textmining.Bioinformatics.2020 Feb 15;36(4):1234-1240。

[0156] Lewis DJ,McCallum JF.Utilizing Advanced Technologies to AugmentPharmacovigilance Systems:Challenges and Opportunities.Ther Innov RegulSci.2020 Jul;54(4):888-899.McMaster C,Liew D,Keith C,Aminian P,Frauman A.AMachine-Learning Algorithm to Optimise Automated Adverse Drug ReactionDetectionfrom Clinical Coding.Drug Saf.2019 Jun;42(6):721-725。

[0157] Mockute R,Desai S,Perera S,Assuncao B,Danysz K,Tetarenko N,Gaddam D,Abatemarco D,Widdowson M,Beauchamp S,Cicirello S,Mingle E.ArtificialIntelligence Within Pharmacovigilance:A Means to Identify Cognitive Servicesand the Framework for Their Validation.Pharmaceut Med.2019 Apr;33(2):109-120。

[0158] Pharmacovigilance.World Health Organization website.Updated 2020。Accessed November 6,2020。

[0159] https: / / www.who.int / teams / regulation-prequalification / pharmacovigilance。

[0160] Schmider J,Kumar K,LaForest C,Swankoski B,Naim K,Caubel PM.Innovationin Pharmacovigilance:Use of Artificial Intelligence in Adverse Event CaseProcessing.Clin Pharmacol Ther.2019 Apr;105(4):954-961。

[0161] Stergiopoulos S,Fehrle M,Caubel P,Tan L,Jebson L.Adverse DrugReaction Case Safety Practices in Large Biopharmaceutical Organizations from2007 to 2017:An Industry Survey.Pharmaceut Med.2019 Dec;33(6):499-510。

Claims

1. A method, characterized in that, include: A set of documents and metadata of each document in the set of documents are received by one or more computing devices, wherein the set of documents corresponds to a domain; The one or more computing devices generate a set of word embeddings for each document in the set of documents, each word embedding including one or more words from the corresponding document; The one or more computing devices tokenize each word embedding in the set of word embeddings into a set of segments, each segment including a word from the word embedding; The learning model is trained by the one or more computing devices to classify each document in the set of documents in the domain by recursively performing the following steps during each iteration of multiple iterations of the training: The one or more computing devices decompose each segment of each document in the set of documents into a set of features; The one or more computing devices assign part-of-speech tags to each segment of the set of segments in each document in the set of documents, based on a predetermined weight for each feature in the set of features assigned to the corresponding segment; The one or more computing devices assign dependency tags to each segment of the set of segments in each document in the set of documents based on the part-of-speech tag assigned to the corresponding segment and a predetermined weight for each feature in the set of features assigned to the corresponding segment; The one or more computing devices assign named entity recognition (NER) tags from a set of predetermined tags corresponding to the domain to each segment of the set of segments in each of the set of documents, based on the part-of-speech tagging and dependency tagging assigned to the corresponding segment, and the predetermined weights assigned to each feature in the set of features assigned to the corresponding segment. The assigned NER tags are verified by the one or more computing devices by comparing the metadata of each document with the assigned NER tags of the corresponding document; and The one or more computing devices modify the predetermined weights of each feature in the set of features assigned to the corresponding segment based on the verification of the assigned NER labels, so as to be used in subsequent iterations of the multiple iterations of the training.

2. The method according to claim 1, characterized in that, Also includes: The one or more computing devices receive a request to classify new documents corresponding to the domain; The one or more computing devices generate new word embeddings that include one or more words from the new document; The one or more computing devices embed the new word corresponding to the domain into a new segment; The one or more computing devices decompose each of the new segments in the new document into a new set of features; The one or more computing devices use a trained learning model to assign new part-of-speech tags to each of the new segments in the new document based on a predetermined weight assigned to each of the new features in the set of new features to the corresponding new segment; Using the trained learning model, one or more computing devices assign new dependency tags to each of the new segments in the new document, based on the new part-of-speech tag assigned to the corresponding new segment and the predetermined weight of each feature of the set of new features assigned to the corresponding new segment; Using the trained learning model, the one or more computing devices assign new NER tags from the set of predetermined tags corresponding to the domain to each segment of the new document, based on the part-of-speech tagging and dependency tagging assigned to the corresponding new segment, and the predetermined weights of each feature in the set of new features assigned to the corresponding new segment; and One or more computing devices use the trained learning model to classify the new document corresponding to the domain based on the assigned NER labels.

3. The method according to claim 2, characterized in that, Also includes: The set of new segments and the assigned NER tags are extracted by the one or more computing devices; and A knowledge base is generated by the one or more computing devices, wherein: The knowledge base includes multiple nodes and multiple edges. The edge connects each of the plurality of nodes to at least one other node based on relationships, and Each of the plurality of nodes includes at least one segment from the set of new segments and a corresponding NER tag from the assigned NER tags.

4. The method according to claim 3, characterized in that, It also includes identifying the relationship between one of the plurality of nodes and different nodes among the plurality of nodes based on a first NER tag among the NER tags assigned in the node and a second NER tag among the assigned NER tags stored in different nodes by the one or more computing devices.

5. The method according to claim 1, characterized in that, The learning model uses a statistical model to assign the part-of-speech tagging to each of the one or more strings in each of the documents in the set of documents, to assign the dependency tagging to each of the segments in the set of documents in the set of documents, and to assign the NER tagging from the set of predetermined tags corresponding to the domain to each of the segments in the set of documents in the set of documents.

6. The method according to claim 1, characterized in that, The trained learning model implements the supervised learning algorithm.

7. The method according to claim 1, characterized in that, Also includes: The one or more computing devices generate a feature map for each document in the set of documents based on the set of features for each corresponding segment of each corresponding document; The one or more computing devices generate a simplified feature map for each document in the set of documents by reducing the dimensionality of the feature map for each corresponding segment of each corresponding document; The simplified feature map of each corresponding segment of each corresponding document is transformed into a vector by one or more computing devices; and The predetermined weights assigned to each feature in the set of features are applied by the one or more computing devices to each corresponding vector.

8. A system, characterized in that, include: Memory; A processor coupled to the memory, wherein the processor is configured to: Receive a set of documents and metadata for each document in the set of documents, wherein the set of documents corresponds to a domain; Generate a set of word embeddings for each document in the set of documents, each word embedding including one or more words from the corresponding document; Each word embedding in the set of word embeddings is tokenized into a set of segments, each segment including a word from the word embedding; The learning model is trained to classify each document in the set of documents in the domain by recursively performing the following steps during each iteration of multiple iterations of the training: Decompose each segment of each document in the set of documents into a set of features; Based on a predetermined weight assigned to each feature in the set of features to the corresponding segment, part-of-speech tags are assigned to each segment in the set of segments of each document in the set of documents; Based on the part-of-speech tag assigned to the corresponding segment and the predetermined weight of each feature in the set of features assigned to the corresponding segment, dependency tags are assigned to each segment in the set of segments of each document in the set of documents; Based on the part-of-speech tagging and dependency tagging assigned to the corresponding segment, and the predetermined weight of each feature in the set of features assigned to the corresponding segment, named entity recognition (NER) tags from a set of predetermined tags corresponding to the domain are assigned to each segment in the set of segments of each document in the set of documents; The assigned NER tags are verified by comparing the metadata of each document with the assigned NER tags of the corresponding document; and Based on the NER labels assigned for verification, the predetermined weights of each feature in the set of features assigned to the corresponding segment are modified for use in subsequent iterations of the multiple iterations of the training.

9. The system according to claim 8, characterized in that, The processor is also configured to: Receive a request to classify new documents corresponding to the domain; Generate new word embeddings that include one or more words from the new document; The new word corresponding to the domain is embedded into a new segment; Each new segment in the new document is decomposed into a new set of features; Using a trained learning model, new part-of-speech tags are assigned to each of the new segments in the new document based on a predetermined weight assigned to each of the new features in the set of new features to the corresponding new segment; Using the trained learning model, new dependency tags are assigned to each of the new segments in the new document's set of new segments, based on the new part-of-speech tags assigned to the corresponding new segments and the predetermined weights of each feature of the set of new features assigned to the corresponding new segments. Using the trained learning model, based on the part-of-speech tagging and dependency tagging assigned to the corresponding new segment, and the predetermined weights of each feature in the set of new features assigned to the corresponding new segment, new NER tags from the set of predetermined tags corresponding to the domain are assigned to each segment in the set of new segments of the new document. and Using the trained learning model, the new document corresponding to the domain is classified based on the assigned NER labels.

10. The system according to claim 9, characterized in that, The processor is also configured to: Extract the new set of segments and the assigned NER tags; and Generate a knowledge base, in which: The knowledge base includes multiple nodes and multiple edges. The edge connects each of the plurality of nodes to at least one other node based on relationships, and Each of the plurality of nodes includes at least one segment from the set of new segments and a corresponding NER tag from the assigned NER tags.

11. The system according to claim 10, characterized in that, The processor is also configured to identify the relationship between one of the plurality of nodes and the different nodes in the plurality of nodes based on a first NER tag among the NER tags assigned in the node and a second NER tag among the assigned NER tags stored in different nodes.

12. The system according to claim 8, characterized in that, The learning model uses a statistical model to assign the part-of-speech tagging to each of the one or more strings in each of the documents in the set of documents, to assign the dependency tagging to each of the segments in the set of documents in the set of documents, and to assign the NER tagging from a set of predetermined tags corresponding to the domain to each of the segments in the set of documents in the set of documents.

13. The system according to claim 8, characterized in that, The trained learning model implements the supervised learning algorithm.

14. The system according to claim 8, characterized in that, The processor is also configured to: A feature map for each document in the set of documents is generated based on the set of features for each corresponding segment of each corresponding document; A simplified feature map for each document in the set of documents is generated by reducing the dimensionality of the feature map for each corresponding segment of each corresponding document. Transform the simplified feature map of each corresponding segment of each corresponding document into a vector; and The predetermined weights assigned to each feature in the set of features are applied to each corresponding vector.

Images