Medical information processing device, medical information processing method, and medical information processing program

Abstract

To make efficient search processing or indexing processing on medial text documents.SOLUTION: A medical information processing apparatus according to the present embodiment comprises a processing unit that, by using a trained model trained to classify medical text documents into medical classification codes, acquires weightings on text terms included in at least one medical text document, while associating the weightings with the medical classification codes, and executes at least one of search processing and indexing processing by using the weightings.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The embodiments disclosed in this specification relate to a medical information processing device, a medical information processing method, and a medical information processing program. [Background technology]

[0002] It is known to perform natural language processing (NLP) to process free or unstructured text to obtain desired information. For example, in a medical context, the text to be analyzed may be a clinician's text note. The clinician's text note may be stored in an Electronic Medical Record. The text may be analyzed to obtain information about, for example, a medical condition or a type of treatment. Natural language processing may be performed using deep learning methods, such as neural networks.

[0003] For example, clinical text, such as that held in an electronic medical record, may be difficult to search. The text typically contains expert jargon. Terms used within the text may be abbreviated. In some situations a single term may be abbreviated in multiple different ways.

[0004] Some terms used in the text may be unclear. Some terms may be written incorrectly, for example there may be typos or spelling errors when entering clinical text. Some terms may require contextual knowledge to be understood.

[0005] Clinical texts may often use synonyms, for example, one or more brand names may be used instead of the scientific name of a drug.

[0006] Searching clinical text using a single search term may not retrieve all relevant information about that search term. For example, if a search application searches for matches to the single search term "alcohol," the search application may not retrieve terms such as "ETOH," which is commonly used as shorthand for alcohol. The search application may not retrieve terms such as "acamprosate," which is a drug used to treat alcohol abuse and indicates alcohol use.

[0007] A simple search with one search term may, for example, face problems involving negation, uncertainty, laterality, and / or coreference analysis.

[0008] Negation is when a term is expressed negatively. For example, a search for "alcohol" might return an entry containing the text "denies use of alcohol". This entry might not be considered useful if the reason for the "alcohol" search was to find instances of alcohol use.

[0009] Uncertainty may appear in text, for example, where "possible" is used before an event or diagnosis. A simple search may not reflect the degree of certainty or uncertainty of the search terms.

[0010] Laterality may include consideration of whether the symptom is on the right or left side of the body. If a simple search for symptoms is used, it may be difficult to distinguish between right-sided and left-sided occurrences.

[0011] Coreference analysis may involve identifying people to which a text item refers. It is common for clinical text to include family medical histories. A simple search may not be able to distinguish between text that refers to the patient and text that refers to another individual. For example, a search for the term "alcohol" may include mentions of alcohol use by the patient's parents but not by the patient.

[0012] Clinical coding systems are known. Clinical coding systems may also be known as terminologies or ontologies. Clinical coding systems represent clinical concepts along with their relationships. Known clinical coding systems such as SNOMED CT (Systematized Nomenclature of Medicine - Clinical Terms), ICD (the 10th revision of the International Statistical Classification of Diseases and Related Health Problems), and OPCS-4 (OPCS Classification of Interventions and Procedures) are resource rich and comprehensive. Clinical coding systems include clinical concepts and relationships between concepts.

[0013] Known clinical coding systems include a list of synonyms for a given clinical term, in which the term is expressed in formal terms that may not correspond well to the informal and / or abbreviated terms commonly used in clinical practice. [Prior art documents] [Patent documents]

[0014] [Patent Document 1] U.S. Patent No. 7,899,764 Summary of the Invention [Problem to be solved by the invention]

[0015] One of the problems that the embodiments disclosed in this specification and the drawings aim to solve is, for example, to improve the efficiency of search processing or indexing processing of medical text documents. However, the problems that the embodiments disclosed in this specification and the drawings aim to solve are not limited to the above problems. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0016] The medical information processing device of this embodiment includes a processing unit that uses a trained model trained to classify medical text documents into medical classification codes, obtains weightings of text terms contained in at least one medical text document in association with the medical classification code, and performs at least one of a search process and an indexing process using the weightings. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 shows a schematic diagram of an apparatus according to an embodiment. [Diagram 2] FIG. 2 shows a schematic flow chart of a method according to an embodiment. [Diagram 3] FIG. 3 shows a simplified example of a thesaurus according to an embodiment. [Figure 4] FIG. 4 shows a schematic flow chart of a method according to an embodiment. [Diagram 5] FIG. 5 shows a schematic representation of a smart search engine according to an embodiment. [Figure 6] FIG. 6 illustrates an example of a user interface for search query expansion according to an embodiment. [Figure 7] FIG. 7 shows an example of a synonym list with associated importance scores. [Figure 8] FIG. 8 shows an example of ranked search results by keyword. [Figure 9]Figure 9 shows an example of ranked search results by code. [Figure 10] FIG. 10 shows a schematic diagram of a connection made between two clinical coding systems using the method of an embodiment. [Figure 11] Figure 11 shows an example of a code with related codes for two sub-concepts. [Figure 12] FIG. 12 shows a flow chart outlining an indexing method according to an embodiment. [Figure 13] FIG. 13 shows a schematic diagram of the use of trained machine learning models in the index. [Figure 14] FIG. 14 shows a schematic diagram of a user interface according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, a medical information processing apparatus, a medical information processing method, and a medical information processing program according to an embodiment will be described in detail with reference to the drawings.

[0019] A medical information processing apparatus 10 according to an embodiment is shown diagrammatically in Fig. 1. In this embodiment, the medical information processing apparatus 10 is configured to process medical text. The medical text may comprise, for example, clinical notes. In other embodiments, the medical information processing apparatus 10 may be configured to process any suitable data, for example non-medical data.

[0020] The medical information processing device 10 includes a computing device 12, in this example a personal computer (PC) or workstation, connected to a display screen 16 or other display device and one or more input devices 18, such as a computer keyboard and / or mouse.

[0021] Computing device 12 receives the medical text from data store 20. In alternative embodiments, computing device 12 receives the medical text from one or more further data stores (not shown) instead of or in addition to data store 20. For example, computing device 12 may receive the medical text from one or more remote data stores (not shown) that may form part of an electronic medical record system or a picture archiving and communication system (PACS).

[0022] The computing device 12 provides processing resources for automatically or semi-automatically processing medical text data. The computing device 12 includes a processing unit 22. The processing unit 22 uses a trained model trained to classify medical text documents into medical classification codes to obtain weights of text terms contained in at least one medical text document in association with the medical classification codes, and performs at least one of a search operation and an indexing operation using the weights. Note that the processing unit 22 is an example of a processing unit.

[0023] More specifically, the processing device 22 includes a training circuit 24 configured to train a machine learning model to classify documents into clinical codes; a data mining circuit 26 configured to data mine the documents for related terms; a search circuit 28 configured to search the documents; and an indexing circuit 29 for indexing the documents.

[0024] In this embodiment, the training circuitry 24, data mining circuitry 26, search circuitry 28, and indexing circuitry 29 are each implemented in computing device 12 by a computer program having computer readable instructions executable to perform the method of the embodiment, however, in other embodiments the various circuits may be implemented as one or more ASICs (application specific integrated circuits) or FPGAs (field programmable gate arrays).

[0025] Computing device 12 includes a hard drive and other components of a PC, including RAM, ROM, a data bus, an operating system including various device drivers, and hardware devices including a graphics card, although such components are not shown in FIG.

[0026] FIG. 2 shows a schematic flow chart of a method according to an embodiment. The device of FIG. 1 is configured to execute the method of FIG. 2. The method of FIG. 2 includes a model training phase, a thesaurus creation phase, and a search phase. In the embodiment of FIG. 2, the model training phase, thesaurus creation phase, and search phase are all performed by the medical information processing device 10 of FIG. 1. In other embodiments, different phases may be performed by different devices. In some embodiments, one phase may be divided among multiple devices. Any suitable one or more devices may be used.

[0027] The model training phase of the method of Figure 2 comprises stages 30 through 36. In stage 30, training circuitry 24 receives a training corpus from data store 20. In other embodiments, training circuitry 24 may receive the training corpus from any suitable store or another device.

[0028] The training corpus comprises a plurality of training documents, in this embodiment each of which comprises an individual entry in an electronic medical record system, which may relate to a large number of patients, for example, all patients treated at a given hospital or other institution.

[0029] The training documents are already classified according to an existing clinical coding system with a predefined clinical code set. In this embodiment, the clinical coding system is the International Statistical Classification of Diseases and Related Health Problems (ICD). The tenth revision of the ICD coding system is referred to as ICD-10 and the ninth revision of the ICD coding system is referred to as ICD-9. In other embodiments, the clinical coding system may be any suitable clinical coding system, such as, for example, SNOMED CT (Systematized Nomenclature of Medicine - Clinical Terms), OPCS-4 (OPCS Classification of Interventions and Procedures), or SMR (Scottish Morbidity Records). In some embodiments, clinical codes from multiple clinical coding systems may be used. The clinical code set comprises clinical codes that may be routinely assigned in clinical practice.

[0030] Clinical codes are also sometimes referred to as medical classification codes. Each code represents a distinct clinical concept. For example, ICD-10 clinical code 3E03317 is used to represent the concept of injection or infusion of a thrombolytic agent into a peripheral vein. ICD-10 clinical code Z72.0 is used to represent the concept of smoking disorder.

[0031] In some embodiments, the medical classification code used may comprise any suitable label relating to, for example, a treatment or a disease. For example, the medical classification code may relate to a prescription order or a radiation treatment order. The medical classification code may relate to, for example, the location of the disease, the stage of the disease, or the disease term as described by a physician, such as the TMN classification of cancer.

[0032] Classification of documents using clinical codes is done according to the content of each document. For example, a document about injection or infusion of a thrombolytic agent into a peripheral vein is classified with the corresponding ICD-10 code for injection or infusion of a thrombolytic agent into a peripheral vein, i.e., 3E03317. A document about smoking disorders is classified with the corresponding ICD-10 code for smoking disorders, i.e., Z72.0. A document will be considered to be related to a given clinical code if it contains information about a concept represented by that clinical code. For example, if a document contains information that a patient is a smoker, the document may be classified with ICD-10 code Z72.0.

[0033] Different documents in the training corpus are classified with different clinical codes. For example, a first subset of documents in the training corpus are classified with ICD-10 code 3E03317. A second subset of documents, which partially overlaps with the first subset, are classified with ICD-10 code Z72.0. Many other clinical codes may be used to classify documents in the training classification.

[0034] In this embodiment, classification of the documents in the training corpus was performed manually by one or more experts, although in other embodiments any suitable classification method may be used.

[0035] At stage 32, the training circuitry 24 receives a set of ground truth data comprising clinical codes for each of the training documents of the training corpus classified by the expert. Each training document is associated with a respective piece of ground truth data. The ground truth data for a document may comprise one or more of a predefined set of clinical codes. In some embodiments, the ground truth data is included in the training corpus such that the ground truth data and the training corpus are received in a single step.

[0036] At stage 34, training circuitry 24 uses the training corpus of stage 30 and the ground truth data of stage 32 to train a machine learning model for document-level classification using the predetermined clinical code set. In this embodiment, the machine learning model is a multi-task model that is trained to classify documents with multiple different clinical codes in the predetermined clinical code set.

[0037] In another embodiment, multiple machine learning models are trained by training circuitry 24. Each machine learning model is trained to classify documents into individual clinical codes or groups of clinical codes.

[0038] During training, documents from the training corpus are input to the machine learning model. The output of the machine learning model is compared to the ground truth data. The output error of the machine learning model is fed back to the machine learning model.

[0039] Any suitable training method may be used to train the machine learning model to classify documents into clinical codes. For example, training methods are described in Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J. and Eisenstein, J., 2018, June. Explainable Prediction of Medical Codes from Clinical Text. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (pp. 1101-1111). In this embodiment, the machine learning model is a neural network model, such as a convolutional neural network. A document of free text is input to the neural network model. The neural network model generates a feature representation of each term in the document of free text. A text term may be, for example, a single word, a group of words, or a part of a word. The feature representations of the text terms are each multiplied with an individual attention contribution to provide a classification output. The classification output of the neural network comprises a probability for each given set of ICD codes, which may be thresholded to determine which ICD codes to use to classify the input document.

[0040] In this embodiment, the neural network model also outputs an attention vector for each ICD code assigned to each document. The attention vector is derived from the attention contributions generated by the neural network model. The attention vector is described in more detail below with reference to stage 38.

[0041] In other embodiments, the machine learning model may not be a neural network model, and may comprise any artificial intelligence or machine learning model with an attention model that uses free text as input and clinical codes as prediction targets.

[0042] At stage 36, the training circuitry 24 outputs a trained machine learning model. The trained machine learning model is trained to perform document-level classification using clinical codes of a predefined clinical code set. In this embodiment, the clinical codes are ICD codes. In other embodiments, the trained machine learning model may be trained to perform document-level classification using any suitable clinical codes.

[0043] The data mining circuit 26 receives the trained machine learning model from the training circuit 24. The indexing circuit 29 may also receive the trained machine learning model from the training circuit 24. Use of the indexing circuit 29 is described below with reference to FIG.

[0044] In this embodiment, the model training phase, thesaurus creation phase, and search phase are all performed by the medical image processing device 10. In other embodiments, the model training phases of stages 30 to 36 are not performed by the medical image processing device 10. In such embodiments, the data mining circuitry 26 may receive the trained machine learning model from the data store 20 or another data store. The data mining circuitry 26 may also receive the training corpus from the data store 20 or another data store.

[0045] After stage 36, the flow chart proceeds to a thesaurus creation phase, which comprises stages 38 to 44 of the flow chart.

[0046] At stage 38, the data mining circuitry 26 applies the trained machine learning model to each training document in the training corpus to obtain a plurality of attention vectors.

[0047] Each training document is a free text document. An example of a training document might be a discharge summary that a physician enters into an electronic medical record system for a patient's discharge.

[0048] For each training document, the data mining circuitry 26 inputs the document into a trained machine learning model that outputs a separate attention vector for each clinical code into which the training document was classified.

[0049] For example, consider the case where a training document is classified with a first clinical code, for example, ICD-10 code 3E03317, which corresponds to the concept of injection or infusion of a thrombolytic agent into a peripheral vein.

[0050] In the process of classifying a document into a first clinical code, the trained machine learning model determines an attention contribution for each text term (e.g., for each word) in the document. The attention contribution of a text term indicates how important the text term was in classifying the document into the first clinical code. If a text term occurs multiple times in a document, each instance of the text term is considered separately. The machine learning model may be trained to take into account the content of the text term. Therefore, different instances of the text term may have different attention contributions.

[0051] The attention vector output by the machine learning model in the first clinical code is a vector with as many elements as the number of text terms in the document. For example, the text terms may be words, groups of words, or parts of words. Each element of the attention vector comprises an attention weighting of the corresponding text term of the document. The attention weighting is derived from the attention contribution. In this embodiment, the attention weighting is obtained by normalizing the attention contribution so that the sum of the attention weightings in the attention vector is 1. The calculation, presentation, etc. of importance scores of the text terms using the attention weighting is an example of an attention mechanism.

[0052] The attention weighting of a text term may be considered to represent the importance of the text term in classifying it into a first clinical code. In some embodiments, the attention weighting may be used as an importance score. For example, the attention weighting may be considered to provide a word-level importance score. In other embodiments, different importance scores may be used, as described below.

[0053] Some text terms of the document have an attention weight of zero. A text term with a zero attention weight is a text term that does not contribute to the classification of the document with the first clinical code. A text term with a non-zero attention weight contributes to the classification of the document with the first clinical code. The more important a text term is to the classification, the larger the size of the attention weight.

[0054] Consider the following example text: "Acute cerebrovascular accident: The patient was not a candidate for tissue plasminogen activator. A neurology consult was obtained from Dr. X."

[0055] An attention vector is obtained for a document having the above text. The attention vector is associated with the classification of the document with the ICD-9 code 99.10. The attention vector has a non-zero attention weight for each word in the sequence "for tissue plasminogen activator A neurology". In the attention vector, other words in the example text have zero attention weights. Among the non-zero attention weights, "activator" has the highest attention weight, "plasminogen" has the second highest attention weight, and "tissue" has the third highest attention weight. In determining the attention weights, the neural network considers the context of each word, not just the word itself.

[0056] Consider the attention vector for a second, different clinical code. For example, the second clinical code may be ICD-9 code 305.1, which corresponds to the concept of smoking disorder. If a document is classified under both the first and second clinical code, an attention vector for the first clinical code and an attention vector for the second clinical code are obtained for the same document. If the attention vector for the first clinical code and the attention vector for the second clinical code relate to the same document, the attention vector for the second clinical code has the same length as the attention vector for the first clinical code. In each case, the length of the attention vector is the number of text terms in the document, e.g., the number of words in the document. A separate attention weight is given for each text term in the document, e.g., for each word in the document.

[0057] Different words are important in classifying a document in a first clinical code and in classifying a document in a second clinical code. The attention vector for the second clinical code is different from the attention vector for the first clinical code.

[0058] Consider another example text. "social history patient smokes pack day name ni current tobacco smoker heavy has been smoking at this level for many years social history currently smokes cigarettes per day"

[0059] An attention vector is obtained for the document with the above text. The attention vector is related to the classification of the document with ICD-9 code 305.1. The attention vector has a non-zero attention weight for each word in the sequence 'history patient smokes pack day', 'ni current tobacco smoker heavy', 'smoking', and 'history currently smokes cigarettes per'. In the attention vector, the other words in the example text have zero attention weights. The word that got the highest attention weight is 'smokes' in the first instance. The second highest attention weight was given to 'tobacco'. The third highest attention weight was given to 'smoker', 'smoking', 'smokes' and 'cigarettes' in the second instance. The third highest attention weight was given to "pack".

[0060] The data mining circuitry 26 derives an attention vector for every training document. For each training document, the data mining circuitry 26 derives a separate attention vector for each clinical code into which the training document is classified. The attention vectors are sometimes referred to as code-specific or label-specific attention vectors.

[0061] The data mining circuitry 26 derives, for each clinical code, a separate plurality of attention vectors associated with the subset of training documents classified into that clinical code.

[0062] At stage 40, the data mining circuitry 26 data mines the attention vector for each clinical code. The data mining circuitry 26 data mines the attention vector for keywords related to each clinical code. Keywords may include individual words, groups of words, parts of words, or abbreviations.

[0063] For each clinical code of a given clinical code set, keywords associated with that clinical code are data mined from the training documents by selecting text terms in the training documents that have an attention weighting greater than a given threshold.

[0064] In this embodiment, the threshold is selected by a user. The threshold may be described as: Data mining circuitry 26 adjusts the user-selected threshold to the document length d according to the following equation (1):

[0065] Threshold k(d) = ck(1 / d) (1)

[0066] Thresholdk(d) is the threshold after adjustment taking into account the document length d.

[0067] The threshold is adjusted to take into account the document length d, since the attention weights of all words in a given document are normalized to sum to 1. One might expect individual attention weights to be lower for long documents and higher for short documents. Adjusting the threshold by document length d allows for fair comparisons between documents.

[0068] The data mining circuitry 26 applies the adjusted threshold to each attention vector for the clinical code of interest. In this embodiment, the data mining circuitry 26 uses the same user-selected threshold for each clinical code in a given clinical code set. In other embodiments, different thresholds may be used for different clinical codes.

[0069] The data mining circuit 26 identifies each text term that has an attention weighting above the adjusted threshold for the document in which the term occurs. The data mining circuit 26 aggregates the identified terms by clinical code.

[0070] In this embodiment, data mining circuitry 26 considers each identified term to be a keyword. In other embodiments, data mining circuitry 26 may apply additional criteria.

[0071] In this embodiment, the data mining circuitry 26 generates an ordered list of keywords for each clinical code in a given clinical code set. The ordered list comprises all text terms identified as exceeding an adjusted threshold of documents in which the term occurs. The list is ordered by the frequency of each keyword identified in the training vectors for the clinical code. In other embodiments, the data mining circuitry 26 may generate an unordered list of keywords.

[0072] For example, consider the attention vector obtained for ICD-9 code 250.00, which corresponds to the concept of diabetes mellitus with no mention of complications.

[0073] In one embodiment, the data mining circuit 26 outputs a list of the Nth words or phrases ordered by frequency in the training data of the attention vector for ICD-9 code 250.00, where N may be a number selected by the user. In the example below, N is 18.

[0074] The following list of examples was obtained: insulin 1249 diabetes<diabetes> 1043 dm803 diabetes mellitus<diabetes mellitus> 592 atorvastatin 394 dm2 339 metformin 310 diabetic 263 protonix 259 diabetes mellitus type <diabetes mellitus type> 254 sugars 238 glyburide 230 simvastatin 223 etoh <ethyl alcohol> 213 type 2 diabetes<type 2 diabetes> 203 type 2 diabetes mellitus<type 2 diabetes mellitus> 124 dmii 124 hyperglycemia 114

[0075] Each text term in the list is accompanied by the number of instances of that text term that have an attention weighting above the adjusted threshold.

[0076] In the embodiment of Figure 2, for each clinical code in a given clinical code set, a separate list of keywords is obtained, each list of keywords comprising text terms identified by applying an adjusted threshold, and each list of keywords is ordered by the number of instances of each word that exceed the adjusted threshold.

[0077] At stage 42, data mining circuitry 26 uses the list of keywords to create a thesaurus with codes representing multiple concepts and a separate synonym list associated with each concept. Data mining circuitry 26 adds multiple terms to each synonym list.

[0078] Some of the terms in the synonym list are taken from an existing knowledge base, such as the ICD clinical coding system itself. The ICD clinical coding system comprises synonyms for concepts in the clinical coding system. As noted above, the synonyms present in the clinical coding system primarily include formal clinical terms and may not include informal terms.

[0079] The other terms in the synonym list are keywords obtained at stage 40 by thresholding the attention vector from stage 38. At least some of the keywords obtained at stage 40 may be less formal than those found in the ICD clinical coding system. For example, the keywords obtained at stage 40 may include abbreviations and / or misspellings.

[0080] In some embodiments, rather than treating terms in a synonym list independently, the terms may be clustered into synonym groups. For example, the clustering may be performed using the first character sequence of the abbreviation. For example, the clustering may be performed based on an edit distance between terms, such as the Levenshtein distance. The edit distance quantifies the number of operations required to transform one string of text into another. The Levenshtein distance takes into account deletion, insertion, or substitution operations. The edit distance determination may be used to find misspellings. The misspellings are then grouped with the correctly spelled term for that misspelling.

[0081] 3 illustrates a simplified example of a thesaurus 60 according to an embodiment. Thesaurus 60 comprises a number of ICD-10 codes 1, 2, 3, and 4. The relationships between ICD-10 codes 1, 2, 3, and 4 are known from the ICD-10 clinical coding system. In the example illustrated in FIG. 3, ICD-10 code 1 is associated with a first concept, and ICD-10 codes 2, 3, and 4 are associated with different subconcepts of the concept of ICD-10 code 1.

[0082] The thesaurus of Figure 3 also includes miscellaneous concepts 5 (Miscellaneous Concepts 5 of Figure 3) that are not part of the ICD-10 clinical coding system. Miscellaneous concepts 5 may be part of another clinical coding system. Or, miscellaneous concepts 5 may be part of a set of local codes, such as, for example, local codes for a particular hospital or region.

[0083] For each concept represented by a code 1, 2, 3, 4, 5 in thesaurus 60, thesaurus 60 includes a respective synonym list 61, 62, 63, 64, 65 associated with that concept.

[0084] ICD-10 code 1 has a synonym list 61 that includes term A, term B, and term C. ICD-10 code 2 has a synonym list 62 that includes term A, term B, term C, and term D. ICD-10 code 3 has a synonym list 63 that includes term A, term B, and term E. ICD-10 code 4 has a synonym list 64 that includes term A, term B, term E, and term F. Miscellaneous concept 5 has a synonym list 65 that includes term H, term I, and term F.

[0085] Some of the terms in the synonym list are obtained from the ICD-10 clinical coding system. Other terms in the synonym list are obtained through data mining as described above. Terms obtained through data mining are sometimes referred to as discovered terms.

[0086] At stage 44, data mining circuitry 26 outputs the thesaurus created at stage 60. The thesaurus comprises a separate list of synonyms for each clinical code in a given clinical code set. In this embodiment, data mining circuitry 26 passes the thesaurus to search circuitry 28.

[0087] After the thesaurus creation phase of stages 38 through 44, the method proceeds to the search phase, which includes stages 46 through 58. The search phase is sometimes also referred to as the expansion phase.

[0088] At stage 46, the search circuitry 28 receives target text documents to be searched. The target text documents are documents that were not included in the training corpus. The target text documents may comprise all text documents related to a particular patient, for example, all of the text documents recorded in an electronic medical record system for that patient. The target documents may be received from the data store 20 or any other suitable data store. In some embodiments, the target text documents may be selected by a user. In some embodiments, a patient may be selected by a user and the search circuitry 28 may select all of the text documents associated with that patient.

[0089] At stage 48, the search circuitry 28 displays a search box on the display screen 16 to the user. The user enters query terms into the search box. The query terms are the terms that the user wishes to use in searching the target text documents. For example, the user might enter the query term "Blood pressure" to find references to blood pressure that occur within the target text documents.

[0090] At stage 50, the search circuitry 28 uses the trained model to identify a number of medical classification codes to which the query term belongs. That is, the search circuitry 28 matches the query term to at least one of the coded concepts included in the thesaurus. Some of the coded concepts included in the thesaurus have ICD-10 clinical codes. Other coded concepts included in the thesaurus may not have an ICD-10 code. Each coded concept included in the thesaurus has an associated synonym list. The search circuitry 28 matches the query term to at least one of the coded concepts by identifying concepts that have the query term in part of their synonym list.

[0091] Returning again to the exemplary thesaurus 60 of FIG. 3, consider the case where a user types term F into a search box. Search circuitry 28 matches the user's query terms to concepts that include term F in their synonym list. Because term F is included in synonym list 64 associated with term F, search circuitry 28 locates ICD-10 code 4. Because term F is included in synonym list 65 associated with term F, search circuitry 28 locates miscellaneous concept 5 in thesaurus 60.

[0092] At stage 52, the search circuitry 28 determines a list of synonyms for the query terms using the coded concepts identified at stage 50. The search circuitry 28 includes in the list of synonyms any terms that are included in the synonym list of the identified concepts.

[0093] In other embodiments, the search circuitry 28 searches the synonym list directly for the query term, rather than first identifying concepts with which the query term is associated. In such embodiments, the search is still based on the association of synonyms to a given coded concept. In searching the synonym list, the search circuitry 28 associates the query term and synonyms with coded concepts.

[0094] The search circuitry 28 finds instances of the query term in the thesaurus. The search circuitry 28 examines synonym lists in which the instance of the query term occurs and returns synonyms from each synonym list in which the query term occurs.

[0095] Synonyms are sometimes also called sister terms. Note that some of the synonyms returned from a synonym list may not mean exactly the same thing as each other. Terms in a synonym list are considered synonyms because they relate to a common concept.

[0096] Returning again to the example of FIG. 3, if the query term is term F, then search circuitry 28 selected a concept with ICD-10 code 4 and miscellaneous concept 5 in stage 50. In stage 52, search circuitry 28 determines a list of synonyms for term F that includes term A, term B, and term E from synonym list 65, and term H and term I from synonym list 65.

[0097] At stage 54, the search circuitry 28 displays a list of synonyms to the user on the display screen 16. For example, if the query term is "Blood pressure," the search circuitry 28 might display "BP," "SBP," "Diastolic," and "Blood-pressure." Each synonym in the list of synonyms is selectable by the user. In some embodiments, the user is given the option to select all of the synonyms.

[0098] By displaying a list of synonyms, the search circuitry 28 provides a list of alternative searches that are related to the user's query terms.

[0099] The user selects synonyms from the list of synonyms that they want to add to the search along with the original search term. For example, the user might select to search for "BP" or "Blood-pressure" in addition to "Blood pressure." The search circuitry 28 receives a selection of one or more of the displayed synonyms from the user.

[0100] In other embodiments, search circuitry 28 automatically selects all of the synonyms on the list of synonyms determined at stage 52. In some embodiments, the user is given the opportunity to deselect one or more synonyms on the list of synonyms.

[0101] At stage 56, the search circuitry 28 searches each target text document for instances of the query terms and instances of the selected synonyms. The search circuitry 28 identifies which target text documents contain at least one instance of the query terms and / or at least one instance of the selected synonyms. Within the identified text documents, the search circuitry 28 locates text portions that comprise the query terms and / or the synonyms.

[0102] At stage 58, the search circuitry 28 displays the target text documents identified at stage 56 to the user. The target text documents are displayed on the display screen 16. The search circuitry 28 may display each target text document such that text portions that include the query terms and / or synonyms are displayed. For example, the search circuitry 28 may scroll the document to the appropriate text portion. The search circuitry 28 may highlight instances of the query terms and selected synonyms.

[0103] A user may use display screen 16 to navigate through any of the displayed text documents.

[0104] Returning again to the example of FIG. 3 in which the user selected term F, the search circuitry 28 displays to the user at stage 54 a list of synonyms comprising term A, term B, term E, term H, and term I. The user may select one or more synonyms from the list of synonyms. For example, the user may select term A and term H. At stage 56, the search circuitry 28 searches the target text document for instances of term F, term A, and term H. The search circuitry 28 displays to the user any target text documents that contain term F, term A, and / or term H. The search circuitry 28 highlights instances of term F, term A, and term H in the displayed target text document.

[0105] The search circuitry 28 may be contemplated to provide a smart search engine that considers not only the query terms entered by the user, but also synonyms of the query terms. The method of FIG. 2 allows for easy navigation of the ontology by augmenting the original list of synonyms of the clinical coding system with natural language terms that may occur in common usage. Synonyms found using the method of FIG. 2 may include common misspellings, abbreviations, and / or brand names. For example, "smoker" or "ppd" may prove to be more intuitive and / or convenient search terms than "tobacco use disorder."

[0106] Machine learning may seamlessly integrate into existing knowledge bases, such as existing clinical coding systems, and may provide improved search. In some situations, the search may be more comprehensive than a text search for only the query terms. The search may also be more comprehensive than a text search for synonyms contained in the clinical coding system, since it includes more informal terms.

[0107] One or more clinical coding systems may be leveraged as the backbone of a search system. The search system uses an attention mechanism of a trained model (e.g., a CNN model) trained to predict clinical codes. The search system uses the attention mechanism to data mine concept-related search terms from clinical texts and uses the data mined terms to augment a synonym list of the selected clinical coding system or systems. An automatic thesaurus is created from the clinical coding system or systems and the data mined terms.

[0108] In the embodiment of FIG. 2, a GPU is used to train the machine learning model and create the thesaurus. However, a GPU is not required when deploying the thesaurus. Once the thesaurus is created, it may be deployed without deploying a trained machine learning model. The computational resources required to deploy the thesaurus are smaller than those required to deploy the machine learning model itself.

[0109] 2, search circuitry 28 creates a list of synonyms for each concept. When a user enters a query term, search circuitry 28 finds concepts related to the query term, finds a synonym list for the concept, and uses the synonym list to display a list of synonyms for the query term.

[0110] In other embodiments, the search circuitry 28 may also display concepts identified as related to the query terms to the user, and the user may select to search for documents related to one or more of the displayed concepts, such as documents categorized under one or more of the displayed concepts.

[0111] Returning again to the example of FIG. 3 where term F is used as a query term, search circuitry 28 may suggest to the user the concept of ICD-10 code 4 and miscellaneous concept 5. The user may select to search for documents classified with ICD-10 code 4. The user may select to search for documents classified with the miscellaneous concept. The user may select to search for documents that include miscellaneous concept 5.

[0112] In further expansion of the thesaurus 60 of FIG. 3, a user types term C into the search box. Because term C occurs in synonym list 61 for ICD-10 code 1 and synonym list 62 for ICD-10 code 2, search circuitry 28 identifies ICD-10 code 1 and ICD-10 code 2. Search circuitry 28 creates a list of synonyms for term C, including term A and term B from synonym lists 61 and 62, and term D from synonym list 62. Search circuitry 28 displays the list of synonyms to the user. The user may select one or more of term A, term B, and term D for searching. Search circuitry 28 may also display the identified concepts, which are ICD-10 code 1 and ICD-10 code 2, to the user. The user may select ICD-10 code 1 and / or ICD-10 code 2 for searching.

[0113] In some embodiments, a user may enter more than one query term into the search box. In one example using thesaurus 60 of FIG. 3, a user enters term A+term C into the search box. Search circuitry 28 identifies concepts in thesaurus 60 that have both term A and term C in their synonym list. Search circuitry 28 identifies ICD-10 code 1 and ICD-10 code 2. Search circuitry 28 then identifies other terms in synonym lists 61 and 62 for ICD-10 code 1 and ICD-10 code 2. Search circuitry 28 identifies and displays term B and term D to the user. The user may select term B and / or term D for searching. Search circuitry 28 may also display ICD-10 code 1 and ICD-10 code 2 to the user. The user may select ICD-10 code 1 and ICD-10 code 2 for searching.

[0114] In some embodiments, the search circuitry 28 provides a user with the opportunity to navigate a tree structure of related concepts. For example, in response to a user's selection of ICD-10 code 2 or a user's selection of a keyword associated with ICD-10 code 2, the search circuitry 28 may display to the user a tree structure that includes ICD-10 code 1, ICD-10 code 2, ICD-10 code 3, and ICD-10 code 4. The user may select any of ICD-10 code 1, ICD-10 code 2, ICD-10 code 3, and ICD-10 code 4, which are conceptually related to ICD-10 code 2.

[0115] 2, the search circuitry 28 finds synonyms for the query terms. The search circuitry 28 displays the synonyms to a user. When the user selects one or more of the synonyms, the search circuitry 28 searches the target text document for the query terms and the selected synonym or synonyms.

[0116] In other embodiments, the search circuitry 28 searches the target text documents for the query terms and / or synonyms before displaying the list of synonyms for the query terms to the user. In some such embodiments, the search circuitry 28 determines how many of the target text documents contain each synonym. The displayed list of synonyms comprises, for each synonym, the number of target text documents that contain that synonym. In other embodiments, the search circuitry 28 determines and displays the number of instances of each synonym that occur in the target text. If the same query term occurs more than once in a document, the number of instances will be greater than the number of documents.

[0117] In some embodiments, the search circuitry 28 selects synonyms to display based on the number of target text documents in which the synonyms appear or the number of instances of the synonyms in the target text documents. In some embodiments, the search circuitry 28 orders the synonyms to be displayed based on the number of target text documents in which the synonyms appear or the number of instances of the synonyms in the target text documents.

[0118] Figure 4 shows a further flow chart relating to the user experience. In the embodiment of Figure 4, the user is a clinician. In other embodiments, the user may be any suitable user, such as, for example, any suitable medical professional or researcher.

[0119] The search circuitry displays a search box 70. The clinician types a search term into the search box 70. A search term may also be referred to as a query term. In Figure 4, the clinician types "blood press". The steps of displaying the search box and accepting user input may be as described above with reference to stage 48 of Figure 2.

[0120] The search circuitry 28 inputs the query terms into the smart search engine 72. In the embodiment of FIG. 4, the smart search engine 72 is implemented in the search circuitry 28. The smart search engine 72 applies an algorithm to generate related terms for the query terms and performs a search. The algorithm performs steps similar to those described above with reference to stages 50 and 52. The smart search engine 72 obtains a list of synonyms for the query terms. The synonyms may also be described as selected search terms or additional search terms.

[0121] 4, the smart search engine 72 also searches the target text documents for instances of the query terms and synonyms. The smart search engine 72 returns the number of target text documents that contain the query terms and a respective number of target text documents that contain each synonym in the synonym list.

[0122] 4, the query term is not included in the thesaurus, so the smart search engine 72 searches the thesaurus for the term in the thesaurus that is most similar to the query term. For example, the smart search engine 72 may use edit distance to determine similarity. If the query term is "blood press", the closest term in the thesaurus is "blood pressure".

[0123] The smart search engine 72 outputs a list 74 including the determined closest terms and a list of synonyms. The list 74 comprises the number of determined target text documents for the determined closest terms and for each list of synonyms. The smart search engine 72 updates the search box to show the determined closest terms 76.

[0124] The search circuitry 28 displays to the user a list 74. In Figure 4, the list 74 comprises: Blood pressure [4] BP [5] SBP[2] Diastolic [2] Blood pressure [1]

[0125] Numbers in square brackets are used to represent the number of target text documents that contain each term in list 74. In other embodiments, the list may display the number of instances of that term in each target text document, or in the entire set of target text documents.

[0126] The search circuitry 28 displays a display element 78 having the text "Select All." The clinician may select either the determined closest term or the list of synonyms by selecting the appropriate entry on the list 74, e.g., by clicking on the entry. The clinician may select all of the entries on the list 74 by selecting element 78, e.g., by clicking on element 78.

[0127] In FIG. 4, the determined closest term "Blood pressure" is selected. Search circuitry 28 displays all of the target text documents that contain "Blood pressure". In FIG. 4, four target text documents 80A, 80B, 80C, and 80D are displayed. Search circuitry 28 uses highlight region 82A to highlight instances of the term "blood pressure" in target text document 80A. Search circuitry 28 uses highlight region 82B to highlight instances of the term "blood pressure" in target text document 80B. Search circuitry 28 uses highlight region 82C to highlight instances of the term "blood pressure" in target text document 80C. Search circuitry 28 uses highlight region 82D to highlight instances of the term "blood pressure" in target text document 80D.

[0128] In the embodiment of FIG. 4, the display presented to the user further includes a list 84 of target document types to be searched. The list includes medical records, nursing records, radiology reports, and all specialties. In FIG. 4, all terms in list 84 are highlighted to indicate that the target documents were searched from all target document types. A display element 86 is displayed having the text "Click here to change." The user may click on display element 86 to change the target document types to be searched. For example, the user may delete one or more of the target document types. The user may select only one target document type, such as nursing records, or may select multiple document types. If the user changes the target document type, search circuitry 28 updates list 74 to include only terms that are in the selected target document type. Search circuitry 28 updates list 84 to indicate the user's new selection. For example, search circuitry 28 may highlight in list 84 only target document types that are relevant to the currently displayed search results.

[0129] FIG. 5 is an alternative representation of the smart search engine 72. A user types one or more query terms into the search box 70. The smart search engine 72 includes three components 90, 92, and 94. Component 90 is a clinical coding system, e.g., a known clinical coding system extended with a thesaurus. Component 92 maps the user's search query to the clinical coding system 90. Component 94 maps the coding system to free text notes to locate related terms in the target text document. Output 95 is a version of the target text document with the related terms located. Output 95 is displayed to the user. Also, for example, the smart search engine 72 accepts multiple query terms for use in a search via the search box 70. The smart search engine 72 identifies medical classification codes that correspond to the received multiple query terms. Note that the multiple query terms are not identical to any of the keywords in the list of keywords.

[0130] The smart search engine 72 provides an interface between the natural language found in clinical texts and a given coding system such as ICD, SNOMED CT, or OPCS-4.

[0131] The method described above with reference to Figures 2 to 5 provides a method for data mining search terms related to concepts from clinical text and using these to augment the synonym list of one or more selected clinical coding systems. Data mining of terms is provided using the attention mechanism of a neural network trained for clinical code prediction. From the coding systems and the data mined terms, an automated thesaurus is created. Coding concepts may be linked to associated data mined terms.

[0132] In the embodiments described above with reference to Figures 2 to 5, a query term is entered by a user. The search circuitry 28 uses a thesaurus to obtain synonyms for the query term. The user may be provided with a list of synonyms and select one or more of the synonyms for a search. The methods of Figures 2 to 5 may be used to find all relevant matches in the text (e.g., synonyms and related terms) for the user's search term. If the search circuitry 28 can identify other terms related to the original term, the user may also search for matches to that term.

[0133] A method for search query expansion may be provided by the methods of Figures 2 to 5. Search query expansion comprises expanding original search terms to related search terms. Synonyms provided in a clinical coding system may be used for each search query expansion. However, the synonyms provided in the clinical coding system may not match well to informal or abbreviated terms used in clinical practice. By using a reminder mechanism from a model trained to classify text using clinical codes, additional synonyms may be obtained that may include informal or abbreviated terms.

[0134] In some embodiments, the search query is expanded to include relevant clinical codes in addition to synonyms. At stage 50 of the method described above with reference to FIG. 2, the user's query terms are matched to one or more coded concepts. In some embodiments, the coded concepts to which the query terms match are displayed to the user. The user may select a coded concept to perform a search that will deliver all target text documents classified under that coded concept.

[0135] An example of a user interface for search query expansion is shown in Figure 6. A user types a query term into a search box 70. In the embodiment of Figure 6, the search term is "high b".

[0136] The search circuitry 28 uses the thesaurus to suggest a list of terms 74. The search circuitry 28 determines the terms 76 that are most similar to the query terms provided by the user and displays the most similar terms 76 to the user. In the embodiment of FIG. 6, the most similar term 76 is "high bp". In other embodiments, the step of determining the most similar terms 76 is omitted and the query terms are used in the search.

[0137] The search circuitry 28 determines a list of synonyms 74 for the most similar terms 76. The search circuitry 28 displays the list of synonyms 74 to the user for selection as search suggestions. In other embodiments, the query terms 76 are part of a thesaurus and the search circuitry 28 determines a list of synonyms for the query terms 76.

[0138] In the embodiment of FIG. 6, the search circuitry 28 also displays the number of target text documents in which each term was found. This may be referred to as the number of hits in the patient's medical system. In some embodiments, the synonyms are ranked by the number of hits. For example, synonyms with a high number of hits may be ranked higher in the list 74 than synonyms with a low number of hits. In other embodiments, more important search terms may be displayed first. Methods for determining the relative importance of search terms are discussed below with reference to FIG. 7.

[0139] The user may select to review results from the suggested search terms 74. The search circuitry 28 displays an option 78 for the user to select all synonyms. If the user selects "select all" 78, hits for all of the suggested search terms are displayed.

[0140] In the embodiment of FIG. 6, the search circuitry 28 also displays a list 96 of clinical codes associated with the most similar terms 76. The clinical codes are the clinical codes determined to be associated with the most similar terms 76 in stage 50 of FIG. 2. In other embodiments, the clinical codes are the clinical codes directly associated with the query terms. The search circuitry 28 may display the closest match first. The closest match may be referred to as the primary code.

[0141] In some embodiments, rather than searching by raw terms, the user may choose to select a suggested code from a list of clinical codes 96. The user may choose to select one of the suggested codes by typing the code number directly into search box 70 or by clicking on the displayed code. The display of clinical codes 96 may enable the user to search by concept. For example, a tree structure may be displayed to allow the user to navigate among related clinical codes. In some embodiments, coding concepts that do not form part of an existing clinical coding system are displayed. The user may navigate by such coding concepts.

[0142] In some embodiments, when a user selects a code, the search circuitry 28 displays all documents classified to that code. In some embodiments, when a user selects a code, the search circuitry 28 displays all terms associated with that code in the thesaurus. The number of target text documents that contain each term associated with the code may be displayed. One or more terms may be selected by the user. The user may navigate the coding system to find more hits.

[0143] In some embodiments, the search results are associated with a ranking based on an individual importance score for each text term of each concept. The importance score for each text term may be derived from the term's frequency in the attention vector, an average attention weight, or an association metric. Figure 7 shows a synonym list 64 for the ICD-10 code 4 shown in Figure 3. The synonym list 64 is augmented with a set of importance scores 98 for the text terms in the synonym list 64.

[0144] To obtain the importance scores of Fig. 7, we use a method considered to comprise an adaptation of the classical TF-IDF (term frequency-inverse document frequency) metric. The method uses the attention vector for the training corpus determined in stage 38 of the method of Fig. 2. N is the total number of documents in the training corpus.

[0145] For a text term associated with a concept c, the weight Wt,c of a term t in concept c is given as:

[0146] Wt,c=CFt,clog(N / CFt) (2)

[0147] CFt,c is the number of occurrences of term t for concept c in the attention vector for the training corpus. In some situations, term t may occur in the attention vector associated with concept c in all subsets of training documents that were classified by the trained machine learning model to concept c. In other situations, term t may occur in the attention vector associated with concept c in some subset of training documents that were classified to concept c.

[0148] CFt,c is the number of concepts associated with term t. Some terms are associated with a large number of concepts. By taking into account the number of concepts associated with a given term t, the importance score weights uncommon words more heavily than common words. In some situations, uncommon terms may better identify concepts.

[0149] Using the above approach, importance scores are assigned to all terms t associated with a given concept c. For example, in the case of ICD-10 code 4, importance scores are associated with each of term A, term B, term E, and term F.

[0150] Once a importance score has been obtained for each term, the terms are ranked by importance score. For example, a list of synonyms 74 may be ordered so that terms with higher importance scores are higher in the list. Normalizing the importance scores may make it possible to compare importance scores obtained for different concepts.

[0151] In the method of Figure 7, the ranking method is derived from the frequency of the terms in the attention vector: the number of times a text term t occurs in the attention vector of a concept c is used to determine the importance score.

[0152] In other embodiments, the importance score may be based on the attention weighting given to the text term t in each attention vector. For example, an average attention weighting for the text term t across all attention vectors may be determined. Terms with higher attention weightings are given higher importance scores and are ranked higher.

[0153] In further embodiments, any suitable metric may be used to determine and thereby rank the importance scores.

[0154] The trained model can also be further trained to classify the medical text document into other medical classification codes different from the medical classification code. For at least one of the text terms included in the medical text document, a first importance score associated with the other medical classification code and a second importance score associated with the other medical classification code can also be obtained. For example, in the development, the matching with the code and the ranking of the results can leverage the importance score between each term and each concept based on the average attention weight and / or the frequency observed in the training data. Also, for example, the results can be ordered according to the importance score.

[0155] Figure 8 is an example of results returned from a search on the list of synonyms shown in Figure 6. This search uses a different ranking methodology than that described above with reference to Figure 7.

[0156] The user selects "select all." A list of results is returned. In the example of Figure 8, the original search term "high bp" does not yield any hits in the retrieved documents, but a search for synonyms returns several potentially related hits 200, 202, 204, 206, 208. Hits 200, 202, 204, 206, 208 are ranked according to the importance scores learned for the related conceptual text terms, with the original query terms first (if present). Arrows 210 indicate the ranking.

[0157] In the example of Figure 8, the first hit 200 on "SBP" actually relates to spontaneous bacterial peritonitis, not systolic blood pressure. In the second hit 202, the blood pressure relates to a family member of the patient, not the patient himself.

[0158] To use clues from the context, instead, the user can perform a concept-based search. FIG. 9 shows an example of a search with selected concepts such as, for example, R03.0 Elevated blood-pressure reading, without diagnosis of hypertension <R03.0 High blood pressure measurement without diagnosis of hypertension>. FIG. 9 shows a list of results returned from the concept-based search. The returned results are instances indexed by the concept R03.0 Elevated blood-pressure reading, without diagnosis of hypertension <R03.0 High blood pressure measurement without diagnosis of hypertension>. Results 204, 206, 202 are returned. Arrow 212 represents the ranking order. Note that result 202, which mentions the family history regarding high blood pressure <hypertension>, remains as a relevant result. However, its rank in the list has dropped. Unrelated hits and mentions of hypertension <hypertension> have been removed.

[0159] FIG. 10 shows a schematic diagram of a connection formed between two clinical coding systems using the method of the embodiment. When the synonym lists of the two clinical coding systems share many terms, the clinical codes may likely be related. For example, when the synonym lists of two clinical codes from different clinical coding systems share many terms, the two clinical codes may represent the same concept. New connections may be formed within or between coding systems.

[0160] In some embodiments, the data mining circuit 26 determines possible connections that can occur between a first clinical code of a first clinical coding system and a second clinical code of a second clinical coding system. The possible connections are based on the number of synonyms shared by the synonym list of the first clinical code and the synonym list of the second clinical code.

[0161] The determination of possible connections may be based on an importance score associated with each term in the synonym list, with terms having higher importance scores being considered more heavily when determining possible connections.

[0162] Figure 10 illustrates ICD-10 codes 2 and 4, which are shown in Figure 3. Figure 10 also illustrates ICD-10 codes 5 and 6. ICD-10 code 6 has a synonym list 66 that includes term A, term B, term C, and term D.

[0163] As shown by the arrows in FIG. 10, the concepts with ICD-10 codes 4, 5, and 6 are subconcepts of the concept with ICD-10 code 2.

[0164] FIG. 10 also shows two OPCS-4 codes, OPCS-4 code 1 (shown at reference numeral 101) and OPCS-4 code 2 (shown at reference numeral 102). The concept of OPCS-4 code 2 is a subconcept of OPCS-4 code 1. OPCS-4 code 2 has a synonym list 103 that includes term A, term B, and term C. Because term A, term B, and term C are shared in synonym list 66 for ICD-10 code 6 and in synonym list 103 for OPCS-4 code 2, data mining circuitry 26 determines that the concept of ICD-10 code 6 and the concept of OPCS-4 code 2 may be related. A possible connection between ICD-10 code 6 and OPCS-4 code 2 is shown in FIG. 10 by dashed arrow 104.

[0165] Connections between codes may be extended to related codes. For example, if ICD-10 code 6 is related to OPCS-4 code 2, it may also be considered related to OPCS-4 code 1. If OPCS-4 code 2 is related to ICD-10 code 6, it may also be considered related to ICD-10 code 2.

[0166] In some embodiments, the coding systems are exposed to the user. The user may perform direct navigation through the codes. By determining possible connections between different coding systems, the user may be able to navigate more complex unified graphs. Modeling of the codes and relationships between concepts may be done as automatically as possible with minimal expert input.

[0167] Additional stages may be added to the flowchart of FIG. 2. For example, additional stages may be added between stages 42 and 44. In the additional stages, the data mining circuitry 26 augments the coding system with proposed new connections or new concepts. The data mining system may perform cross-mapping between ontologies or coding systems. In some embodiments, codes from an existing coding system, such as ICD-10, may be mapped to other codes, such as codes used internally by a hospital or other institution.

[0168] In some embodiments, multiple models are trained to classify documents into a given clinical code. For example, the models may be initialized with different random weights. Different models may return different attention vectors for a given code. For some codes, different attention vectors pick out different concepts or different parts of a concept. In some embodiments, the difference in attention vectors is used to decompose a concept into two or more constituent parts. The two or more constituent parts may represent different sub-concepts.

[0169] The data mining circuitry 26 may be further trained to classify the medical text document into another medical classification code different from the medical classification code. The processing device 22 can use the data mining circuitry 26 to obtain a first importance score associated with the other medical classification code and a second importance score associated with the other medical classification code for at least one of the text terms included in the medical text document.

[0170] FIG. 11 shows ICD-10 code 6 with two sub-concepts, ICD-10 code 6a and ICD-10 code 6b. An example of an ICD-9 code is 812.0 Other Motor Vehicle Accidents Related to Crash. Note that the two sub-concepts, ICD-10 code 6a and ICD-10 code 6b, are examples of other medical classification codes that are different from ICD-10 code 6.

[0171] Different attention vectors produce results that focus on different parts of the concept of code 821.0. As an example, a first attention vector may give more importance to words related to car crashes, such as "vehicle", "car", "speed", "driver", and "unrestrained". A second attention vector trained on the same text for the same code may give more importance to words related to patient symptoms associated with car crashes, such as "fracture", "contusion", and "bone". The first and second attention vectors may be used to identify corresponding first and second sub-concepts of the concept of code 821.0.

[0172] In some embodiments, known relationships between concepts are used. A common reminder mechanism may be used for multiple concepts. Related codes may learn at least some of the same terms. For example, a parent code, a parent's parent code, and a child code may all learn at least some of the same terms.

[0173] In some embodiments, a trained model is used to index documents using highly weighted terms in each of the code concepts. A search by code will retrieve indexed instances. A search by code may refine instances found by the search string. This approach may allow for negation, uncertainty, laterality, and / or coreference analysis.

[0174] 12 is a flow chart showing an outline of an indexing method according to an embodiment of the present invention. The medical image processing apparatus 10 is configured to perform the indexing method of FIG.

[0175] Stages 30 through 36 of the method of Figure 12 are the same as stages 30 through 36 of the method of Figure 2. At stage 30, training circuitry 24 receives a training corpus. At stage 32, training circuitry 24 receives ground truth clinical codes for the training corpus. At stage 34, training circuitry 24 trains a machine learning model, e.g., a neural network, to classify documents using a given set of clinical codes. At stage 36, training circuitry 24 outputs the trained machine learning model. Indexing circuitry 29 receives the trained machine learning model.

[0176] 12, at stage 110, the indexing circuitry 29 receives new input documents from the data store 20. In other embodiments, the new input documents may be received from any suitable data store, such as, for example, a data store forming part of an electronic medical record system or a PACS.

[0177] A new input document is a text document to be indexed. The new input document may be a document that is not part of the training corpus. The new input document may be any medical text document, for example, a clinician's discharge summary.

[0178] At stage 112, the indexing circuit 29 inputs the new input document to the trained machine learning model. The trained machine learning model classifies the new input document into at least one clinical code. The trained machine learning model outputs a separate attention vector for each clinical code into which the new input document was classified. The attention vector comprises a separate attention weighting for each text term of the new input document, e.g., for each word or group of words of the new input document.

[0179] As noted above, different instances of a given text term (e.g., different instances of a word or group of words) may be given different attention weights by the trained machine learning model. The attention weight for a given instance may depend on the context of that instance.

[0180] In stage 116, the indexing circuitry 29 determines which of the text terms of the new input document received a high attention weighting for each clinical code. For example, the indexing circuitry 29 may compare the attention weighting to a threshold. By determining which text terms received a high attention weighting, the indexing circuitry 29 determines which text terms were important in classifying the new input document into a clinical code. That is, the indexing circuitry 29 determines a set of index terms for the medical text document based on the weightings in classifying the new input document into a clinical code.

[0181] In stage 118, the indexing circuitry 29 indexes new input documents using the text terms found to have high attention weights.

[0182] 13 shows an example of indexing: A new input document 120 is input to a trained machine learning model 122. The trained machine learning model 122 classifies the new input document 120 into ICD-10 code 2 and ICD-10 code 4.

[0183] The indexing circuit 29 determines a list of instances of the term that have been given a high attention weighting by the trained machine learning model 122. A first list of instances 132 is associated with ICD-10 code 2. A second list of instances 134 is associated with ICD-10 code 4.

[0184] The first list of instances 132 comprises: Term A: Instance 1 Term B: Instance 1 Term B: Instance 2 Term B: Instance 3

[0185] 3 includes a synonym list 62 for ICD-10 code 2. Synonym list 62 includes term A, term B, term C, and term D.

[0186] When the trained machine learning model is deployed to new input documents, the new input documents may not contain some of the text terms in the thesaurus. Instead, the new input documents may contain instances of text terms that are not relevant or should be less relevant to the correct classification of the new input document. For example, the text terms may be included as negations, or the text terms may relate to persons other than the patient.

[0187] A trained machine learning model may make a decision to classify a new input document as ICD-10 code 2 based on instances of some, but not all, of the synonyms of ICD-10 code 2. Only some of the synonyms may be given a high attention weight. Only some instances of those synonyms may be given a high attention weight. The attention weights may be context dependent.

[0188] 13, it can be seen that only terms A and B are significant in classifying new input document 120 into ICD-10 code 2. Indexing circuitry 29 indexes new input document 120 with terms A and B. Indexing circuitry 29 does not index new input document 120 with either term C or term D.

[0189] A second list of instances 134 associated with ICD-10 code 4 comprises: Term A: Instance 1 Term B: Instance 1 Term B: Instance 2 Term B: Instance 3 Term E: Instance 1 Term E: Instance 2 Term E: Instance 3 Term E: Instance 4

[0190] 3 comprises a synonym list for ICD-10 code 4. Synonym list 64 includes term A, term B, term E, and term F.

[0191] 13, it can be seen that terms A, B, and E are important in classifying new input document 120 into ICD-10 code 4. Indexing circuitry 29 indexes new input document 120 with terms A, B, and E. Indexing circuitry 29 does not index new input document 120 with term F.

[0192] The use of trained machine learning for document indexing, as described above with reference to Figures 12 and 13, may provide an intelligent indexing method. Searching by clinical code may be improved such that individual terms can be identified and located within documents according to the importance of each instance of the term in the classification process. In contrast to traditional methods, the above-described machine learning model may take advantage of contextual clues and return a targeted list of results by indexing instances from the output of the trained machine learning model.

[0193] The indexing process of Figures 12 and 13 is performed using the trained machine learning model itself, whereas the search phase of Figure 3 is performed using thesaurus 60 and does not use the trained machine learning model. In some circumstances, a GPU is used to index documents by coding concepts in the deployment of the indexing method of Figures 12 and 13. When performing document search using thesaurus 60, a GPU may not be required.

[0194] In some embodiments, the machine learning model continues to be trained. For example, the system may continue to be trained according to codes that are routinely assigned to clinical documents at an institution, such as a hospital. Repeated or continued training may provide opportunities for domain transfer and / or personalization to the institution.

[0195] 14 illustrates a simple user interface 140 for continuous training of a machine learning model. The user interface 140 is a smart search training configuration. In other embodiments, any suitable user interface may be used. The content and / or layout of the user interface may differ from that shown in FIG.

[0196] In the embodiment of Figure 14, a trained machine learning model is received by an institution. The trained machine learning model was trained using data from a different institution or group of institutions. A user wishes to further train the machine learning model with local data from the institution.

[0197] A first portion 142 of the user interface 140 relates to one or more coding systems for training the machine learning model. The first portion 142 comprises a list of selectable coding systems. In the embodiment of FIG. 14, the coding systems are ICD, SNOMED-CT, OPCS-4, and SMR. The coding systems can be selected by clicking on a box corresponding to each coding system. In other embodiments, any suitable coding system and / or selection method may be used. The user selects at least one coding system from the list of coding systems.

[0198] Element 144 of user interface 140 allows a user to upload a new code. For example, the code uploaded by the user may be a local code that is not part of any of the coding systems in the list of coding systems.

[0199] A second portion 150 of the user interface 140 relates to the data to use for training. A selectable date range 152 allows the user to select a date range of data for training the machine learning. In the illustrated example, the selected date range is from 2015 to 2018. A start date and an end date are selectable via drop down menus. In other embodiments, any suitable date input may be used. The user selects the desired date range.

[0200] The second portion 150 further comprises a list 154 of selectable document types. In the embodiment of FIG. 14, the document types are EMR discharge summaries and PACS radiology reports. The document types may be selected by clicking on a box corresponding to each document type. In other embodiments, any suitable document type and / or selection method may be used. The user selects at least one document type.

[0201] Element 156 of user interface 140 allows a user to upload a new coded document. The new coded document may comprise a document that is not included in the document types listed in the document types list and / or a document that is not included in the selected date range.

[0202] Element 160 of user interface 140 allows a user to initiate training. A user selects element 160 when they want to begin training a machine learning model with a selected coding system, date range, and document type. The machine learning model is trained using the selected coding system, date range, and document type. An updated machine learning model is output.

[0203] By training a machine learning model with data from a particular institution, the machine learning model may be tailored to that institution. The performance of the machine learning model may be improved.

[0204] In some situations, the machine learning model may be trained repeatedly, e.g., periodically. By repeatedly training the machine learning model, the machine learning model may be periodically updated with new training data. The updated machine learning model may adapt to changes in practice, such as changes in clinical note-taking or coding, that occur over time.

[0205] One embodiment provides a method for medical text analysis, in which a neural network model is trained on clinically coded documents using an attention mechanism, where the codes are part of a medical coding system such as ICD or SNOMED CT. The method data mines highly weighted terms in the attention vector for each clinically coded concept; it takes the data mined terms and links them to concepts in the medical coding system to create a thesaurus.

[0206] The method may additionally comprise using weights and the frequency with which terms appear in the neural network attention vector to rank terms by importance for each coded concept.

[0207] The method may additionally comprise indexing documents with highly weighted terms for each coded concept.

[0208] The indexing may be used to retrieve search results and may allow users to search by medical concepts rather than by terms.

[0209] The method may additionally comprise suggesting new connections between concepts within a coding system or between coding systems, and enabling automatic cross-mapping between coding systems.

[0210] The method may additionally comprise suggesting sub-concepts of an existing medical concept, which sub-concepts may also be searchable.

[0211] The method may additionally comprise allowing a user to perform personalized training specific to the institution. The user may be able to control what data and codes the system is trained on.

[0212] An embodiment provides a medical text search apparatus that includes a processing circuit configured to receive a query for search, identify a medical classification code to which the query belongs, and find medical text that includes keywords that are not identical to the query but that belong to the identified medical classification code.

[0213] The processing circuitry may be further configured to find the medical text based on a frequency of keywords contained in the identified medical classification code.

[0214] The processing circuitry may be further configured to receive threshold information regarding the frequency and to locate the medical text based on the received threshold information.

[0215] The processing circuitry may be further configured to identify a plurality of medical classification codes to which the query belongs.

[0216] The processing circuitry may be further configured to receive a plurality of queries.

[0217] One embodiment provides a method for medical text analysis, comprising: training a model on a training set of clinically coded documents, each clinically coded document comprising or associated with at least one label comprising one or more codes of a clinical coding system. The training comprises generating, for each code of the clinical coding system, a set of associated text terms extracted from the training set of clinically coded documents. The method further comprises using the generated set of text terms in a process for searching or indexing additional medical documents.

[0218] The model might be a neural network.

[0219] Such training may comprise assigning individual relevance or importance scores to the extracted text terms, and optionally using the relevance or importance scores for subsequent searching or indexing.

[0220] For at least one of the text terms, the text term may have a different relevance or importance score in its use in association with a first code in the codes of the clinical coding system than in its use in association with a second code in the codes of the clinical coding system.

[0221] The set of extracted text terms may comprise natural language words or phrases to supplement the more formal clinical terminology of a clinical coding system.

[0222] Using the generated set of text terms in a process of searching or indexing additional medical documents may comprise receiving search terms from a user via a user interface and suggesting additional search terms and / or clinical codes, or performing a search using additional search terms or clinical codes based on the received search terms and the generated set of text terms.

[0223] The method may comprise matching the received search terms to one or more terms in the generated set of text terms and associated clinical codes, and suggesting further search terms from the generated set of text terms and associated clinical codes based on the matching.

[0224] Using the generated set of text terms in a process of searching or indexing further medical documents may comprise indexing the medical documents with one or more of the text terms and / or with one or more of the codes based on the content of the medical documents and the generated set of text terms.

[0225] The clinical coding system may include a synonym list associated with a clinical code, and the training may include generating additional synonyms for the synonym list.

[0226] The training may comprise generating additional codes or concepts for use in indexing or searching to supplement the codes of the clinical coding system.

[0227] The method may further comprise creating connections between codes of the clinical coding system based on the generated set of text terms and using the connections when searching, indexing, or representing the coding system.

[0228] The method may further comprise creating a sub-code or sub-concept for the code of the clinical coding system based on the generated set of text terms.

[0229] The clinical coding system may include an ICD or SNOMED CT system.

[0230] Although particular circuits are described herein, in alternative embodiments, the functionality of one or more of those circuits may be provided by a single processing resource or other component, or the functionality provided by a single circuit may be provided by a combination of two or more processing resources or other components. Reference to a single circuit encompasses multiple components that provide the functionality of that circuit, whether or not such components are separate from one another. Reference to multiple circuits encompasses single components that provide the functionality of those circuits.

[0231] According to at least one of the embodiments described above, it is possible to improve the efficiency of searching or indexing medical text documents.

[0232] While certain embodiments have been described, these embodiments are presented for illustrative purposes only and are not intended to limit the scope of the invention. In fact, the novel methods and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the spirit of the invention. The appended claims and their equivalents are intended to cover such forms and modifications as fall within the scope of the invention. [Explanation of symbols]

[0233] 10 Medical information processing device 12 Computing Equipment 16 display screens 18 Input Devices 20 Data storage unit 22 Processing equipment 24 Training Circuit 26 Data Mining Circuit 28 Search Circuit 29 Indexing Circuit

Claims

1. a processor for obtaining weightings of text terms contained in at least one medical text document in association with said medical classification codes using a trained model trained to classify medical text documents into medical classification codes, and for performing at least one of a search process and an indexing process using said weightings; Equipped with The processing unit includes: Using the weighting, obtain a list of keywords associated with the medical classification code; Obtaining a list of the keywords associated with the medical classification code; identifying text portions in at least one further medical text document that correspond to keywords in said list of keywords; In at least one of obtaining the list of keywords and identifying the text portion, assigning an individual importance score to the obtained list of keywords or the identified text portion. Medical information processing equipment.

2. The trained model comprises an attention mechanism; The weighting is an attention weighting. The medical information processing device according to claim 1 .

3. The processing unit performs at least one of obtaining a list of the keywords and identifying the text portion based on a frequency at which the keywords are included in medical text documents classified into the medical classification code. The medical information processing device according to claim 1 .

4. The processing unit performs at least one of obtaining the list of keywords and identifying the text portion based on a threshold or an importance score based on the weighting. The medical image processing device according to claim 1 .

5. The processing unit uses the trained model to identify a plurality of medical classification codes to which a query term used in the search process belongs. The medical image processing device according to claim 1 .

6. The processing unit: identifying the medical classification codes corresponding to a plurality of query terms used in the search process; the plurality of query terms are not identical to any of the keywords in the list of keywords; The medical image processing device according to claim 1 .

7. The processing unit: if the list of keywords is obtained, ranking the keywords in the list of keywords according to the importance score; if the text portions are identified, ranking the text portions according to the importance scores; The medical image processing device according to claim 1 .

8. The trained model is further trained to classify the medical text documents into other medical classification codes different from the medical classification codes; obtaining a first importance score associated with the other medical classification code and a second importance score associated with the other medical classification code for at least one text term included in the medical text document; The medical image processing device according to claim 1 .

9. The search process includes at least one of a search by concept and a search by text term. The medical image processing device according to claim 1 .

10. The medical classification code forms part of a first clinical coding system; the processor creates at least one connection between the first clinical coding system and a different second clinical coding system. The medical image processing device according to claim 1 .

11. The processing unit creates at least one sub-classification code of the medical classification code. The medical image processing device according to claim 1 .

12. The processing unit: determining a set of index terms for the medical text document based on the weightings; performing the indexing process using the determined set of index terms; The medical image processing device according to claim 1 .

13. The processing unit: ranking the text terms according to the weightings; determining the set of index terms using the ranked text terms; The medical information processing device according to claim 12.

14. The trained model is trained using a set of training documents classified into ground truth medical classification codes. The medical image processing device according to claim 1 .

15. The trained model is updated with new training data based on an institution, at least one coding system, and at least one date range. The medical image processing device according to claim 2 .

16. Obtaining weights of text terms contained in at least one medical text document in association with said medical classification code using a trained model trained to classify medical text documents into medical classification codes; performing at least one of a search process and an indexing process using the weighting; Using the weighting, obtain a list of keywords associated with the medical classification code; Obtaining a list of the keywords associated with the medical classification code; identifying text portions in at least one further medical text document that correspond to keywords in said list of keywords; In at least one of obtaining the list of keywords and identifying the text portions, assigning an individual importance score to the obtained list of keywords or the identified text portions; A medical information processing method comprising:

17. On the computer, Obtaining weights of text terms contained in at least one medical text document in association with said medical classification code using a trained model trained to classify medical text documents into medical classification codes; performing at least one of a search process and an indexing process using the weighting; Using the weighting, obtain a list of keywords associated with the medical classification code; Obtaining a list of the keywords associated with the medical classification code; identifying text portions in at least one further medical text document that correspond to keywords in said list of keywords; In at least one of obtaining the list of keywords and identifying the text portions, assigning an individual importance score to the obtained list of keywords or the identified text portions; A medical information processing program that makes this possible.

18. a processor for obtaining weightings of text terms contained in at least one medical text document in association with said medical classification codes using a trained model trained to classify medical text documents into medical classification codes, and for performing at least one of a search process and an indexing process using said weightings; Equipped with The trained model is updated with new training data based on an institution, at least one coding system, and at least one date range. Medical information processing equipment.

19. Using a trained model trained to classify medical text documents into medical classification codes, weights of text terms contained in at least one medical text document are obtained in association with said medical classification codes; performing at least one of a search process and an indexing process using the weighting; The trained model is updated with new training data based on an institution, at least one coding system, and at least one date range. Medical information processing method.

20. A computer comprising: Obtaining weights of text terms contained in at least one medical text document in association with said medical classification code using a trained model trained to classify medical text documents into medical classification codes; performing at least one of a search process and an indexing process using the weighting; The trained model is updated with new training data based on an institution, at least one coding system, and at least one date range. Medical information processing program.

Images