Fingerprinting Unstructured Patient Data

Abstract

A convolutional neural network (CNN) for use in a medical workflow is generated by receiving a diagnostic indication, unstructured patient data (UPD), and a diagnostic report vocabulary list having ordered vocabulary terms, generating a blank bitmap having a plurality of pixels with (i) a number of pixels corresponding to an amount of the vocabulary terms and (ii) a correspondence between pixel positions in the bitmap and an order of the vocabulary terms, generating a fingerprint with a mapping of each occurrence of the vocabulary terms in the UPD to a corresponding pixel position in the blank bitmap, and training a convolutional neural network (CNN) using the fingerprint as an input and the diagnostic indication as a target output.

Description

[Technical Field]

[0001] The present invention relates to fingerprinting unstructured patient data. [Background technology]

[0002] The use of digital technology in radiology has led to a significant increase in the amount of clinical images and non-imaging information used by medical professionals for diagnosis. Despite this increase, clinical evaluation is still performed using traditional methods. This mismatch leads to a significant increase in medical professionals' workload and data overload. Artificial intelligence (AI) methods using deep learning analysis with convolutional neural networks (CNNs) are being used more and more frequently to partially address the aforementioned issues in this field. Summary of the Invention [Problem to be solved by the invention]

[0003] While AI offers one potential means of solving the workload problem, patient context (e.g., non-imaging information such as patient history and previous diagnostic reports) remains a critical component of reliable diagnosis. Finding the right patient information relevant to a specific clinical situation remains a time-consuming task.

[0004] While AI can be used to find the right patient information, unstructured patient data (UPD) must first be converted into a structured format using natural language processing (NLP) or other methods. This is not a trivial task, and automated NLP techniques are still in their infancy. Errors made by NLP during data structuring can create a cascade of new problems when the structured data is fed into an AI engine. Given current global healthcare trends (e.g., declining reimbursements, value-driven healthcare policies, global medical staff shortages, and high rates of sick leave due to work stress), improvements to this data flow and structuring are urgently needed. [Means for solving the problem]

[0005] Some exemplary embodiments relate to a method of receiving a diagnostic indication, unstructured patient data (UPD), and a diagnostic report vocabulary list, the diagnostic report vocabulary list consisting of ordered vocabulary terms, generating a blank bitmap including a plurality of pixels having (i) a number of pixels corresponding to an amount of vocabulary terms and (ii) a correspondence between pixel locations in the bitmap and an order of the vocabulary terms, generating a fingerprint including a mapping of each occurrence of the vocabulary terms in the UPD to a corresponding pixel location in the blank bitmap, and training the fingerprint using a convolutional neural network (CNN) as an input and the diagnostic indication as a target output.

[0006] Another example embodiment relates to a method for receiving unstructured patient data (UPD) and a diagnostic report vocabulary list, the diagnostic report vocabulary list including ordered vocabulary terms, generating a blank bitmap including a plurality of pixels having (i) a number of pixels corresponding to an amount of the vocabulary terms and (ii) a correspondence between pixel positions in the bitmap and an order of the vocabulary terms, and generating a first fingerprint comprising a mapping of positions of each occurrence of the vocabulary terms in the UPD by modifying pixel values ​​associated with corresponding pixel positions in the blank bitmap; generating a first fingerprint comprising a mapping of positions of each occurrence of the vocabulary terms in the UPD to corresponding positions in the blank bitmap by modifying pixel values ​​associated with corresponding pixel positions in the bitmap, and displaying the fingerprint to a user.

[0007] Another example embodiment relates to a system for creating a data structure for use in clinical diagnostic support, the system having a memory containing a diagnostic indication, unstructured patient data (UPD), and a diagnostic report vocabulary list, the diagnostic report vocabulary list having ordered vocabulary terms. The system also has a processor configured to generate a blank bitmap having a plurality of pixels having (i) a number of pixels corresponding to an amount of the vocabulary terms and (ii) a correspondence between pixel positions in the bitmap and an order of the vocabulary terms. The processor is also configured to generate a fingerprint having a mapping of each occurrence of the vocabulary terms in the UPD to a corresponding pixel position in the blank bitmap, and to train a convolutional neural network (CNN) using the fingerprint as an input and the diagnostic indication as a target output.

[0008] Another example embodiment relates to a system for creating a data structure for use in clinical diagnostic support, the system having a memory containing unstructured patient data (UPD) and a diagnostic report vocabulary list, the diagnostic report vocabulary list having ordered vocabulary terms. The system also has a processor configured to generate a blank bitmap having a plurality of pixels having (i) a number of pixels corresponding to an amount of the vocabulary terms and (ii) a correspondence between pixel positions in the bitmap and an order of the vocabulary terms. The processor is also configured to generate a first fingerprint having a mapping of each occurrence of the vocabulary terms in the UPD to a corresponding position in the blank bitmap by modifying pixel values ​​associated with corresponding pixel positions in the bitmap. The system also has a display for displaying the fingerprint to a user.

[0009] Another example embodiment relates to a computer program product operable to perform the methods described herein when executed on a computer. [Brief explanation of the drawings]

[0010] [Figure 1] 1 illustrates an exemplary fingerprint reference matrix in accordance with various exemplary embodiments. [Figure 2] 1 illustrates an unstructured fingerprint in accordance with various exemplary embodiments. [Figure 3] 1 illustrates an unstructured fingerprint with frequently occurring entries labeled in accordance with various exemplary embodiments. [Figure 4] 1 illustrates a differential fingerprint according to various exemplary embodiments. [Figure 5] 1 illustrates an unstructured fingerprint with an interactive word cloud in accordance with various exemplary embodiments. [Figure 6] FIG. 1 is a flow diagram for training a convolutional neural network using unstructured patient data and corresponding diagnostic reports, according to various exemplary embodiments. [Figure 7]FIG. 1 illustrates a methodology diagram for training a convolutional neural network using unstructured patient data and corresponding diagnostic reports, according to various exemplary embodiments. [Figure 8] 1 illustrates a vectorized unstructured fingerprint in accordance with various exemplary embodiments. [Figure 9] FIG. 1 illustrates a flow diagram for the use of image-based convolutional neural networks with fingerprint-based convolutional neural networks for use in clinical decision support, according to various exemplary embodiments. [Figure 10] FIG. 1 illustrates a methodology diagram for the use of image-based convolutional neural networks with fingerprint-based convolutional neural networks for use in clinical decision support, according to various exemplary embodiments. [Figure 11] FIG. 1 is a schematic diagram of an exemplary system in accordance with various exemplary embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0011] Exemplary embodiments may be further understood with reference to the following description and associated accompanying drawings, where like elements are provided with the same reference numerals. Exemplary embodiments relate to the use of "fingerprints" to assist in both clinical and AI analysis of unstructured patient data (UPD).

[0012] As described above, the use of natural language processing (NLP) on UPD can result in many errors when an AI analyzes the resulting structured patient data. The exemplary embodiments provide an alternative data structure for the AI ​​to analyze. This alternative data structure is referred to as a "fingerprint" throughout this description. Fingerprints can be utilized for clinical diagnostic support. At a high level of abstraction, a fingerprint may be understood to be an image-based representation of relevant clinical terms (e.g., from a UPD source). It should be understood that the term fingerprint is used throughout this specification to refer to a data structure with certain characteristics as described herein. Thus, a fingerprint should be understood to be a data structure with characteristics as described herein.

[0013] The use of fingerprints along with corresponding clinical diagnostic reports can be used to train a convolutional neural network (CNN) using the fingerprints as input data and the structured findings in the clinical report as the desired output. The fingerprints may take the form of small grayscale image files (in terms of data usage) and can be analyzed without modification by existing AI platforms for image-related training. Once trained, the CNN can function as a "virtual NLP engine" and can be used to support image-based CNNs in a straightforward manner using diagnostic images and corresponding UPDs.

[0014] The term Virtual NLP is justified because AI networks can be trained to trigger on the co-occurrence of certain terms in patient data, or on certain terms in a particular order, if they are associated with consistent clinical diagnostic findings. In this way, the AI ​​may benefit from the analysis performed by medical professionals who created the diagnostic reports that were later used for AI training.

[0015] Another advantage of fingerprints is their small size (ideally less than 100 kilobytes, but larger sizes are possible if desired by the operator), which is independent of the amount of data they represent and allows for an efficient data format for storing essential aspects of UPD for use in NLP algorithms and longitudinal studies.

[0016] Fingerprints are useful for analyzing longitudinal studies. As described in more detail below, fingerprints utilize no search or indexing algorithms, instead relying solely on image subtraction to reveal any relevant differences between two large volumes of unstructured data. The use of image subtraction reveals only the terms that changed between any two given studies for a given patient. Fingerprints may also be used as an attractive and efficient user interface (UI) visualization for user interaction with large volumes of UPD, as described in more detail below.

[0017] To create a fingerprint in a clinical context, various information and operations may be performed. This information and operations may include, for example, dictionaries (e.g., medical dictionaries), extraction algorithms that create fingerprints from any source of unstructured text-based data, and an image-driven AI engine that converts fingerprints into diagnostic suggestions for review by clinicians. A large collection of UPD along with associated clinical diagnostic findings may be used to train such an AI engine.

[0018] Ideally, this information should be available longitudinally (i.e., various versions of the same case / study over time). The use of longitudinal UPD and associated clinical diagnostic findings may improve the efficiency of training AI. However, it should be understood that longitudinal data is not required.

[0019] As an example using longitudinal data, consider two different data reports for the same patient. A first data report may include patient data generated at time t3 and a most recent diagnostic report for a clinical condition diagnosed shortly after time t3. A second data report may include patient data generated at time t1 along with a diagnostic report generated shortly after time t1, and may also include information generated at time t2 along with a diagnostic report generated shortly after time t2, and may also include information generated at time t3 along with a diagnostic report generated shortly after time t3. The first data report provides less information than the second data report because the incremental aspect of the information is lost when all the data is bundled together.

[0020] The AI ​​engine may be operated in parallel with existing algorithms for image-based AI, integrated with such image-based AI engines (as fingerprints are ordinary digital images), or as a standalone application for the generation of diagnostic suggestions derived from unstructured patient data during diagnosis.

[0021] As mentioned above, a dictionary may be used to create the fingerprint. It should be understood that the term "dictionary" encompasses any other representative word list as well. In one example, a medical dictionary containing 98,119 words is used as the fingerprint reference, with a 120-page PDF file used as the UPD.

[0022] First, a 314 x 314 digital grayscale image of 1-byte pixels is created, resulting in a total file size of 98.6 kilobytes. 314 is the smallest integer whose square is greater than the number of words in the dictionary used as the fingerprint reference (314 2 =98,596) > 98,119)).

[0023] Next, each pixel of the grayscale image is assigned one unique word from the dictionary. This operation can be performed in several ways, but in this example, all pixels are filled with the words from the dictionary in alphabetical order from the top left to the bottom right of the grayscale image (i.e., reading order). It should be understood that other mappings between the dictionary and the grayscale image are possible.

[0024] FIG. 1 shows an exemplary fingerprint reference matrix 100 according to various exemplary embodiments. FIG. 1 depicts the top left corner of a 314×314 fingerprint reference matrix 100. The top left corner is filled with the first term in alphabetical order of the corresponding dictionary (in this case, "abasia"). The matrix is filled in alphabetical order from left to right, row by row, until the dictionary is exhausted.

[0025] The use of the fingerprint reference matrix enables the unique assignment of the nth word of the dictionary by the following formula, which indicates that the pixel of the 314×314 image corresponding to the nth word of the dictionary is assigned to the pixel at the ith column from the right and the jth row from the top. n=(j - 1)×314 + i, where 1 < i ≤ 314 and 1 < j ≤ 314

[0026] Using both the assignment formula and the fingerprint reference matrix, fingerprint creation is straightforward. Starting with a 314×314 image where all pixels are set to value 0 (i.e., a uniformly black image), a standard text reading algorithm reads each word from the unstructured dataset (the 120-page UPD PDF mentioned above). Each time a word read from the UPD exists in the fingerprint reference matrix, the corresponding pixel value of the fingerprint is incremented by 1. In this example, each pixel is bounded by a maximum value of 255, corresponding to the maximum value of 1 byte. This maximum value seems sufficient for all practical purposes, but it may be extended.

[0027] 2 illustrates an unstructured fingerprint 200 according to various exemplary embodiments. Of note in fingerprint 200 are the more prominent (brighter) pixels scattered throughout the image. The location of any given pixel corresponds to the nth term in the dictionary, which is converted row-by-row into a pixel grid, from top-left to top-right. The intensity of any given pixel corresponds to the frequency with which the nth term in the dictionary appears in the UPD. Thus, more frequently occurring terms correspond to brighter pixels up to and including the term that appears 255 times (the byte maximum) in the UPD.

[0028] Figure 3 illustrates an unstructured fingerprint 300 with frequently occurring entries labeled according to various exemplary embodiments. Figure 3 shows the same UPD as fingerprint 200 as shown in Figure 2, but with frequently occurring terms (lighter pixels) labeled with their corresponding dictionary entries. The threshold of what constitutes a frequently occurring entry may be defined by an operator (e.g., 10 times, 50 times, 100 times, etc.). Figure 3 illustrates an attractive means of visualizing the UPD for a user.

[0029] Note that a fingerprint can be an ambiguous identifier of a scanned UPD because it is possible to create identical fingerprints with different data. However, in a clinical context, this ambiguity is irrelevant because clinicians are interested in changes to the fingerprint (e.g., in longitudinal studies). This clinical need is met because any change in the number of associated terms (i.e., dictionary terms used to populate the fingerprint reference matrix) increases or decreases the brightness of the corresponding pixel in the fingerprint.

[0030] This property of fingerprints makes them attractive for use in longitudinal studies. Image subtraction of two fingerprints created on different dates creates a new, third differential fingerprint that reveals all relevant terms added on the second date. It should be appreciated that image subtraction can similarly reveal relevant terms that were deleted / removed from the second fingerprint.

[0031] FIG. 4 illustrates a difference fingerprint 400 according to various exemplary embodiments. A difference fingerprint may be created by pixel-by-pixel image subtraction of two fingerprints created at different times (Fingerprint 2 "F2" - Fingerprint 1 "F1"). FIG. 4 does not depict the two fingerprints (F1, F2) used for its creation, as the relevant aspect is the depicted change. Cursor analysis of difference fingerprint 400 reveals that F2 has data entered related to an acute sudden myocardial infarction diagnosed on a delayed gadolinium enhancement (LGI) magnetic resonance imaging (MRI) scan. It should be understood that the changes occurring in F2 occurred after the creation of F1.

[0032] Figure 4 illustrates the value fingerprints have for longitudinal studies. The small data size of fingerprints (less than 100 kilobytes, regardless of the amount of UPD they represent) allows for efficient visualization of changes in UPD. Changes in a patient's chart can be quickly compared for any two fingerprints taken at different times.

[0033] The visualization and search capabilities of fingerprints can be enhanced by deep links. A deep link can be understood as a connection between the exact location in the UPD that corresponds to a given pixel in the fingerprint. As an example, if the term "lymphoma" appears 117 times in the UPD (corresponding to the pixel having the value 117), a user can hover their mouse (or any other suitable interaction device) over the corresponding pixel in the fingerprint and be presented with a list or directory of other locations where the term "lymphoma" appears directly in the UPD. This list or directory may include an exact link to the location in the UPD (the specific page or line where "lymphoma" appears) or simply to the document (the entire report).

[0034] 98,696 pieces (314 2 Manually selecting a single pixel from a grid of pixels (such as a 3D image) can be a challenging exercise in dexterity. To account for this, the labeling system shown in FIGS. 3-4 may be further enhanced as a "word cloud." The labeling system may also feature deep linking, as described above. From a UI perspective, the labeling system may increase the font size of a particular label based on the corresponding pixel value (which itself corresponds to the number of terms in the UPD). Pixels with values ​​below a specified threshold may not be labeled to reduce visual clutter. It is also possible that labels may be colored or otherwise indicated based on the corresponding term (e.g., the clinical concept associated with the term). For example, all cardiac terms may be colored red, all tumor terms may be colored green, etc. It should be understood that any combination of label font sizes and coloring schemes is possible based on the needs of the operator.

[0035] FIG. 5 illustrates an unstructured fingerprint 500 with an interactive word cloud, according to various exemplary embodiments. In this example, the fingerprint 500 includes specific terms with increased size based on their corresponding pixel values. A color-coding scheme can also be utilized to group terms in specific categories. A user may mouse over non-zero (i.e., non-black) pixels / labels to see a preview of the corresponding UPD file. A user may click on any of the non-zero-valued pixels, which correspond to deep links to the term's source. Upon mouse click, the user is presented with all documents containing the corresponding term, with the corresponding term in the document highlighted for easy reference. While adding this functionality to the fingerprint significantly increases file size, those skilled in the art will recognize the value of simplified visualization of the UPD.

[0036] As discussed above, fingerprints can be utilized with AI methods to use unstructured data as a source of additional information for clinical diagnosis, for example, to support traditional or AI-assisted clinical image analysis. As discussed above, the use of NLP to convert UPD into structured data often introduces errors that reduce or eliminate any efficiency gains of feeding structured data to an AI engine. The use of fingerprints can eliminate these types of errors by eliminating the data structuring step of UPD.

[0037] Instead of using structured patient data, a fingerprint of the UPD can be used along with the corresponding clinical diagnostic report to generate a CNN. The fingerprint is used as input data, and the structured findings in the clinical report are used as the desired output. Thus, the CNN is trained to trigger on the co-occurrence of specific words, or specific words in a specific order, in the patient data with the results of relevant and consistent clinical diagnoses previously concluded by healthcare professionals. Once trained, the CNN can operate as a "virtual NLP engine" and can be used to support image-based CNNs in a straightforward manner, using diagnostic images and the corresponding UPD in the form of a fingerprint.

[0038] 6 illustrates a flow diagram 600 for training a convolutional neural network using unstructured patient data and corresponding diagnostic reports, according to various exemplary embodiments. The UPD 605 may be unstructured patient text data (e.g., test results, clinical notes, etc.) that has been NLP-generated into an AI-scannable format.

[0039] The diagnostic report vocabulary 610 may be a medical dictionary or any other representative list of words stored in a matrix (e.g., the example of FIG. 1) that can be mapped to fingerprint images from text analysis of the UPD.

[0040] The diagnostic indication 615 may be a clinical finding, recommendation, or conclusion made by a medical professional. The diagnostic indication 615 should be understood to be a separate entity from the unstructured patient data 605. The value of the diagnostic indication 615 is that a human medical professional has reached a finding, recommendation, or conclusion, which may enhance the capabilities of the CNN.

[0041] At 620, a patient fingerprint is created. The UPD 605 is scanned with a standard text reading algorithm. Each time a word that appears in the dictionary 610 matrix is ​​found by the text reading algorithm, the intensity of a single pixel in the 314x314 grayscale grid is increased by 1 / 255. As described above, the pixel's location corresponds to the grid location of the found word in the dictionary 610 matrix. The nth word in the dictionary may be located by the following formula: n=(j-1),314+i,1 <i≦314,1<j≦314 This indicates that the nth word in the dictionary is assigned to the pixel in the ith column from the right and the jth row from the top. The intensity of each pixel corresponds to the number of times the word appears in the UPD 605, up to a maximum of 255.

[0042] At 630, a CNN is trained using the fingerprint 620 as input and the diagnostic indication 625 as the desired output. The CNN 630 can be trained to trigger on the co-occurrence of specific words in specific sequences within patient data, or specific words in specific orders, with the results of relevant and consistent clinical diagnoses previously concluded by human medical professionals. As a result, once trained, the CNN can operate as a "virtual NLP engine" and can be used to directly support image-based CNNs using diagnostic images and corresponding unstructured patient data in the form of fingerprints.

[0043] FIG. 7 illustrates a methodology diagram 700 for training a convolutional neural network using unstructured patient data and corresponding diagnostic reports, according to various exemplary embodiments. Methodology diagram 700 discloses one method for implementing selected aspects of the present disclosure, according to various embodiments. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computing systems. Furthermore, although the operations of method 700 are shown in a particular order, this order is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0044] In block 702, unstructured patient data (UPD) may be obtained. The UPD may be UPD 605 referenced in Figure 6. The obtained UPD cannot be analyzed by AI due to its unstructured nature.

[0045] At block 704, a fingerprint reference matrix is ​​obtained. The fingerprint reference matrix may be understood to be equivalent to the diagnostic report vocabulary 610 described in Figure 6. The fingerprint reference matrix contains the matrix-organized contents of a dictionary or representative word list.

[0046] A diagnostic indication is obtained at block 706. The diagnostic indication may be the diagnostic indication 615 described in Figure 6. A diagnostic indication is a clinical finding, recommendation, or conclusion made by a medical professional.

[0047] At block 708, a patient fingerprint is generated, which may be understood to be the patient fingerprint 620 described at 620. As noted above, the patient fingerprint 620 is a mapping of occurrences of words occurring in the diagnostic report vocabulary 610 and UPD 605 to grayscale images.

[0048] At block 710, a convolutional neural network (CNN) is generated. The CNN may be understood to be the CNN 630 described in FIG. 6. The CNN may be trained using the fingerprint 620 as input and the diagnosis indication 615 as the desired output. The CNN is trained to trigger on the co-occurrence of specific words, or specific words in a specific order, within patient data with the results of relevant and consistent clinical diagnoses previously concluded by human medical professionals. As a result, once trained, the CNN can operate as a "virtual NLP engine" and can be used to directly support image-based CNNs using diagnostic images and corresponding unstructured patient data in the form of fingerprints.

[0049] To provide a concrete example of the use of a fingerprint CNN (e.g., fingerprint CNN 630 generated using method 700), a cardiology workflow may be considered. In this example, fingerprint CNN 630 may be considered to have been generated and ready for use by a cardiologist. It may also be considered that a patient has a previously generated fingerprint based on previous interactions, including both imaging information (e.g., previous scans) and non-imaging information (e.g., patient medical history, previous diagnostic reports, etc.). However, it should be understood that a patient need not have an existing fingerprint; for example, fingerprint CNN 630 may be applied to a newly generated patent fingerprint.

[0050] A patient's cardiology workflow may include an imaging procedure performed to obtain images of the patient's heart. The imaging procedure may include, for example, an MRI or an ultrasound. The workflow may also include collecting non-image information, such as comments by a medical professional (e.g., a radiologist) performing or reviewing the images. As described above, all of this data generated using the cardiology workflow may be unstructured data. This unstructured data may be stored in a Picture Archiving and Communication System (PACS) system configured to securely store electronic images and clinically relevant reports.

[0051] As described above, a cardiologist may review newly acquired images and clinical reports to make a diagnosis, but such a diagnosis may rely on incomplete data. An exemplary embodiment may extract newly acquired information from a PACS system and add this new data to a patient's existing fingerprint to generate an updated fingerprint. This updated patient fingerprint may then be analyzed by a fingerprint CNN to determine whether the updated patient fingerprint exhibits any symptoms related to a cardiac diagnosis. This fingerprint CNN analysis may be inserted into a cardiology workflow, for example, in the same way that an image-based CNN (e.g., a CNN that analyzes only cardiac images) may be inserted into a cardiology workflow. The cardiology workflow may then include the cardiologist being presented with one or more potential diagnoses generated by the fingerprint CNN and the image-based CNN. Again, as described above, the fingerprint CNN is generated using unstructured data, e.g., analyzing unstructured data that may be extracted from a PACS system for an individual patient. This eliminates any errors associated with attempted structuring of unstructured data.

[0052] The example workflow above relates to a cardiology workflow. However, it should be understood that the workflow may relate to any condition, e.g., an oncology workflow, a stroke workflow, etc. It should be understood that the imaging system may be any type of imaging system (e.g., an MRI, ultrasound, X-ray, CT scanner, PET scanner, etc.) and the data storage system may be any type of medically specific data storage system, some examples of which are provided below.

[0053] It should be understood that the method described in 700 relies on correlations between the co-occurrence of specific terms in the UPD and the outcomes of associated clinical diagnoses previously concluded by human medical professionals. To increase the sensitivity of the CNN, the CNN can be trained to trigger not only on word co-occurrence, but also on the specific order in which words appear. This approach is strengthened by the fact that medical professionals frequently utilize standardized language in their reports, such as "no signs of malignancy" or "patient with a history of hypertension."

[0054] In some example embodiments, the fingerprint may be extended with an explicit search for such standard representations, which can be seen as a vectorization of the individual terms in the fingerprint image.

[0055] FIG. 8 illustrates a vectorized unstructured fingerprint according to various exemplary embodiments. In this example, FIG. 8 illustrates "constellations" 805 and 810, which appear as white vector lines in the fingerprint image. Constellation 805 has the six-word phrase "patient with a history of hypertension." Each of these words is connected to adjacent words in the phrase via white vector lines (e.g., "patient" is connected to "has"). The six words of the phrase appearing in the fingerprint appear as six vertices of the "constellation." Similar logic applies to constellation 810 for the phrase "no signs of malignancy." Adding these constellations as triggers to the CNN specifically for the desired output signal can improve the CNN's performance.

[0056] It should be understood that other methods, such as those used in traditional NLP approaches, can be integrated into the fingerprint. Conditions related to the raw data that need to be monitored may be assigned to auxiliary pixels added at the bottom or periphery of the fingerprint. For example, several additional bottom rows of pixels may be added that are used to store occurrences of specific phrases, including "no signs of malignancy" or "patients with a history of high blood pressure." Then, during the creation of the fingerprint, not only the occurrences of individual terms are counted, but also the occurrences of "constellations," or any other criteria obtained by algorithms designed to analyze and interpret the raw data.

[0057] 9 illustrates a flow diagram 900 for using an image-based convolutional neural network with a fingerprint-based convolutional neural network for use in clinical decision support, according to various exemplary embodiments. It should be understood that the unstructured patient data 910, diagnostic report vocabulary 915, unstructured patient fingerprint 925, and fingerprint CNN 930 proceed in the same manner as their corresponding numbering in FIG. 6.

[0058] The image data 905 may be any type of medical imaging data (e.g., CT scan, MRI scan, X-ray image, etc.). The image data may be processed by an image-based CNN 920. The image-based CNN 920 and the fingerprint CNN 930 may be fed to an AI engine to generate a diagnostic suggestion 935. The diagnostic suggestion is the final product of the dictionary, image data, and UPD. The diagnostic suggestion may assist a medical professional in making a diagnosis by surfacing potentially unnoticed information.

[0059] FIG. 10 illustrates a methodology diagram for using an image-based convolutional neural network in conjunction with a fingerprint-based convolutional neural network for use in clinical decision support, according to various exemplary embodiments. An exemplary process 1000 for implementing selected aspects of the present disclosure, according to many embodiments, is disclosed. For convenience, the operations of the flowcharts are described with reference to a system that performs the operations. This system may include various components of various computing systems. Furthermore, although the operations of process 1000 are shown in a particular order, this order is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0060] It should be understood that blocks 1002, 1004, 1008, and 1010 are performed in the same manner as operations 702, 704, 708, and 710, respectively. The relevant point is that in 1010, a fingerprint CNN is created. The fingerprint CNN may be understood to be the fingerprint CNN 930 described in FIG. 9.

[0061] At block 1006, patient image data is acquired, which may be understood to be the image data 905 discussed with respect to FIG.

[0062] In block 1012, an image CNN is generated from the patient image data 905. The image-based CNN may be understood to be the image-based CNN 920 described in FIG.

[0063] In block 1014, the image-based CNN 930 and the fingerprint CNN 830 are used to generate a diagnostic suggestion, which can be used to assist a medical professional in making an accurate diagnosis of a patient (e.g., a patient with UPD).

[0064] It should be understood that the methods and operations of the exemplary embodiments may be performed on a system, which may include, for example, a radiology information system (“RIS”), a PACS system such as Philips VuePACS or Philips Intellispace PACS, an advanced visualization system for radiologists such as Philips Intellispace Portal, a teleradiology system, a cardiology PACS such as Philips Intellispace Cardiovascular, a CT workstation, an imaging system, or other medical devices and systems having specialized hardware and software for processing medical diagnostic information.

[0065] FIG. 11 shows a schematic diagram of an exemplary system according to various exemplary embodiments. As shown in FIG. 11, the system 1100 generates fingerprints for both data visualization and CNN training purposes. The system 1100 includes a processor 1102, a user interface 1104, a display 1106, and a memory 1108. The memory 1108 includes a database 1120 that can store UPD, image data, clinical findings, and diagnostic report vocabulary lists. The database 1120 can be a local storage medium, such as an HDD or SSD, on a local computer that serves as the storage medium for the system 1100. However, it should be understood that the database 1120 can also be an off-site storage medium, such as cloud storage, or a distributed local network storage accessible by a computer.

[0066] Data accessible via database 1120 may include, for example, clinical data from a variety of sources, such as medical images (e.g., MRI, CT, CR ultrasound), problem lists, lab values, medication lists, and documents including admission and discharge notes and pathology, radiology, and operational reports.

[0067] The processor 1102 may include a fingerprint generation engine 1110 for creating fingerprints from the UPD for use in training a CNN and for use in data visualization by medical professionals. The processor 1102 may further include a CNN training engine 1112 for using the fingerprints, diagnostic indications, and images to train a CNN and for use in generating diagnostic suggestions. Those skilled in the art will understand that the engines 1110-1112 may be implemented by the processor 1102, for example, as lines of code executed by the processor 1102, as firmware executed by the processor 1102, or as functions of the processor 1102 being an application specific integrated circuit (ASIC).

[0068] By making a selection on user interface 1104, a user, which may include, for example, a medical professional including a doctor, nurse, medical technician, etc., may initiate fingerprinting and CNN training. A user may also edit and / or set parameters for the above-mentioned engines 1110-1112 via user interface 1104.

[0069] The display 1106 may be used to display any of the information described herein, such as, for example, fingerprints, difference fingerprints, linked data, etc.

[0070] Those skilled in the art will appreciate that the above exemplary embodiments can be implemented in any suitable software or hardware configuration, or combination thereof. Exemplary hardware platforms for implementing the exemplary embodiments may include, for example, Intel x86-based platforms with compatible operating systems, Windows OS, Mac platforms, and mobile devices with operating systems such as MAC OS, iOS, Android, etc. In a further example, the exemplary embodiments of the above-described methods may be embodied as a program including lines of code stored on a non-transitory computer-readable storage medium that, when compiled, can be executed on a processor or microprocessor.

[0071] Although the present application has described various embodiments each having different features in various combinations, those skilled in the art will understand that any of the features of one embodiment may be combined with the features of other embodiments in any manner not specifically disclaimed or that is not functionally or logically inconsistent with the operation of the apparatus or described functionality of the disclosed embodiments.

[0072] It is fully understood that use of personally identifiable information should comply with generally recognized privacy policies and practices that meet or exceed industry or government requirements for maintaining user privacy. In particular, personally identifiable information data should be managed and handled in a manner that minimizes the risk of unintended or unauthorized access or use, and the nature of authorized uses should be clearly indicated to users.

[0073] It will be apparent to those skilled in the art that various modifications can be made in the present disclosure without departing from the spirit or scope of the disclosure. Thus, it is intended that the present disclosure cover the modifications and variations of the present disclosure provided they come within the scope of the appended claims and their equivalents.

Claims

1. receiving a diagnostic indication, unstructured patient data (UPD), and a diagnostic report vocabulary list, the diagnostic report vocabulary list having ordered vocabulary terms; generating a blank bitmap having a number of pixels (i) corresponding to the quantity of said vocabulary terms, and (ii) a plurality of pixels having a correspondence between pixel positions in the bitmap and the order of said vocabulary terms; generating a fingerprint having a mapping of each occurrence of a vocabulary term in the UPD to a corresponding pixel location in the blank bitmap; training a convolutional neural network (CNN) using the fingerprint as an input and the diagnostic representation as a target output; A method having the following.

2. generating a second fingerprint using a second UPD and the diagnostic report vocabulary list; inputting the second fingerprint into the CNN; receiving a second diagnostic indication as an output from the CNN; The method of claim 1 further comprising:

3. receiving patient image data; training an image-based CNN using the patient image data; The method of claim 1 further comprising:

4. generating a second fingerprint using a second UPD and the diagnostic report vocabulary list; receiving second patient image data corresponding to the second fingerprint; inputting the second fingerprint and second patient image data into an artificial intelligence (AI) model including the CNN and the image-based CNN; receiving a second diagnostic indication as output from the AI ​​model; The method of claim 3 further comprising:

5. The method of claim 1 , wherein the diagnostic report vocabulary list comprises a medical dictionary.

6. The method of claim 1 , wherein the UPD comprises text-based data.

7. The method of claim 1 , wherein the diagnostic indication comprises a clinical finding, a recommendation, or a conclusion.

8. The method of claim 1 , wherein generating the fingerprint further comprises modifying a pixel value associated with at least one pixel location in the bitmap.

9. The method of claim 7 , wherein the pixel values ​​vary between 0 and 255 inclusive.

10. receiving unstructured patient data (UPD) and a diagnostic report vocabulary list, the diagnostic report vocabulary list including ordered vocabulary terms; generating a blank bitmap having a number of pixels (i) corresponding to the quantity of said vocabulary terms, and (ii) a plurality of pixels having a correspondence between pixel positions in said bitmap and the order of said vocabulary terms; generating a first fingerprint comprising a mapping of each occurrence of a vocabulary term in the UPD to a corresponding position in the blank bitmap by modifying pixel values ​​associated with corresponding pixel positions in the bitmap; displaying the fingerprint to a user; A method having the following.

11. displaying the vocabulary term on the first fingerprint when the pixel value is equal to or exceeds a threshold value; The method of claim 10 further comprising:

12. generating links between said vocabulary terms and their corresponding locations in said UPD; The method of claim 11 further comprising:

13. The method of claim 11 , wherein the displayed vocabulary terms include colors associated with clinical concepts.

14. generating vectorized connections between a plurality of input vocabulary terms; displaying the vectorized connections on the first fingerprint; The method of claim 10 further comprising:

15. generating a second fingerprint using a second UPD and the diagnostic report vocabulary list; performing image subtraction between the second fingerprint and the first fingerprint; displaying the result of the image subtraction; The method of claim 10 further comprising:

16. The method of claim 10 , wherein the pixel values ​​vary between 0 and 255 inclusive.

17. 1. A system for creating a data structure for use in clinical diagnostic support, comprising: a memory containing a diagnostic indication, unstructured patient data (UPD), and a diagnostic report vocabulary list, the diagnostic report vocabulary list containing ordered vocabulary terms; generating a blank bitmap having a number of pixels (i) corresponding to the quantity of said vocabulary terms, and (ii) a plurality of pixels having a correspondence between pixel positions in said bitmap and the order of said vocabulary terms; generating a fingerprint comprising a mapping of each occurrence of a vocabulary term in the UPD to a corresponding pixel location in the blank bitmap; training a convolutional neural network (CNN) using the fingerprint as an input and the diagnostic representation as a target output; a processor configured to: A system having:

18. 1. A system for creating a data structure for use in clinical diagnostic support, comprising: a memory containing unstructured patient data (UPD) and a diagnostic report vocabulary list, the diagnostic report vocabulary list containing ordered vocabulary terms; generating a blank bitmap having a number of pixels (i) corresponding to the quantity of said vocabulary terms, and (ii) a plurality of pixels having a correspondence between pixel positions in said bitmap and the order of said vocabulary terms; generating a first fingerprint comprising a mapping of each occurrence of a vocabulary term in the UPD to a corresponding position in the blank bitmap by modifying pixel values ​​associated with corresponding pixel positions in the bitmap; a processor configured to: A system having:

19. A computer program operable to perform the method of claim 1 when run on a computer.

20. A computer program operable to perform the method of claim 10 when the computer program is run on a computer.

Images