Fingerprint identification of unstructured patient data

Abstract

A convolutional neural network (CNN) for a medical workflow is generated by receiving a diagnostic indication, unstructured patient data (UPD), and a diagnostic report vocabulary list, where the diagnostic report vocabulary list includes ordered vocabulary terms, generating a blank bitmap including a plurality of pixels having (i) a number of pixels corresponding to the number of vocabulary terms, (ii) a number of pixels corresponding to the number of vocabulary terms, and (ii) a number of pixels corresponding to the number of vocabulary terms. And (ii) correspondence between pixel positions in the bitmap and an order of vocabulary terms, generating a fingerprint comprising a mapping of each occurrence of vocabulary terms in the UPD to corresponding pixel positions in the blank bitmap, and training a convolutional neural network (CNN) using the fingerprint as an input and using the diagnostic indication as a target output.

Description

BACKGROUND OF THE INVENTION

[0001] The use of digital technology in radiology has led to a significant increase in the amount of clinical images and non-image information used by healthcare professionals for diagnosis. Despite this growth, clinical assessments are still performed in a traditional manner. This inconsistency has led to a substantial increase in the workload and data overload of healthcare professionals. Artificial intelligence (AI) methods with deep learning analysis via convolutional neural networks (CNNs) are being used more frequently to partially address the above problems in the art.

[0002] Although AI provides a potential means of addressing the workload problem, the background information of patients (e.g., non-image information such as patient history, previous diagnostic reports, etc.) remains a crucial component for reliable diagnosis. Finding the appropriate patient information relevant to a specific clinical context remains a time-consuming task.

[0003] Although AI can be used to find the appropriate patient information, unstructured patient data (UPD) must first be converted to a structured format by using natural language processing (NLP) or other methods. This is not a trivial task, and automated NLP methods are still in their infancy. When the structured data is fed into the AI engine, errors generated by NLP during data structuring may introduce a new set of problems. Given the current dynamics of healthcare worldwide (e.g., declining reimbursements, value-driven healthcare policies, shortages of healthcare personnel worldwide, and high rates of sick leave due to work stress), there is an urgent need to improve this data flow and structuring. SUMMARY OF THE INVENTION

[0004] Some example embodiments relate to a method for receiving a diagnostic indication, unstructured patient data (UPD), and a list of diagnostic report vocabulary terms, where the list of diagnostic report vocabulary terms includes ordered vocabulary terms, generating a blank bitmap including a plurality of pixels, the plurality of pixels having (i) a number of pixels corresponding to the number of vocabulary terms, and (ii) a correspondence between the pixel positions in the bitmap and the order of the vocabulary terms, generating a fingerprint, the fingerprint including a mapping of each occurrence of a vocabulary term in the UPD to a corresponding pixel position in the blank bitmap, and training a convolutional neural network (CNN) using the fingerprint as an input and using the diagnostic indication as a target output.

[0005] Other example embodiments relate to a method for receiving unstructured patient data (UPD) and a diagnostic report vocabulary list, where the diagnostic report vocabulary list includes ordered vocabulary terms, generating a blank bitmap including a plurality of pixels, the plurality of pixels having: (i) a number of pixels corresponding to the number of vocabulary terms, and (ii) a correspondence between the pixel positions in the bitmap and the order of the vocabulary terms, generating a first fingerprint by changing the pixel values associated with the corresponding pixel positions in the bitmap, the first fingerprint including a mapping of each occurrence of a vocabulary term in the UPD to a corresponding position in the blank bitmap, and displaying the fingerprint to a user.

[0006] Other exemplary embodiments relate to a system for creating a data structure for clinical diagnosis support, where such a system includes a memory that includes diagnostic indications, unstructured patient data (UPD), and a diagnostic report vocabulary list, where the diagnostic report vocabulary list includes ordered vocabulary terms. The system also includes a processor configured to generate a blank bitmap including a plurality of pixels, the plurality of pixels having: (i) a number of pixels corresponding to the number of vocabulary terms, and (ii) a correspondence between the pixel positions in the bitmap and the order of the vocabulary terms. The processor is also configured to generate a fingerprint that includes a mapping of each occurrence of a vocabulary term in the UPD to a corresponding pixel position in the blank bitmap, and to train a convolutional neural network (CNN) using the fingerprint as an input and using the diagnostic indications as a target output.

[0007] Other exemplary embodiments relate to a system for creating a data structure for clinical diagnosis support, where such a system includes a memory that includes unstructured patient data (UPD) and a diagnostic report vocabulary list, where the diagnostic report vocabulary list includes ordered vocabulary terms. The system also includes a processor configured to generate a blank bitmap including a plurality of pixels, the plurality of pixels having: (i) a number of pixels corresponding to the number of vocabulary terms, and (ii) a correspondence between the pixel positions in the bitmap and the order of the vocabulary terms. The processor is also configured to generate a first fingerprint by changing the pixel values associated with the corresponding pixel positions in the bitmap, the first fingerprint including a mapping of each occurrence of a vocabulary term in the UPD to a corresponding position in the blank bitmap. The system also includes a display for displaying the fingerprint to a user.

[0008] Other example embodiments relate to a computer program product that is operable, when executed on a computer, to perform the methods as described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1Illustrates an exemplary fingerprint reference matrix according to various exemplary embodiments.

[0010] Figure 2 Illustrates an unstructured fingerprint according to various exemplary embodiments.

[0011] Figure 3 Illustrates an unstructured fingerprint with frequently occurring marked entries according to various exemplary embodiments.

[0012] Figure 4 Illustrates a differential fingerprint according to various exemplary embodiments.

[0013] Figure 5 Illustrates an unstructured fingerprint with an interactive word cloud according to various exemplary embodiments.

[0014] Figure 6 Illustrates a flowchart for training a convolutional neural network using unstructured patient data and corresponding diagnostic reports according to various exemplary embodiments.

[0015] Figure 7 Illustrates a method diagram for training a convolutional neural network using unstructured patient data and corresponding diagnostic reports according to various exemplary embodiments.

[0016] Figure 8 Illustrates a vectorized unstructured fingerprint according to various exemplary embodiments.

[0017] Figure 9 Illustrates a flowchart for the use of an image-based convolutional neural network with a fingerprint-based convolutional neural network for clinical decision support according to various exemplary embodiments.

[0018] Figure 10 Illustrates a method diagram for the use of an image-based convolutional neural network with a fingerprint-based convolutional neural network for clinical decision support according to various exemplary embodiments.

[0019] Figure 11 Illustrates a schematic diagram of an exemplary system according to various exemplary embodiments. Detailed Description

[0020] Exemplary embodiments can be further understood with reference to the following description and the related drawings, in which the same elements are provided with the same reference numerals. The exemplary embodiments relate to using "fingerprints" to assist in both clinical and AI analysis of unstructured patient data (UPD).

[0021] As described above, when the AI analyzes structured patient data, the use of natural language processing (NLP) on the UPD may introduce a cascade of errors. Exemplary embodiments provide an alternative data structure for AI analysis. This alternative data structure is referred to throughout the specification as a "fingerprint". The fingerprint can be used for clinical diagnosis support. At a higher level of abstraction, the fingerprint can be understood as an image-based representation of relevant clinical terms (e.g., from UPD sources). It should be understood that the term fingerprint is used throughout the specification to refer to a data structure having certain characteristics as described herein. Thus, a fingerprint should be understood as a data structure having the characteristics as described herein.

[0022] Using the fingerprint along with the corresponding clinical diagnosis report can be used to train a convolutional neural network (CNN), where the fingerprint serves as the input data and the structured findings in the clinical report serve as the desired output. The fingerprint can take the form of a small (with respect to data usage) grayscale image file, which can be analyzed by existing AI platforms for image-related training without modification. Once trained, the CNN can act as a "virtual NLP engine" and can be used to support image-based CNNs in a straightforward manner using diagnostic images and the corresponding UPD.

[0023] The term virtual NLP is reasonable because the AI network can be trained to be triggered when specific terms or specific words in a specific order that occur simultaneously in the patient data are encountered, in case they are associated with consistent clinical diagnosis findings. Thus, the AI can benefit from the analysis performed by healthcare professionals who create the diagnostic reports that are subsequently used for AI training.

[0024] Another advantage of fingerprints is that they are small-sized images (ideally less than 100 kilobytes, although larger sizes are possible if the operator desires). This small size is independent of the amount of data represented by the fingerprint, thus allowing an efficient data format to store the essential aspects of the UPD for NLP algorithms and longitudinal studies.

[0025] Fingerprints are useful for the analysis of longitudinal studies. As will be described in more detail below, fingerprints reveal any relevant differences between two large amounts of unstructured data without the use of search and indexing algorithms, but rather by simply using image subtraction. The use of image subtraction reveals only the terms that have changed between any two given studies for a given patient. Fingerprints can also be used as an attractive and effective user interface (UI) visualization for user interaction with large amounts of UPD, which will be described in more detail below.

[0026] To create a fingerprint in a clinical environment, various information and operations can be performed. For example, the information and operations can include a dictionary (e.g., a medical dictionary), an extraction algorithm for creating a fingerprint from any unstructured text-based data source, and an image-driven AI engine that converts the fingerprint into diagnostic recommendations for clinicians to consider. A large collection of UPDs and related clinical diagnostic findings can be used to train the above AI engine.

[0027] Ideally, the information should be available longitudinally (i.e., various versions of the same case / study over time). Using longitudinal UPDs and related clinical diagnostic findings can improve AI training efficiency. However, it should be understood that longitudinal data is not required.

[0028] As an example of using longitudinal data, consider two different data reports for the same patient. The first data report can include patient data created at time t3 and an up-to-date diagnostic report for a clinical condition diagnosed shortly after time t3. The second data report can include patient data created at time t1 and a diagnostic report created shortly after time t1, plus information created at time t2 and a diagnostic report created shortly after time t2, plus information created at time t3 and a diagnostic report created shortly after time t3. The first data report will provide less information than the second data report because the incremental aspects of the information will be lost if all the data is bundled together.

[0029] The AI engine can operate in parallel with existing algorithms for image-based AI, integrate with such an image-based AI engine (since the fingerprint is a normal digital image), or be an independent application for generating diagnostic recommendations obtained from unstructured patient data during diagnosis.

[0030] As described above, a dictionary can be used to create a fingerprint. It should be understood that the term "lexicon" also encompasses any other representative list of words. In an example, a medical dictionary containing 98,119 words is used as a fingerprint reference, along with a 120-page PDF file to be used as UPD.

[0031] Initially, a digital grayscale image of 314×314 one-byte pixels is created, resulting in a total file size of 98.6 kilobytes. 314 is the smallest integer whose square is greater than the number of words in the dictionary used as the fingerprint reference ((314 2 = 98,596) > 98,119).

[0032] Next, each pixel in the grayscale image is assigned a unique word from the dictionary. This operation can be performed in several ways, but in this example, all pixels of the grayscale image from top left to bottom right (i.e., reading order) are filled alphabetically with the words in the dictionary. It should be understood that other mappings between the dictionary and the grayscale image are possible.

[0033] Figure 1 An exemplary fingerprint reference matrix 100 according to various exemplary embodiments is shown. Figure 1 The upper leftmost corner of a 314×314 fingerprint reference matrix 100 is depicted. The upper leftmost corner is filled with the first term in alphabetical order of the corresponding dictionary (in this case, "abasia"). The matrix is filled alphabetically, from left to right, row by row until the dictionary is exhausted.

[0034] The use of the fingerprint reference matrix allows each pixel in a 314×314 image to be uniquely assigned the nth word in the dictionary by the following formula:

[0035] n = (j - 1)·314 + i, where 1 ≤ i ≤ 314 and 1 ≤ j ≤ 314

[0036] Indicates that the nth word in the dictionary will be assigned to the pixel at the nth column from the right and the jth row from the top.

[0037] Using both the assignment formula and the fingerprint reference matrix, the creation of the fingerprint is straightforward. Starting with a 314×314 image where all pixels are set to the value 0 (i.e., a uniformly black image), a standard text reading algorithm reads each word from the unstructured data set (the aforementioned 120 - page UPD PDF). Whenever the word read from the UPD exists in the fingerprint reference matrix, the corresponding pixel value in the fingerprint is incremented by one. In this example, each pixel is limited to a maximum value of 255, corresponding to the maximum value of a single byte. Although this maximum value may be sufficient for all practical applications, it can be extended.

[0038] Figure 2 An unstructured fingerprint 200 according to various exemplary embodiments is shown. Noticeable in the fingerprint 200 are the more prominent (brighter) pixels scattered throughout the image. The position of any given pixel corresponds to the nth term in the dictionary, and the nth term is translated row by row from left to right to the top of the pixel grid. The brightness of any given pixel corresponds to the frequency with which the nth term in the dictionary appears in the UPD. Thus, more frequently occurring terms correspond to brighter pixels, up to and including the term that appears 255 times in the UPD (the maximum value of a byte).

[0039] Figure 3 An unstructured fingerprint 300 with frequently occurring marked entries according to various exemplary embodiments is shown.Figure 3 depicts a UPD that is identical to the fingerprint 200 shown in Figure 2 However, frequently occurring terms (brighter pixels) are marked with their corresponding dictionary entries. The threshold for constituting frequently occurring entries can be defined by the operator (e.g., 10 times, 50 times, 100 times, etc.). Figure 3 shows an attractive means for visualizing the UPD by the user.

[0040] It should be noted that the fingerprint can be a fuzzy identifier of the scanned UPD, because it is possible to create the same fingerprint with different data. However, in a clinical context, this ambiguity is irrelevant because clinicians are concerned with changes in fingerprints (e.g., longitudinal studies). Since any change in the number of relevant terms (i.e., dictionary terms used to populate the fingerprint reference matrix) will increase or decrease the brightness of the corresponding pixels in the fingerprint, this clinical need is met.

[0041] This property of fingerprints makes their use in longitudinal studies attractive. Image subtraction of two fingerprint images created on different dates will create a new third differential fingerprint that reveals all relevant terms added at the second date. It should be understood that image subtraction can similarly reveal relevant terms deleted / removed from the second fingerprint.

[0042] Figure 4 shows a differential fingerprint 400 according to various exemplary embodiments. The differential fingerprint can be created by per-pixel image subtraction of two fingerprints (fingerprint 2 "F2" - fingerprint 1 "F1") created at different times. Figure 4 The two fingerprints (F1, F2) used for its creation are not depicted because the relevant aspect is the depicted change. A cursory analysis of the differential fingerprint 400 reveals that F2 has data related to an acute myocardial infarction diagnosed by gadolinium delayed enhancement (LGE) magnetic resonance imaging (MRI). It should be understood that the changes that occurred in F2 occurred after F1 was created.

[0043] Figure 4 demonstrates the value of fingerprints with respect to longitudinal studies. The small data size of fingerprints (less than 100 kilobytes, regardless of the number of UPDs they represent) allows for effective visualization of changes in the UPD. Changes in the patient chart can be quickly compared relative to any two fingerprints obtained at different times.

[0044] The visualization and lookup capabilities of fingerprints can be extended through deep links. A deep link can be understood as a connection between the exact positions in the UPD corresponding to a given pixel in the fingerprint. As an example, if the term "lymphoma" appears 117 times in the UPD (corresponding to a pixel with a value of 117), the user can hover their mouse (or any other suitable interaction device) over the corresponding pixel in the fingerprint, and a list or dictionary in which the term "lymphoma" directly appears in the UPD is presented. The list or dictionary can include exact links to positions in the UPD (specific pages or lines where "lymphoma" appears) or simply to the document (the entire report).

[0045] Manually selecting a single pixel from a grid of 98,696 (314 2 ) pixels can be a difficult exercise in terms of flexibility. Taking this into account, Figure 3-4 the tagging system shown in can be further enhanced to a "word cloud". The tagging system can be characterized by the above-mentioned deep links. From a UI perspective, the tagging system can increase the font size of specific tags based on the corresponding pixel values (which themselves correspond to the number of terms in the UPD). Pixels with values below a specified threshold can be tagged to reduce visual clutter. It is also possible that the tags can be colored or otherwise indicated based on the corresponding terms (e.g., clinical concepts related to the terms). For example, all cardiac terms can be colored red, all oncology terms can be colored green, and so on. It should be understood that any combination of tag font sizes and coloring schemes is possible based on the operator's needs.

[0046] Figure 5 An unstructured fingerprint 500 with an interactive word cloud according to various exemplary embodiments is shown. In this example, the fingerprint 500 includes certain terms with sizes increased based on the corresponding pixel values. A color-coding scheme can also be utilized to group certain categories of terms. The user can hover their mouse over non-zero (i.e., non-black) pixels / tags to view a preview of the corresponding UPD file. The user can click on any non-zero value pixel, corresponding to a deep link to the source(s) of the term(s). Upon a mouse click, all documents containing the corresponding term can be presented to the user, where the corresponding term(s) in the document(s) are highlighted for easy reference. Although adding this functionality to the fingerprint will significantly increase the file size, those skilled in the art will recognize the value of the simplified visualization of the UPD.

[0047] As described above, fingerprints can be used in conjunction with AI methods to use unstructured data as a source of additional information for clinical diagnosis, e.g., to support traditional or AI-assisted clinical image analysis. As described above, converting UPDs to structured data using NLP typically introduces errors, which reduces or eliminates any efficiency gains from feeding structured data into an AI engine. The use of fingerprints can eliminate these types of errors by eliminating the data structuring step for UPDs.

[0048] Unlike using structured patient data, fingerprints of UPDs can be used with corresponding clinical diagnosis reports to generate a CNN. The fingerprints are used as input data, and the structured findings in the clinical reports are used as the desired output. Thus, the CNN will be trained to be triggered when specific words in the patient data occur simultaneously or in a specific order, where the relevant consistent clinical diagnosis meanings were earlier summarized by healthcare professionals. Once trained, the CNN can act as a "virtual NLP engine" and can be used to directly support image-based CNNs using diagnostic images and corresponding UPDs in the form of fingerprints.

[0049] Figure 6 A flowchart 600 for training a convolutional neural network using unstructured patient data and corresponding diagnostic reports in accordance with various exemplary embodiments is shown. The UPD 605 can be patient text data (e.g., test results, clinical notes, etc.) that has not been structured by NLP into an AI-scannable format.

[0050] The diagnostic report vocabulary 610 can be a medical dictionary or any other representative list of words stored in a matrix (e.g., Figure 1 example) that can be mapped from text analysis of the UPD to a fingerprint image.

[0051] The diagnostic indication 615 can be clinical findings, recommendations, or conclusions created by a medical professional. The diagnostic indication 615 should be understood as an entity separate from the unstructured patient data 605. The value of the diagnostic indication 615 is that the findings, recommendations, or conclusions reached by a human healthcare professional can enhance the capabilities of the CNN.

[0052] At 620, a patient fingerprint is created. The UPD 605 is scanned using a standard text reading algorithm. Whenever the text reading algorithm finds a word that appears in the dictionary 610 matrix, the brightness of a single pixel in a 314×314 grayscale grid is increased by 1 / 255. As described above, the position of the pixel corresponds to the grid position of the word found in the dictionary 610 matrix. The nth word in the dictionary can be located by the following formula:

[0053] n = (j - 1) * 314 + i, where 1 < i <= 314 and 1 < j <= 314

[0054] It is indicated that the nth word in the dictionary will be assigned to the pixel at the nth column from the right and the jth row from the top. The brightness of each pixel corresponds to the number of times the word appears in the UPD 605, up to a maximum of 255.

[0055] At 630, the fingerprint 620 is used as the input and the diagnostic indication 625 is used as the desired output to train the CNN. The CNN 630 can be trained to be triggered when specific words in the patient data appear simultaneously or in a specific order, where the relevant consistent clinical diagnostic meanings were previously summarized by human healthcare professionals. Thus, once trained, the CNN can act as a "virtual NLP engine" and can be used to support the image-based CNN in a direct manner using the corresponding unstructured patient data in the form of diagnostic images and fingerprints.

[0056] Figure 7 FIG. 700 shows a method diagram for training a convolutional neural network using unstructured patient data and corresponding diagnostic reports according to various exemplary embodiments. The method diagram 700 discloses a method for practicing selected aspects of the present disclosure. For convenience, the operations of the flowchart are described with reference to the system performing the operations. The system may include various components of various computing systems. Additionally, although the operations of method 700 are shown in a specific order, that order is not meant to be restrictive. One or more operations may be reordered, omitted, and / or added.

[0057] At block 702, unstructured patient data (UPD) can be obtained. The UPD can be the Figure 6 UPD 605 mentioned in. The obtained UPD cannot be analyzed by AI due to its unstructured nature.

[0058] At block 704, a fingerprint reference matrix is obtained. The fingerprint reference matrix can be understood as equivalent to the Figure 6 diagnostic report vocabulary 610 described in. The fingerprint reference matrix contains the matrix-organized content of a dictionary or a list of representative words.

[0059] At block 706, a diagnostic indication is obtained. The diagnostic indication can be the Figure 6 diagnostic indication 615 described in. The diagnostic indication is a clinical finding, recommendation, or conclusion created by a medical professional.

[0060] At block 708, a patient fingerprint is generated. The patient fingerprint can be understood as the patient fingerprint 620 described at 620. As described above, the patient fingerprint 620 is a mapping between the words that appear in the diagnostic report vocabulary 610 and the UPD 605 and the grayscale image.

[0061] At block 710, a Convolutional Neural Network (CNN) is generated. The CNN can be understood as the CNN 630 described in Figure 6 The CNN can be trained by using the fingerprint 620 as the input and the diagnostic indication 615 as the desired output. The CNN will be trained to trigger when specific words in the patient data occur simultaneously or in a specific order, where the relevant consistent clinical diagnostic meanings have been previously summarized by a human healthcare professional. Thus, once trained, the CNN can act as a "virtual NLP engine" and can be used to support the image-based CNN in a straightforward manner using the corresponding unstructured patient data in the form of diagnostic images and fingerprints.

[0062] To provide a specific example of using the fingerprint CNN (e.g., the fingerprint CNN 630 generated using method 700), a cardiology workflow can be considered. In this example, it can be assumed that the fingerprint CNN 630 has been generated and is ready to be used by a cardiologist. It can also be assumed that the patient has a previously generated fingerprint based on previous interactions that include both imaging information (e.g., previous scans) and non-image information (e.g., patient history, previous diagnostic reports, etc.). However, it should be understood that it is not required for the patient to have a pre-existing fingerprint. For example, the fingerprint CNN 630 can be applied to newly generated patient fingerprints.

[0063] The cardiology workflow for a patient can include an imaging process that is performed to obtain an image of the patient's heart. The imaging process can include, for example, MRI or ultrasound. The workflow can also include collecting non-image information, such as the comments of the healthcare professional (e.g., radiologist) who performed or viewed the image. As described above, all of this data generated using the cardiology workflow can be unstructured data. This unstructured data can be stored in a PACS (Picture Archiving and Communication System) system that is configured to securely store electronic images and clinically relevant reports.

[0064] As described above, a cardiologist can view newly acquired images and clinical reports for diagnosis, but such diagnosis may rely on incomplete data. Exemplary embodiments can extract newly acquired information from a PACS system and add this new data to an existing fingerprint of a patient to generate an updated fingerprint. The updated patient fingerprint can then be analyzed by a fingerprint CNN to determine whether the updated patient fingerprint exhibits any signs relevant to a cardiac diagnosis. This fingerprint CNN analysis can be inserted into a cardiology workflow in the same manner as, for example, an image-based CNN (e.g., a CNN that only analyzes cardiac images) can be inserted into a cardiology workflow. The cardiology workflow can then include presenting to the cardiologist one or more potential diagnoses generated by the fingerprint CNN and the image-based CNN. Similarly, as described above, the fingerprint CNN is generated using unstructured data and the analysis can be of unstructured data extracted from, for example, a PACS system of a single patient. This eliminates any errors associated with attempting to structure unstructured data.

[0065] The example workflow described above relates to a cardiology workflow. However, it should be understood that the workflow can relate to any condition, such as an oncology workflow, a stroke workflow, etc. It should also be understood that the imaging system can be any type of imaging system (e.g., MRI, ultrasound, X-ray, CT scanner, PET scanner, etc.) and the data storage system can be any type of medically specific data storage system, some examples of which are provided below.

[0066] It should be understood that the method described in 700 relies on the correlation between the co-occurrence of specific terms in the UPD and the relevant clinical diagnostic meanings previously summarized by a human healthcare professional. To increase the sensitivity of the CNN, the CNN can be trained to trigger not only when words co-occur, but also in the specific order in which they occur. This approach is supported by the fact that healthcare professionals typically utilize standardized wording in their reports, such as "no signs of malignancy" or "patient with a history of hypertension".

[0067] In some exemplary embodiments, an explicit search for such standard wording can be utilized to enhance the fingerprint. This can be regarded as the vectorization of individual terms in the fingerprint image.

[0068] Figure 8 A vectorized unstructured fingerprint according to various exemplary embodiments is shown. In this example, Figure 8Shows "constellations" 805 and 810 that appear as white vector lines in the fingerprint image. Constellation 805 has six word phrases "patient with a history of hypertension". Each of these words is connected via a white vector line to the adjacent word in the phrase (e.g., "patient" is connected to "with"). The six words of the phrase appear in the fingerprint as the six vertices of the "constellation". Similar logic applies to constellation 810 of the phrase "no signs of malignancy". Adding these constellations as triggers to the CNN for a specific desired output signal can improve the performance of the CNN.

[0069] It should be understood that other methods (such as those used in traditional NLP methods) can be integrated into the fingerprint. Conditions related to the raw data to be monitored can be assigned to auxiliary pixels added at the bottom or periphery of the fingerprint. For example, several additional bottom pixel rows can be added for storing the occurrences of specific phrases, including "no signs of malignancy" or "patient with a history of hypertension". Subsequently, during the creation of the fingerprint, not only the occurrences of individual terms are counted, but also the occurrences of "constellations" or any other specifications obtained by algorithms designed to analyze and interpret the raw data are counted.

[0070] Figure 9 Shows a flowchart 900 for the use of an image-based convolutional neural network with a fingerprint-based convolutional neural network for clinical decision support according to various exemplary embodiments. It should be understood that the unstructured patient data 910, diagnostic report vocabulary 915, unstructured patient fingerprint 925, and fingerprint CNN 930 are carried out with the same corresponding numbers as in Figure 6 in.

[0071] The image data 905 can be any type of medical imaging data (e.g., CT scan, MRI scan, X-ray image, etc.). The image data can be processed by the image-based CNN 920. The image-based CNN 920 and the fingerprint CNN 930 can be fed into an AI engine to generate a diagnostic recommendation 935. The diagnostic recommendation is the final result of the dictionary, image data, and UPD. The diagnostic recommendation can assist medical professionals in making a diagnosis by revealing information that might otherwise go unnoticed.

[0072] Figure 10A method diagram showing the use of an image-based convolutional neural network with a fingerprint-based convolutional neural network for clinical decision support according to various exemplary embodiments is shown. According to many embodiments, an example process 1000 for practicing selected aspects of the present disclosure is disclosed. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computing systems. Additionally, although the operations of process 1000 are shown in a particular order, that order is not meant to be restrictive. One or more operations may be reordered, omitted, and / or added.

[0073] It should be understood that blocks 1002, 1004, 1008, and 1010 are executed identically to operations 702, 704, 708, and 710, respectively. The relevant point is that at 1010, a fingerprint CNN has been created. The fingerprint CNN can be understood as Figure 9 the fingerprint CNN 930 described in

[0074] At block 1006, patient image data is obtained. The patient image data can be understood as Figure 9 the image data 905 discussed.

[0075] At block 1012, an image CNN is generated based on the patient image data 905. The image-based CNN can be understood as Figure 9 the image-based CNN 920 described in

[0076] At block 1014, a diagnostic recommendation is generated using the image-based CNN 930 and the fingerprint CNN 830. The diagnostic recommendation can be used to assist a medical professional in making a correct diagnosis of a patient (e.g., a patient corresponding to UPD).

[0077] It should be understood that the methods and operations of the exemplary embodiments can be executed on a system. For example, the system may include a radiology information system (“RIS”), a PACS system (such as Philips VuePACS or Philips Intellispace PACS), an advanced visualization system for radiologists (such as Philips Intellispace Portal), a teleradiology system, a cardiology PACS (such as Philips Intellispace Cardiovascular), a CT workstation, an imaging system, or other medical devices and systems having dedicated hardware and software for processing medical diagnostic information.

[0078] Figure 11 A schematic diagram of an exemplary system according to various exemplary embodiments is shown. As Figure 11As shown, system 1100 generates fingerprints for data visualization and CNN training purposes. System 1100 includes a processor 1102, a user interface 1104, a display 1106, and a memory 1108. Memory 1108 includes a database 1120, which can store UPD, image data, clinical findings, and a list of diagnostic reports. It should be understood that database 1120 can be a local storage medium, such as an HDD or SSD on a local computer used as the storage medium of system 1100, but database 1120 can also be understood as an off-site storage medium, such as cloud storage, or distributed local network storage accessible by a computer.

[0079] Data accessible through database 1120 can include clinical data from various sources, such as medical images (e.g., MRI, CT, CR ultrasound), problem lists, laboratory values, medication lists, and documents including admission and discharge records, as well as pathology, radiology, and surgical reports.

[0080] Processor 1102 can include a fingerprint generation engine 1110, which is used to create fingerprints based on UPD for training CNNs and for data visualization by medical professionals. Processor 1102 can also include a CNN training engine 1112, which is used to train a CNN with fingerprints, diagnostic indicators, images, and generate diagnostic recommendations. Those skilled in the art will understand that engines 1110 - 1112 can be implemented by processor 1102 as, for example, lines of code executed by processor 1102, firmware executed by processor 1102, a function of processor 1102 (application-specific integrated circuit (ASIC)), etc.

[0081] By making selections on user interface 1104, users, including medical workers (including, for example, doctors, nurses, medical technicians, etc.), can initiate fingerprint recognition and CNN training. Users can also edit and / or set parameters for the above engines 1110 - 1112 via user interface 1104.

[0082] Display 1106 can be used to display any information described herein, such as fingerprints, differential fingerprints, linked data, etc.

[0083] Those skilled in the art will understand that the above exemplary embodiments can be implemented in any suitable software or hardware configuration or a combination thereof. Exemplary hardware platforms for implementing the exemplary embodiments can include, for example, Intel x86-based platforms with compatible operating systems, Windows OS, Mac platforms and MACOS, and mobile devices with operating systems such as iOS, Android, etc. In another example, the exemplary embodiments of the above method can be embodied as a program containing lines of code stored on a non-transitory computer-readable storage medium, which can be executed on a processor or microprocessor when compiled.

[0084] Although this application describes various aspects, each aspect having different features in various combinations, those skilled in the art will understand that any feature of one aspect can be combined with the features of other aspects in any way that is not particularly stated or is not functionally or logically inconsistent with the operation of the device or the described functions of the disclosed aspects.

[0085] It is well understood that the use of personally identifiable information should follow privacy policies and practices that are generally considered to meet or exceed industry or government requirements for maintaining user privacy. In particular, personally identifiable information data should be managed and processed so as to minimize the risk of unintentional or unauthorized access or use, and the nature of the authorized use should be clearly indicated to the user.

[0086] It will be apparent to those skilled in the art that various modifications can be made to the present disclosure without departing from the spirit or scope of the disclosure. Accordingly, the present disclosure is intended to cover modifications and variations of the present disclosure as long as they fall within the scope of the appended claims and their equivalents.

Claims

1. A method, comprising: Receiving a diagnostic indication, unstructured patient data (UPD), and a diagnostic report vocabulary list, wherein the diagnostic report vocabulary list includes ordered vocabulary terms; Generating a blank bitmap including a plurality of pixels, the plurality of pixels having (i) a number of pixels corresponding to the number of the vocabulary terms, and (ii) a correspondence between pixel positions in the bitmap and the order of the vocabulary terms; Generating a fingerprint, the fingerprint including a mapping of each occurrence of a vocabulary term in the UPD to a corresponding pixel position in the blank bitmap; and Training a convolutional neural network (CNN) using the fingerprint as an input and using the diagnostic indication as a target output.

2. The method according to claim 1, further comprising: Using a second UPD and the diagnostic report vocabulary list to generate a second fingerprint; Inputting the second fingerprint into the CNN; And Receiving a second diagnostic indication as an output from the CNN.

3. The method according to claim 1, further comprising: Receiving patient image data; Training an image-based CNN using the patient image data.

4. The method according to claim 3, further comprising: Using a second UPD and the diagnostic report vocabulary list to generate a second fingerprint; Receiving second patient image data corresponding to the second fingerprint; Inputting the second fingerprint and the second patient image data into an artificial intelligence (AI) model including the CNN and the image-based CNN; and Receiving a second diagnostic indication as an output from the AI model.

5. The method according to claim 1, wherein, The diagnostic report vocabulary list includes a medical dictionary.

6. The method according to claim 1, wherein The UPD includes text-based data.

7. The method according to claim 1, wherein The diagnostic indication includes clinical findings, recommendations, or conclusions.

8. The method according to claim 1, wherein Generating the fingerprint further includes changing a pixel value associated with at least one pixel position in the bitmap.

9. The method according to claim 7, wherein The pixel value varies between 0 and 255, including 0 and 255.

10. A method, comprising: Receiving unstructured patient data (UPD) and a diagnostic report vocabulary list, wherein the diagnostic report vocabulary list includes ordered vocabulary terms; Generating a blank bitmap including a plurality of pixels, the plurality of pixels having (i) a number of pixels corresponding to the number of the vocabulary terms, and (ii) a correspondence between pixel positions in the bitmap and the order of the vocabulary terms; Generating a first fingerprint by changing a pixel value associated with a corresponding pixel position in the bitmap, the first fingerprint including a mapping of each occurrence of a vocabulary term in the UPD to a corresponding position in the blank bitmap; and Displaying the fingerprint to a user.

11. The method according to claim 10, further comprising: Displaying the vocabulary term on the first fingerprint when the pixel value is equal to or exceeds a threshold.

12. The method according to claim 11, further comprising: Generating a link between the vocabulary term and the corresponding position of the vocabulary term in the UPD.

13. The method according to claim 11, wherein, The displayed vocabulary term includes a color related to a clinical concept.

14. The method according to claim 10, further comprising: Generate vectorized connections between multiple input vocabulary terms; And Display the vectorized connections on the first fingerprint.

15. The method according to claim 10, further comprising: Using a second UPD and the diagnostic report vocabulary list to generate a second fingerprint; Performing image subtraction between the second fingerprint and the first fingerprint; And Displaying the result of the image subtraction.

16. The method according to claim 10, wherein, The pixel values vary between 0 and 255, including 0 and 255.

17. A system for creating a data structure used in clinical diagnosis support, the system comprising: A memory including a diagnosis indication, unstructured patient data (UPD), and a diagnostic report vocabulary list, wherein the diagnostic report vocabulary list includes ordered vocabulary terms; and A processor configured to: Generate a blank bitmap including a plurality of pixels, the plurality of pixels having (i) a number of pixels corresponding to the number of the vocabulary terms, and (ii) a correspondence between the pixel positions in the bitmap and the order of the vocabulary terms; Generate a fingerprint including a mapping of each occurrence of a vocabulary term in the UPD to a corresponding pixel position in the blank bitmap; and Use the fingerprint as an input and use the diagnosis indication as a target output to train a convolutional neural network (CNN).

18. A system for creating a data structure used in clinical diagnosis support, the system comprising: A memory including unstructured patient data (UPD) and a diagnostic report vocabulary list, wherein the diagnostic report vocabulary list includes ordered vocabulary terms; A processor configured to: Generate a blank bitmap including a plurality of pixels, the plurality of pixels having (i) a number of pixels corresponding to the number of the vocabulary terms, and (ii) a correspondence between the pixel positions in the bitmap and the order of the vocabulary terms; Generate a first fingerprint by changing pixel values associated with corresponding pixel positions in the bitmap, the first fingerprint including a mapping of each occurrence of a vocabulary term in the UPD to a corresponding position in the blank bitmap; and A display for displaying the fingerprint to a user.

19. A computer program product that, when executed on a computer, is operable to perform the method according to claim 1.

20. A computer program product that, when executed on a computer, is operable to perform the method according to claim 10.

Images