Medical report writing techniques

Abstract

A method for generating a medical report for a patient is disclosed, the method including: (a) selecting at least one program module from a plurality of program modules having input requirements that match medical image-related data of the patient, (b) acquiring medical findings using the at least one program module, (c) selecting at least one program module from the plurality of program modules having input requirements that match the acquired medical findings, (d) acquiring medical findings based on previously acquired medical findings using the at least one program module, and (e) generating a medical report for the patient, the medical report including at least the acquired medical findings.

Description

[Technical Field]

[0001] The present disclosure relates generally to the field of medical data processing. In particular, techniques are presented for providing an interactive display of medical images based on a patient's medical report. Techniques for creating the medical report are also presented. The techniques may be embodied in methods, computer programs, and apparatuses. [Background technology]

[0002] In the medical field, patient medical reports containing medical findings about a patient are frequently used. Such medical reports allow a user (e.g., a doctor or other medical staff) to quickly grasp an overview of a patient's overall or specific medical condition. The user can then determine whether the patient needs treatment, whether the current treatment is effective, or how the patient should be medically treated. Therefore, in the medical field, medical reports are important because they help users make decisions and directly affect clinical outcomes.

[0003] Currently, medical reports are provided to users as printed documents. Because medical reports contain numerous medical findings, they can be lengthy and span several pages. Furthermore, in some cases, such as when a patient experiences an acute illness (e.g., organ failure or sepsis), a user's decision may be required within a relatively short time frame. Under such time pressure, it is obvious that a user cannot read the entire printed medical report. Furthermore, even if a user is only interested in a specific part of the medical report (e.g., a part related to a specific body part of the patient), it can be difficult to extract relevant information from the printed medical report. Furthermore, correlations between medical findings and the patient's body parts are rarely visible in such printed medical reports. Summary of the Invention

[0004] Therefore, there is a need for technical implementations that provide improved displays related to patient medical reports.

[0005] According to a first aspect, there is provided a method for providing an interactive display of medical images based on a patient's medical report. The method includes the step of (a) displaying the patient's medical image on a first portion of a display. The method further includes the step of (b) displaying a text representation of the medical finding on a second portion of the display in response to a user selecting a region in the displayed medical image that is associated with a medical finding included in the medical report. Alternatively or additionally to step (b), the method includes the step of (c) displaying a text representation of the medical finding on the second portion of the display, the medical finding being included in the medical report and associated with a region in the medical image, and displaying an indicator of the region on the first portion of the display in response to the user selecting the displayed text representation.

[0006] The method may display the text representation in step (b) and the indicator of the region in step (c) in response to a user selection. In other words, the display in steps (b) and (c) may be triggered by user input on a graphical user interface (GUI). Instead of simultaneously displaying all available regions and all representations of medical findings contained in the patient's medical report, the display may react in response to a user selection, thereby displaying relevant data (e.g., only that). That is, the relationship between a region in a medical image and a text representation of a medical finding associated with that region may be visualized to the user by responsively displaying relevant content in response to a user selection.

[0007] A medical image may be an image representing physical or anatomical properties of a patient's body. The medical image may be a computed tomography (CT) image, a magnetic resonance (MR) image, an X-ray image, a pathology tissue sample image, etc. The medical image may be an image generated based on medical measurements of a patient's anatomical properties, such as an electroencephalography (EEG) measurement, an electrocardiogram (ECG) measurement, blood analysis data, genetic data, etc.

[0008] The first and second portions of the display may be part of the same display unit, such as a screen. In other variations, the first portion may be part of a first display unit and the second portion may be part of a second display unit different from the first display unit.

[0009] The region may be selected in step (b) by the user selecting an indicator of the region, the indicator being displayed on the first portion of the display. The region may be selected in step (b) by the user tracing the outline of the region in the displayed image. The region may be one-dimensional, two-dimensional, or three-dimensional. That is, the region may be a point, a plane with a predetermined outline, or a volume with a predetermined outer surface. The region may be a region of interest (ROI). The text representation may be selected in step (c) by the user clicking or touching the text representation displayed on the second portion of the display.

[0010] The medical findings may be derived from or determined based on data describing a patient's body, e.g., medical images. The medical findings may be indicative of a medical condition, a medical abnormality, a disease, etc. The medical findings may include a region of interest (ROI) in a medical image or properties of the ROI (e.g., size, location, volume, shape, geometry, density, etc.). The medical findings may include properties of a medical image (e.g., of a portion such as an ROI), such as color, saturation, brightness, etc. The medical findings may include physical properties of tissue in the patient's body, such as (e.g., optical) density, somatic cell type, etc. The medical findings may be indicative of (e.g., indicate) a medical diagnosis. The medical findings may include identification of an anatomical body part in the patient's body.

[0011] In a more general variant, the term "clinically relevant information" may be used instead of the term "medical findings" generally referred to herein. Clinically relevant information may include medical findings in the above sense (as a subset of clinically relevant information), but may also include other clinically relevant information. Clinically relevant information may be patient-specific and / or medical image-specific. Clinically relevant information may, for example, indicate entities of interest to radiologists. Clinically relevant information may indicate RadLex entities, diagnoses or terms from the Radiology Gamuts Ontology (RGO), entries from the Human Phenotype Ontology, or entries according to other ontologies (e.g., available at https: / / bioportal.bioontology.org / ontologies). Clinically relevant information may indicate, specify, correspond to, or include one or more of: anatomical entities, clinical findings, image observations, image observations, image specialties, non-anatomical materials, objects, procedures, procedural steps, processes, properties, RadLex descriptors, RadLex non-anatomical sets, reports, report components (e.g., patient age), and temporal entities. Clinically relevant information may indicate, specify, correspond to, or include one or more higher-level or lower-level entries in the RadLex tree. RadLex is a comprehensive set of radiology terminology used in radiology reporting, decision support, data mining, data registration, education, and research. RadLex is developed and maintained by the Radiological Society of North America (RSNA).

[0012] A textual representation of the medical finding may be created based on the medical finding, which may be associated with textual data that includes or constitutes the textual representation of the medical finding.

[0013] The medical report may include multiple medical findings, may be a collection of medical findings related to a patient and / or may be derived from medical images of the patient, and may include a patient identifier.

[0014] An indicator of the region may be generated based on the region. The indicator may include a visual representation of the region. The indicator may include an outline (or "marking") of the region and / or filling the region with, for example, a particular color or visual pattern.

[0015] The method may further include updating the medical report by adding (e.g., a first) medical finding based on the user input in the medical report. In other words, the medical report may be configured by the user by adding the medical finding based on the user input. This allows the user to enrich the medical report.

[0016] Alternatively or additionally, the medical report may be updated by deleting one or more medical findings from the medical report based on user input. In other words, the medical report may be configured by the user by deleting medical findings based on user input. This allows the user to thin the medical report, for example, by deleting invalid medical findings that no longer match the patient and / or medical image.

[0017] The (e.g., first) added medical finding may be associated with a user-defined region in the displayed medical image. In this case, the (e.g., first) added medical finding may be determined based on or derived from the user-defined region in the displayed medical image. The user may define an ROI in the displayed medical image, and the (e.g., first) added medical finding may be determined autonomously (e.g., automatically) with respect to the ROI. This allows the user to specify a relevant or interesting region in the medical image and enrich the medical report with the (e.g., first) added medical finding associated therewith.

[0018] The method may further include displaying a plurality of indicators of different regions on the first portion of the display, and the (e.g., first) added medical finding may be associated with a group of regions, the group of regions including or consisting of a set of different regions selected by the user. In other words, the user may select a set of different regions, and the (e.g., first) added medical finding may be determined based on the set of different regions. The same medical finding may be added to the medical report for each of the different regions included in the set. Alternatively or additionally, the (e.g., first) added medical finding may be associated only with such a set, and not with each individual region included within the set. This allows a user to enrich the medical report by grouping related or interesting regions (e.g., only) and adding a (e.g., first) medical finding associated with the set.

[0019] The (e.g., first) added medical finding may be defined by a user. The user can define the (e.g., first) added medical finding by selecting the (e.g., first) added medical finding from a plurality of possible (e.g., predetermined, defined, feasible, or available) medical findings. Text representations of the plurality of possible medical findings may be displayed on a display, for example, in list form, to allow the user to select one or more of the possible medical findings. The (e.g., first) added medical finding selected by the user may be assigned to or associated with a user-defined region or set of different regions. The user can define the (e.g., first) added medical finding by entering a voice command or text input. The voice command or text input may be converted into or used as text data of the added medical finding, for example, a text representation of the (e.g., first) added medical finding. This allows the user to specify exactly (eg, type or properties thereof) of the (eg, first) added medical finding in order to update the medical report.

[0020] At least one additional text representation of a different medical finding associated with the region (e.g., another region in the medical image or the like) may be displayed together with the text representation of the medical finding in the second portion of the display, and the different medical finding may be included in the medical report. In other words, a region may have multiple associated medical findings, and their text representations may be displayed simultaneously with the text representation of the medical findings. This allows a user to learn about all medical findings included in the medical report that are associated with the region and / or about the region associated with the different medical finding represented by the additional text representation.

[0021] The method may further include, in response to the user specifying one or more of the text representation and the at least one additional text representation, updating the medical report by deleting from the medical report any (e.g., different) medical findings represented by text representations not specified by the user. This allows the user to select (e.g., only) correct and / or relevant (e.g., different) medical findings to remain in the medical report. In another variation, the method may further include, in response to the user specifying one or more of the text representation and the at least one additional text representation, updating the medical report by deleting from the medical report any (e.g., different) medical findings represented by the text representations specified by the user. This allows the user to select (e.g., only) incorrect and / or irrelevant (e.g., different) medical findings to be deleted from the medical report.

[0022] The method may further include updating the medical report by adding a (e.g., second) medical finding in the medical report, where the (e.g., second) medical finding is determined by an artificial intelligence (AI) program module. The (e.g., second) added medical finding may be the (e.g., first) added medical finding described above and / or may be determined by the AI ​​module based on user input (e.g., a user-defined region in the medical image). The AI ​​program module may be configured to autonomously determine the (e.g., second) added medical finding based on one or more medical findings included in the medical report, and in some cases, further based on user input. Alternatively, or additionally, the AI ​​program module may be configured to autonomously determine the (e.g., second) added medical finding based on the medical image. This allows for autonomous or automatic enrichment of the medical report with the additional medical finding.

[0023] The method may further include using at least one of the specified text representation and the medical finding represented by the specified text representation as training data for training an AI module. Alternatively or additionally, the method may include using at least one of the (e.g., first and / or second) added medical findings and the text representation of the (e.g., first and / or second) added medical findings as training data for training the AI ​​module. A machine learning algorithm may be used to provide or improve the AI ​​module based on the training data. The AI ​​module may include an artificial neural network, an ensemble of (e.g., AI) program modules, a support vector machine, etc. This may improve the AI ​​module so that the reliability of the (e.g., second) medical finding determined by the AI ​​module can be increased.

[0024] At least one of steps (b) and (c) may be repeated after updating the medical report, thereby allowing the indicator and / or text representation(s) included in the updated medical report to be displayed. In other words, the displayed content may be updated as the medical report is updated, thereby informing the user of the current content of the medical report and ensuring that the user is up to date.

[0025] A medical finding may be stored as a (e.g., first) node of a graph in the graph database. The (e.g., first) node may have a label matching a type of the medical finding. The (e.g., first) node may have at least one property specifying details of the medical finding and / or a textual representation of the medical finding. A region related to the medical finding may be stored as a (e.g., second) node of the graph. The (e.g., second) node may have a label (e.g., "ROI") matching the region related to the medical finding. The (e.g., second) node may have at least one property specifying details of the region related to the medical finding and / or an indicator of the region related to the medical finding. The (e.g., second) node representing the region related to the medical finding may be connected or linked to the (e.g., first) node representing the medical finding in the graph via a (e.g., first) edge of the graph. The (e.g., first) edge may have a label matching the first node and the second node.

[0026] For example, the added medical finding (first or second) may be stored as a (e.g., third) node of a graph in the graph database. The (e.g., third) node may have a label matching the type of the added medical finding. The (e.g., third) node may have at least one property identifying details of the added medical finding and / or a textual representation of the added medical finding. A region associated with the added medical finding may be stored as a (e.g., fourth) node of the graph. The (e.g., fourth) node may have a label (e.g., "ROI") matching the region associated with the added medical finding. The (e.g., fourth) node may have at least one property specifying details of the region associated with the added medical finding and / or an indicator of the region associated with the added medical finding. The (e.g., fourth) node representing the region associated with the added medical finding may be connected or linked to the (e.g., third) node representing the added medical finding in the graph via a (e.g., second) edge of the graph. The (eg, second) edge may have labels that match the third and fourth nodes.

[0027] The medical report may be generated based on the graph, either based on all nodes of the graph or based on a pre-defined subset of nodes of the graph, which allows for fast updating of the medical report and fast display of the text representation in step (b) and the area indicator in step (c).

[0028] According to a second aspect, there is provided an apparatus including at least one processor and at least one memory, the at least one memory including instructions executable by the at least one processor such that the apparatus is operable to perform the method of the first aspect.

[0029] According to a third aspect, a computer program product is provided. The computer program product includes program code portions for performing the method of the first aspect when the computer program product is executed on one or more processors (e.g., at least one processor of the apparatus of the second aspect). The computer program product may be stored on one or more computer-readable recording media. The computer program product may be carried by a data carrier signal, such as a digital signal stream.

[0030] According to a fourth aspect, there is provided a method for generating a medical report for a patient, the method including: (i) in response to acquiring medical image-related data of the patient, selecting at least one program module from a plurality of program modules having input requirements that match the medical image-related data of the patient, (ii) acquiring a medical finding based on the medical image-related data of the patient using the at least one program module selected in step (i), (iii) selecting at least one program module from the plurality of program modules having input requirements that match previously acquired medical findings, (iv) acquiring a medical finding based on the previously acquired medical findings using the at least one program module selected in step (iii), and (v) generating a medical report for the patient, the medical report including at least one of the acquired medical findings.

[0031] This allows multiple medical findings to be determined based on the patient's medical image-related data, and a medical report to be generated based thereon. Selecting an appropriate (e.g., matching) program module in steps (i) and (iii) can improve the reliability of the acquired medical findings. It can also be ensured that the selected program module can provide medical findings based on the respective input data (e.g., medical image-related data and / or previously acquired medical findings). In other words, the smooth execution of method steps (i) to (v) without interruption due to the selected program module being unable to provide a medical finding can be avoided. Selecting an appropriate program module in steps (i) and (iii) can generate a chain of program modules, whereby a subsequent selected program module can provide a medical finding based on previously acquired medical findings. That is, the chain of program modules can provide multiple medical findings, each derived from another. This may allow the use of highly specialized program modules to improve individually acquired medical findings. Additionally, program modules may be readily replaced with newer or improved versions without adversely affecting the method of the fourth aspect.

[0032] The method of the fourth aspect may be part of or combined with the method of the first aspect, or vice versa. In particular, individual steps or all steps of the method of the fourth aspect may be combined with individual steps or all steps of the method of the first aspect. For example, the medical report described above for the first aspect may be the medical report described herein with reference to the fourth aspect. The (e.g., additional) medical finding(s) described above for the first aspect may be the medical finding(s) described herein with reference to the fourth aspect.

[0033] The patient's medical image related data may include at least one of the patient's medical image (e.g., the medical image described above in relation to the first aspect), a region of interest (ROI) in the patient's medical image (e.g., the region or ROI described above in relation to the first aspect), properties of the ROI (e.g., the properties described above in relation to the first aspect), and medical findings derived from the patient's medical image (e.g., the medical findings described above in relation to the first aspect or additional medical findings).

[0034] Each of the multiple program modules may be triggered (or "launched" / "started") individually. A selected program module may be triggered in response to its selection in step (i) or (iii). Step (ii) may be performed in response to the selection in step (i). Step (iv) may be performed in response to the selection in step (iii). Each of the multiple program modules may be executed on a variety of computing environments, such as, for example, a local computer, a server, or the cloud.

[0035] The input requirements of the at least one program module may be properties of the at least one program module. The input requirements may be obtained from a provider of the at least one program module. The input requirements may specify the type or content of data that the at least one program module requires (e.g., as input for providing a medical finding). The input requirements may specify the type or content of data that the at least one program module can use as input (e.g., for providing a medical finding). The input requirements may specify the type or content of data that the at least one program module requires to provide (e.g., based on) a medical finding.

[0036] Step (ii) may include providing patient medical image-related data to at least one program module selected in step (i). Step (ii) may include triggering or initiating execution of at least one program module selected in step (i). Step (iv) may include providing previously acquired medical findings (e.g., medical findings acquired in step (ii)) to at least one program module selected in step (iii). Step (iv) may include triggering or initiating execution of at least one program module selected in step (iii).

[0037] In step (iv), a medical report may be generated based on the medical findings as described above for the first aspect. The medical report may be generated by including all acquired medical findings in the medical report, or by including all medical findings of a particular nature or type in the medical report. Alternatively, the medical report may be generated by including the most recent acquired medical findings in the medical report.

[0038] Steps (iii) and (iv) may be repeated at least once. This allows more medical findings to be obtained. Thus, the generated medical report may be enriched with more medical findings. In step (iv), the selected at least one program module may be used to obtain a medical finding based on multiple previously obtained medical findings. This allows the level of detail or "depth" of the obtained medical findings to be increased with each repetition of steps (iii) and (iv), since these steps may use previously determined medical findings(s). Again, it should be noted that the medical report used in the method of the first aspect may be the medical report generated in step (v). Steps (b) and / or (c) may be performed or repeated after step (v). The content displayed in steps (b) and (c) (e.g., the text representation(s) and / or the region indicator) may be updated in response to step (v).

[0039] Steps (iii) and (iv) may be repeated at least once before generating a medical report in step (v). In this case, when step (iii) is repeated, the previously obtained medical findings referred to in step (iii) may be the medical findings obtained in step (ii) or the medical findings previously obtained in step (iv) (e.g., obtained before repeating steps (iii) and (iv)). Furthermore, when step (iv) is repeated, the previously obtained medical findings referred to in step (iv) may be the medical findings obtained referred to in step (iii).

[0040] Steps (iii), (iv), and (v) may be repeated at least once after generating the medical report in step (v). In this case, when repeating step (iii), the obtained medical findings referred to in step (iii) may be medical findings included in a previously generated medical report (e.g., generated in step (v) before repeating steps (iii) and (iv)) and / or medical findings previously obtained in step (iv). Furthermore, when repeating step (iv), the previously obtained medical findings referred to in step (iv) may be medical findings obtained referred to in (e.g., repeated) step (iii). After repeating steps (iii) and (iv), step (v) of generating the medical report may include or consist of updating the previously generated medical report.

[0041] At least one program module selected in step (iii) may have input requirements that match a predetermined subset of (e.g., previously) acquired medical findings or all of (e.g., previously) acquired medical findings when repeating step (iii). In other words, the input requirements of at least one program module selected when repeating step (iii) may match some or all of the previously acquired medical findings, for example, the medical findings acquired in step (ii) and the medical findings acquired when performing step (iv). In this case, at least one program module selected when repeating step (iii) may be used to acquire a medical finding (e.g., a predetermined subset) based on the medical findings that match the input requirements. This allows multiple medical findings to be used as input data to select an appropriate program module. Such a program module may combine multiple medical findings provided as input and provide a medical finding based on the combination. This can improve the reliability and specificity of the acquired medical findings.

[0042] Some or all of the selected at least one program module may be used to obtain different types of medical findings. For example, a first module may be configured to provide an ROI, a second module may be used to obtain an anatomical body part identification, and a third program module may be configured to determine a medical diagnostic indication. This allows multiple different types of medical findings to be obtained based on the same input data (e.g., based on medical image-related data in step (ii) or based on previously obtained medical findings in step (iv)). Therefore, the medical report generated in step (v) may include different types of medical findings. This improves the usefulness of the medical report, and allows a user to more reliably determine whether to treat a patient and how to treat the patient, particularly when steps (b) and / or (c) of the method of the first aspect are performed using such a medical report.

[0043] One or more of the at least one program modules selected in step (i) may be configured, for example, in response to being selected, to autonomously (e.g., automatically) determine a medical finding based on the acquired medical image-related data. Alternatively, or additionally, one or more of the at least one program modules selected in step (iii) may be configured, for example, in response to being selected, to autonomously determine a medical finding based on previously acquired medical findings. One or more of the selected at least one program modules may be an artificial intelligence (AI) module (e.g., an AI module described above for the first aspect, or an AI module including one or more functions of the AI ​​module described above for the method of the first aspect). This allows for rapid determination of a medical finding.

[0044] If multiple AI modules providing the same type of medical findings are selected in step (i) or (iii), the method may further include combining the multiple AI modules into an ensemble and obtaining a medical finding using the ensemble, thereby improving the reliability of the obtained medical finding.

[0045] The medical findings obtained using the ensemble may correspond to a determined indication of a medical condition as described in European Patent Application EP20159958.6, filed February 28, 2020. In this case, the program modules described herein may correspond to the models described in the aforementioned European Patent Application. Furthermore, in this case, the selected AI modules described herein may correspond to at least one selected model described in the aforementioned European Patent Application.

[0046] One or more of the at least one program modules selected in step (i) may be a user interface (UI) module that requests user input defining a medical finding (or, for example, a region related to the medical finding) in step (ii). Alternatively, or additionally, one or more of the at least one program modules selected in step (iii) may be a user interface (UI) module that requests user input defining a medical finding (or, for example, a region related to the medical finding) in step (iv). Such UI modules may provide the functionality described above with respect to the first aspect. In particular, such UI modules may perform one or more of steps (a), (b), and (c). The medical findings added based on the user input described with respect to the first aspect may correspond to the medical findings requested by the UI module. In other words, the medical findings “added” based on the user input described with respect to the first aspect may correspond to the medical findings obtained using the UI module. In other words, the UI module may be configured to request user input from a user, and the UI module may be used to obtain the user input as a medical finding, or to obtain the medical finding by determining the medical finding based on the user input. The medical finding obtained using the UI module may then be included in the medical report in step (v), thereby "adding" the medical finding to the medical report. This allows the user to influence the medical finding used to generate the medical report. In other words, the generated medical report may be improved or updated based on the user input.

[0047] If (e.g., only if) the plurality of program modules does not include a program module configured to autonomously determine medical findings, at least one selected program module may be one or more UI modules. In this manner, acquisition of medical findings may be ensured by relying on user input (e.g., definition of medical findings). In this manner, a complete medical report including a predetermined minimum number (e.g., types) of medical findings may be generated.

[0048] The method may further include training an artificial intelligence (AI) module (for example) of the plurality of program modules with the medical findings (e.g., their labels and / or properties) obtained using one or more UI modules. As described above with reference to the first aspect, this may improve the reliability of the medical findings obtained using the AI ​​module.

[0049] In certain variations, the method may further include, in step (a), if a module configured to autonomously determine a medical finding is selected, displaying visualization information of the medical finding determined by the selection module configured to autonomously determine a medical finding, and hiding the visualization information if a UI module providing the same type of medical finding as the selection module configured to autonomously determine a medical finding is also selected in step (a). Alternatively or additionally, the method may include, in step (c), if a module configured to autonomously determine a medical finding is selected, displaying visualization information of the medical finding determined by the selection module configured to autonomously determine a medical finding, and hiding the visualization information if a UI module providing the same type of medical finding as the selection module configured to autonomously determine a medical finding is also selected in step (c). The selected module configured to autonomously determine a medical finding may be an artificial intelligence (AI) module. The method may further include the selected UI module providing the same type of medical findings as the selected AI module to train the selected AI module with the obtained medical findings.

[0050] The selection of the at least one program module in steps (i) and / or (iii) may be performed by an artificial intelligence (AI) selection module. The AI ​​selection module may match input requirements of the program modules or select at least one preferred module from among multiple program modules having matching input requirements. The AI ​​selection module may select the at least one program module based on the output (e.g., type of medical findings) provided by the program modules. The AI ​​selection module may select a program module that provides a preferred output based on medical findings included in multiple predetermined medical reports (e.g., created by a user). This allows the selection of a program module that provides an output that conforms to the multiple predetermined medical reports, thereby enabling the creation of consistent medical reports.

[0051] The method may further include training an AI selection module using some or all of the obtained medical findings. The AI ​​selection module may be trained based on the medical findings (e.g., contained in a plurality of predetermined medical reports), thereby improving the performance of the AI ​​selection module and generating consistent medical reports.

[0052] In one particular variation, the AI ​​selection module may be trained using reinforcement learning.

[0053] The method of the fourth aspect may further include storing each acquired medical finding as a node of a graph in a graph database. Each acquired medical finding may be stored as a separate node of the graph. For example, the acquired medical finding in step (iv) may be stored as a node connected to a node representing a previously acquired medical finding used to acquire the medical finding in step (iv). The acquired medical finding in step (ii) may also be stored as a node connected to a node representing image-related data for the patient. That is, the graph may represent dependencies of the acquired medical findings and / or identify input data used to acquire the medical finding. The graph may include a chain of nodes representing the acquired medical findings, where each node representing the acquired medical findings may be created, generated, or determined based on the previous node in the chain. This allows for fast storage and retrieval of the acquired medical findings, enabling fast and reliable generation of the medical report in step (v).

[0054] The graph may correspond to the graph described above for the first aspect. The nodes may correspond to the nodes described above for the first aspect. Similarly, the nodes may be connected via edges. Each of the acquired medical findings may correspond to one of the medical findings described above for the first aspect and the added medical findings (e.g., first or second). The user input used to add the medical findings described above for the first aspect may be required by the UI module to acquire the medical findings. Updating the medical report described above for the first aspect may correspond to performing step (ii) or (iv) (e.g., using the UI module) and then performing step (v).

[0055] According to a fifth aspect, there is provided an apparatus comprising at least one processor and at least one memory, the at least one memory comprising instructions executable by the at least one processor such that the apparatus is operable to perform the method of the fourth aspect.

[0056] According to a sixth aspect, a computer program product is provided. The computer program product includes program code portions for performing the method of the fourth aspect when the computer program product is executed on one or more processors (e.g., at least one processor of the apparatus of the fifth aspect). The computer program product may be stored on one or more computer-readable recording media. The computer program product may be carried by a data-carrying signal, such as a digital signal stream.

[0057] When the term "based on" is used herein, it can mean "based on at least" in one variation and "based only on" in another variation. When "at least one" is referred to, it can mean "exactly one," "only one," or "two or more." [Brief explanation of the drawings]

[0058] Further details and advantages of the techniques presented herein will be explained with reference to the exemplary embodiments illustrated in the figures, in which: [Figure 1] FIG. 1 is a diagram illustrating an exemplary configuration of an apparatus according to the present disclosure. [Figure 2] FIG. 2 illustrates a method for providing an interactive display of medical images that may be performed by an apparatus according to the present disclosure. [Figure 3] FIG. 3 illustrates a method for generating a patient medical report that may be performed by an apparatus according to the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example node specification database according to the present disclosure. [Figure 5]FIG. 5 illustrates an exemplary edge specification database according to the present disclosure. [Figure 6] FIG. 6 illustrates an exemplary graph according to the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating a method and its associated components according to the present disclosure. [Figure 8a] FIG. 8a is a diagram illustrating a display according to the present disclosure. [Figure 8b] FIG. 8b illustrates another display according to the present disclosure. [Figure 9] FIG. 9 is a diagram illustrating a method and its associated components according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0059] In the following description, for purposes of explanation and not limitation, specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent to those skilled in the art that the present disclosure may be practiced in other embodiments that depart from these specific details.

[0060] Those skilled in the art will further appreciate that the steps, services, and functions described below may be implemented using discrete hardware circuits, using software working in conjunction with a programmed microprocessor or general-purpose computer, using one or more application-specific integrated circuits (ASICs), and / or using one or more digital signal processors (DSPs). It will also be understood that when the present disclosure is described in terms of methods, they may also be embodied with one or more processors and one or more memories coupled to the one or more processors, the one or more memories being encoded with one or more programs that, when executed by the one or more processors, perform the steps, services, and functions disclosed herein.

[0061] FIG. 1 illustrates an exemplary configuration of an apparatus 100 according to the present disclosure. The apparatus 100 includes a processor 102, a memory 104, and, optionally, an interface 106. The processor 102 is connected to the memory 104 and, optionally, to the interface 106. The interface 106 is configured to acquire, receive, send, or transmit data from or to an external device, such as a data storage device, a server, a user input device, an output device such as a display unit 108, or a speaker. The interface 106 may be configured to send a trigger signal to the output unit to trigger the output of an acoustic and / or optical notification (message) to a user. The memory 104 is configured to store a program including instructions that, when executed by the processor 102, cause the processor 102 to perform the methods described herein. The program may be stored in a storage medium. The apparatus 100 may be connected to the display unit 108, for example, via the interface 106. Alternatively, the display unit 108 may be part of the apparatus 100.

[0062] FIG. 2 illustrates an exemplary method according to the present disclosure. The method may be performed by the device 100. The method of FIG. 2 may correspond to the method of the first aspect described above. The method includes step 202 of displaying a medical image of a patient on a first portion of a display, e.g., the display unit 108. The method includes at least one of steps 204 and 206. In step 204, in response to a user selecting an area in the displayed medical image that is associated with a medical finding included in the patient's medical report, a textual representation of the medical finding is displayed in a second portion of the display. In step 206, a textual representation of the medical finding is displayed in the second portion of the display, e.g., the above-mentioned, the medical finding being included in the patient's medical report and associated with the area in the medical image, and an indicator of the area is displayed in the first portion of the display, in response to a user selecting the displayed textual representation. The second portion may be different from the first portion.

[0063] FIG. 3 illustrates an exemplary method according to the present disclosure. The method may be performed by the apparatus 100. The method of FIG. 3 may correspond to the method of the fourth aspect described above. The method includes step 302 of selecting, in response to acquiring medical image-related data of a patient, at least one program module from a plurality of program modules having input requirements that match the medical image-related data of the patient. The method further includes step 304 of acquiring a medical finding based on the medical image-related data of the patient using the at least one program module selected in step 302. The method includes step 306 of selecting, from the plurality of program modules, at least one program module having input requirements that match the acquired medical finding. The method further includes step 308 of acquiring a medical finding based on a previously acquired medical finding using the at least one program module selected in step 306. The method includes step 310 of generating a medical report for the patient, the medical report including at least one (e.g., the most recent) of the acquired medical findings.

[0064] The method shown in Figure 2 may be combined with the method shown in Figure 3. One or more of steps 202-206 may be performed in combination with one or more of steps 302-310. When reference is made below to a "method," unless otherwise stated or apparent from the following description, the following description may relate to one or both of the methods of Figures 2 and 3, or to combinations of individual steps of these methods that constitute other methods.

[0065] In general, the methods described herein can be implemented using a graph that includes nodes connected by edges, and the graph may be stored in a graph database.

[0066] Node properties and constraints may be stored in a node specification database. In other words, a graph may include (e.g., only) nodes that have properties and satisfy constraints defined in the node specification database. Edge properties and constraints may be stored in an edge specification database. In other words, a graph may include (e.g., only) edges that have properties and satisfy constraints defined in the edge specification database. Additional node properties and constraints may be added to the node specification database, and additional edge properties and constraints may be added to the edge specification database. In other words, the node specification database and edge specification database may describe conditions that the nodes and edges of the graph must satisfy.

[0067] The node specification database and the edge specification database may be part of the same condition database. The node specification database, the edge specification database, and the graph database may be implemented centrally. One or more of the node specification database, the edge specification database, and the graph database may be stored in different locations. The node specification database, the edge specification database, and the graph database may be implemented in a fully decentralized manner using distributed ledger technology, using a proof-of-stake mechanism, a work mechanism, an authority mechanism, etc. to manage access control. As an example, a node labeled "CT" may be stored in a patient database, and a node labeled "report" may be stored in a hospital server. An edge connecting a node labeled "CT" and a node labeled "report" may store either the edge "CT" or "report," or both, to connect the two nodes as appropriate.

[0068] Each node may contain or be tagged with a label indicating the type of node (e.g., "Patient", "CT", "Finding"). Each node may contain, for example, a key-value pair (e.g., "Name: <name>"). Each edge may have at least one property that is a ':HAD_AN_EXAMINATION', ':HAS_REPORT') that indicates the type of edge or the correlation between the start and end nodes connected by the edge. Each edge may include at least one property (e.g., weight, cost, or further information). The at least one property of an edge may include metadata about the connection provided by the edge, or a weight that indicates, for example, a probability. An edge may have a direction. For example, the direction may indicate that the connection only makes sense along a particular direction and not the other way around, such as that a patient underwent a CT-type exam.

[0069] For example, a node with the label "CT," which is a node having a computed tomography (CT) image as a property, may be connected to a node with the label "Disease" via an edge with a weight indicating a confidence in the correctness of this connection, at least as a property. As another example, a node with the label "Patient" may be connected to another node with the label "Patient" via an edge with the label "IS_SIMILAR_TO." In this case, the weight assigned to the edge with the label "IS_SIMILAR_TO" may indicate the degree of similarity between the two patients represented by the two nodes with the label "Patient."

[0070] FIG. 4 illustrates an exemplary node specification database 400 including properties and constraints for multiple nodes, each with a unique label 402-416. The properties include a unique identification tag ("ID"), the patient's full name ("Name"), the patient's birth date ("BirthDate"), a past illness ("prior_disease"), etc. Each node's constraints may include constraints on one or more of the respective node's properties. Constraints include, but are not limited to, a unique node property constraint, a node property existence constraint, and a node property type constraint. As shown in FIG. 4 for a node with label 402, a unique node property constraint may be, for example, that the property "ID" in a node with the label "Patient" must be unique across all existing nodes with the label "Patient." A node property existence constraint may require that a particular property be filled in. As shown in FIG. 4, the node property existence constraint for the node with label 402 may specify that the field "Name" must be filled in and cannot be left empty. A node property type constraint may require that entries for a particular property of a node be of a particular type (eg, a string, an integer, a date, or an address to a data store).

[0071] In summary, the node specification database 400 can contain all possible nodes for which information can be filled in. Each element in the node database describes a node and, among other things, includes a label indicating the type of node (e.g., "Patient", "CT", "Report"), a list of properties associated with that type of node (e.g., "ID:<string>", "Name:<string>", "Birthdate: <ddmmyyy>"), which may describe at least one constraint associated with that node type.

[0072] FIG. 5 illustrates an exemplary edge specification database 500 including definitions of multiple edges, each with a unique label 502-516. The edge specification database 500 may include edge property existence constraints and / or edge property type constraints, similarly as described above for nodes in the node specification database 400. The edge specification database 500 may also include edge assignment constraints that define the label or type of the starting node (and, e.g., the label or type of the ending node). An edge assignment constraint may require that an edge with a particular label connects only nodes of a particular type; for example, an edge with the label ":HAD_AN_EXAMINATON" may only connect a starting node of type "Patient" with target nodes "CT," "MR," etc. Further edge assignment constraints may only allow one source (e.g., starting) node to connect one target (e.g., ending) node; a source node may not be able to connect multiple target nodes, or multiple source nodes and the same target node. In other words, edge specification database 500 may define, for each edge, the possible types of start nodes and the possible types of end nodes. For example, an edge with label 502 need only connect a node with label 412 as its start node and a node with label 416 as its end node, an edge with label 508 need only connect a node with label 406 and a node with label 414 as its end nodes, etc. Edge specification database 500 may also include properties of edges, as shown for edges 512-516, which have a property "date" that defines the date the end node (e.g., its properties) was created.

[0073] In short, the edge specification database 500 may contain all possible edges that can be used to link nodes of a particular type. Each element in the edge specification database describes an edge, in particular a label indicating the type of edge (e.g., "HAD_AN_EXAMINATION", "IS_BASED_ON"), a list of properties associated with the type of edge (e.g., "Date: <ddmmyyyy>"), the type of input node (e.g., start node) required by the edge, the type of output node (e.g., end node) required by the edge, and / or any constraints associated with the edge type.

[0074] In the example shown in Figure 5, note that the edge ":HAD_AN_EXAMINATION" may have the constraint that a patient can have multiple CTs (1: many), but a single CT cannot be assigned to multiple patients. Multiple regions of interest (ROIs) may be associated with the same local finding (many: 1). Multiple nodes of local or global finding types may point to one impression (many types: 1).

[0075] The nodes and edges of the graph may be completely defined by the types of nodes specified in node database 400 and the types of edges specified in edge database 500. Node database 400 and edge database 500 may be changed (e.g., dynamically) by modifying their contents, for example, by adding additional types of nodes or additional edges. A user may delete particular (e.g., types) of nodes, edges, properties, and constraints in databases 400 and / or 500 and replace them with others. Editing of databases 400 and 500 may be performed via a graphical user interface (GUI). Different groups of people may have different access rights to modify the contents of databases 400 and 500.

[0076] Databases 400 and 500 may execute queries. The command may be "LIST_ALL_EDGES WITH SOURCE NODE of type 'Patient'" or "LIST_ALL_EDGES WITH TARGET NODE of type 'Finding'." Such a query may result in the creation of a graph representing a medical report. The corresponding graph may be generated or created only if the query is valid with respect to node specification database 400 and edge specification database 500.

[0077] 6 shows an example of a graph 600 created using node specification database 400 and edge specification database 500. The nodes of graph 600 have properties and satisfy constraints defined in node specification database 400. The edges of graph 600 conform to the edge definitions stored in edge specification database 500.

[0078] In this example, CT image data was acquired for a patient. The CT image data may include medical images that may show two regions of interest (ROIs), only one of which may be found to be a local finding, while the other may be considered normal and not associated with the local finding. Additionally, the patient may exhibit a global finding, such as atrophy. The local and global findings, together with the patient's medical history, may constitute a statement (e.g., "The patient is not healthy, but is suffering from disease UVW, and treatment XYZ is recommended").

[0079] A program module may be provided that can infer new portions of graph 600 based on other portions of graph 600. In other words, the program module may use a particular portion of an existing graph to determine new nodes to be included in or added to the graph. The module may obtain some or all properties of the input nodes to determine or provide an output, e.g., create an output node. The modules may be stored in a module database.

[0080] In certain aspects, at least some (e.g., all) of the modules may use the same type of input. At least some (e.g., all) of the modules may generate the same type of output. The types of inputs and / or output types may be standardized, for example, as part of a predefined application programming interface (API). The modules may be developed, for example, using a software development kit (SDK) that supports the predefined API. This may ensure that modules from multiple software vendors can be used with the method(s) described herein. Alternatively or additionally, some or all of the modules may be stored and / or executed on one or more servers. Some or all of the modules may be executed on one or more virtual machines. Some or all of the modules may be included in one or more Docker containers. Such containers may provide lightweight virtualization.

[0081] The module may be triggered fully automatically whenever the required information (e.g., the required node) is available in graph 600. The module may alternatively be triggered by user interaction or by metamodel logic, also referred to herein as an AI selection module.

[0082] In one particular variation, the AI ​​selection module may be a recommender system. The AI ​​selection module may be trained using reinforcement learning.

[0083] A program module may be a user interface (UI) module that allows a human to provide user input, an artificial intelligence (AI) module, an ensemble of AI modules, or other software product that autonomously generates output. A program module may also be a database entry in a graph database. A program module may be executed on an external system that can communicate with device 100, for example, via interface 106.

[0084] Each program module may have input requirements that specify the type of input node it requires. Each program module can specify the output it generates (e.g., the type of node it generates). Modules can have a variety of output types, including modules that output a single filled node (e.g., a node with a label and properties) without edge connections, and modules that output filled nodes and connections that specify how the filled node is added to the input node. Modules may also specify additional constraints. For example, a module may only be able to generate specified outputs based on certain properties of the input nodes. For example, a module may be usable only if an input node of type "Patient" has the property "Gender = 'Female'". Output generated by a program module can be automatically added to an existing graph if the output conforms to the node specification database 400 and edge specification database 500, particularly the constraints specified therein. If not, the program module may return a warning.

[0085] The program modules are capable of performing database queries, such as radiology information systems (RIS) and picture archiving and communication systems (PACS).

[0086] For example, if a node labeled "Patient" is created, a program module (e.g., named or labeled "QUERY_FROM_RIS") may query the patient's medical history from the RIS system. The program module may perform further processing or aggregation, creating a node labeled "Medical History" and an edge labeled "HAS_HISTORY" and attaching the same to the node labeled "Patient," thereby adding the new node and edge to the graph.

[0087] As another example, if a node labeled "Patient" is created, a program module (e.g., named or labeled "QUERY_FROM_CT_FROM_PACS") may query the PACS system for the last CT examination performed for the patient with the respective ID, store it in object storage, output a node labeled "CT", and add the address to this object storage to the property "Address" of the output node.

[0088] The module may represent a user interface (UI) or graphical user interface (GUI) module for querying information from a user.

[0089] A simple example would be a text field that may be displayed (e.g., on display 108) along with a CT study (e.g., a CT image of a patient). When text is entered into the text field by a user, this text may be attached to a node labeled "CT" as a property of a node labeled "Image Analysis Results" that is connected to the node labeled "CT" via an edge.

[0090] Another example may be a program module that specifies a CT image as an input requirement. The module may display the image in a viewer (e.g., in a first portion of display unit 108) and query a click from the user. The coordinates of the click may be attached as a property of a node labeled "ROI" to a node labeled "CT" via an edge labeled ":SHOWS" (see also FIG. 6).

[0091] Another example may be a program module that identifies as input requirements a CT image and multiple regions of interest (e.g., regions constituting multiple sclerosis (MS) lesions). This module may display the CT image with the regions of interest as an overlay map in a viewer (e.g., in a first portion of the display unit 108) and query a user to mark and group multiple regions of interest into "findings." The "findings" may be labels for nodes in a graph, whose connections may be added or modified based on the user's groupings, for example, by linking several nodes labeled "ROI" corresponding to the marked ROIs to a node labeled "findings."

[0092] Other user interactions may be voice. For example, the module may define providing CT images as an input requirement. The module may query the user to formulate the user's sentiment (e.g., "the user is healthy"), convert it (e.g., via natural language processing (NLP)) into filled nodes and edges, and append the same to the graph.

[0093] The modules may perform algorithmic operations to generate outputs, for example, based on machine learning algorithms.

[0094] For example, a module may require a node labeled "CT" as input. The module may run a general anomaly detector and output multiple regions of interest, with each ROI appended as a separate node labeled "ROI" to the node labeled "CT." The anomaly detector may be pre-trained.

[0095] As another example, a module (e.g., labeled "ATLAS_AI") may require a node labeled "ROI" as input, determine the anatomical location of a finding via atlas matching based on the properties of the node labeled "ROI," and add the location to the node "ROI" as a property of an additional node labeled "LOC."

[0096] For example, a module may combine several types of modules, such as a module that performs a database query in a first step, then performs an algorithmic calculation based on the queried information, and fills or adds nodes to the graph. As another example, a module may perform an algorithmic calculation based on a CT image, and then request confirmation of the determined ROI by a user via a user interface, and then fills or adds nodes to the graph.

[0097] As described above, modules may be stored in a module database. The module database may itself be queried, e.g., for a given input node, it may output all modules that require this type of input node. For example, a query for the input node "CT" may return the following modules: "PERFORM_ANOMALY_DETECTION," which may be an algorithm module; "PERFORM_BLEED_SEGMENTATION," which may be an algorithm module; and "USER_ANOMALY_SEGMENTATION," which may be a UI module and prompts the user to segment an anomaly in a CT image. As described above, the CT image may be included as a property of the node "CT" or may be linked to the node "CT" by a property of the node "CT."

[0098] A query to the module database may be made each time the graph changes, for example, when an additional node is added. A query may be made and available modules may be returned based on the current state of the graph. In other words, at least one of the modules may be selected that has input requirements that match (e.g., at least) the additional node that was added. The input requirements may specify multiple node types that are required. A query of the module database may return all modules that have input requirements that match a subgraph of the current graph. Each returned or selected module may include information about how it is triggered. In response to such a query, a list of available modules may be returned, and depending on their respective module types (e.g., UI modules, AI modules, other algorithmic modules), they may be triggered differently.

[0099] That is, the selected (e.g., returned) module may be triggered in different ways. Options include automatic triggering, triggering via a metamodel, and triggering via user interaction.

[0100] Modules may be automatically triggered as soon as the inputs they require become available in the graph. For example, if an algorithm module configured to perform general anomaly detection in CT images requires a node labeled "CT" that has CT images as a property, the module may be automatically triggered.

[0101] Modules may be triggered via logic within the metamodel (e.g., via an AI selection module). By using existing data, e.g., multiple previously created graphs, a machine learning metamodel may be trained that learns to recommend specific modules and may trigger them automatically. For example, based on existing data, the metamodel may learn that the "Alzheimer's" detection module should always be run for patients over 70 years old, but that for younger patients, this model may be omitted.

[0102] A module may be triggered by user interaction. As an example, if a user draws a rectangle on a CT image to define a region (e.g., ROI), an algorithm module may be triggered to search for abnormalities in that particular region. A simple example of a trigger by user interaction would be a simple button clicked by the user.

[0103] Another related type of user interaction may be an “autofill” function. For example, a module database query may be executed when the user places the cursor over a highlighted anomaly, which may display input for a node labeled “ROI.” All modules requiring an input node labeled “ROI” may be listed and displayed to the user. Examples of such modules may include “COMPUTE_SIZE,” a module that may be configured to calculate the size of the ROI; “SPECIFY_FINDING,” a GUI module that may prompt the user to identify various types of disease associated with this finding via a drop-down menu; and “COMPUTE_LOCATION,” an algorithmic module configured to calculate the location of the finding by atlas matching. In other words, the selection of at least one program module may be performed by the user, possibly based on a list of program modules whose input requirements match the region specified by the user.

[0104] A single node (e.g., a node labeled "CT") may trigger a cascade of modules. For example, when a single node labeled "CT" is created, it may automatically trigger a general anomaly detection module. After the general anomaly detection module outputs one or more ROIs, each node labeled "ROI" is added to the graph connected to the node labeled "CT," and another module may calculate the anatomical location of the one or more ROIs. Each anatomical location may then be added as an additional node labeled "LOC" and connected to the respective node labeled "ROI." For each of the one or more ROIs, their anatomical locations, and a patient history node labeled "Medical History," the algorithm module may calculate a list of possible diagnoses and add a node labeled "DIAGNOSIS_LIST" that includes the list as a property. This node may be connected to the respective nodes labeled "ROI," "LOC," and "Medical History." In response to generating the node labeled "DIAGNOSIS_LIST," the UI module may display a text representation of the node labeled "DIAGNOSIS_LIST" via the GUI, for example, by displaying each of the possible diagnoses in the list in text form in a second portion of the display 108. The radiologist can then select one of the text representations to confirm the diagnosis, and a corresponding node labeled "DIAGNOSIS" may be added to the graph.

[0105] The filled-in graph or a medical report generated therefrom may be displayed via a UI, e.g., a GUI, which may occur dynamically and incrementally as more information is filled into the graph.

[0106] As an example, when a radiologist clicks on a patient entry, the viewer may open and trigger the "PACS_QUERY" module, as well as the "RIS_QUERY" module. As soon as the "PACS_QUERY" module returns the last CT image from the patient, this image may be displayed in the viewer (e.g., in a first portion of the display 108). As soon as the "RIS_QUERY" module returns the patient's medical history from the RIS and a respective node labeled "Medical History" is created, the patient's medical history may be displayed via text next to the CT image (e.g., in a second portion of the display 108). As soon as the anomaly detection module returns ROIs as output and a respective node labeled "ROI" is created, each ROI may be displayed overlaid on the original image by an indicator.

[0107] Not all information contained in the graph may be displayed via a GUI. For example, initially, a module database may include a UI module (e.g., prompting the user to select a diagnosis) based on a CT image and an algorithm module (e.g., including a machine learning algorithm configured to predict a diagnosis). Both modules may be triggered each time a CT image is received (e.g., each time a node labeled "CT" is added to the graph), but the output of the algorithm module may not be displayed to the user. The two nodes created as output by the algorithm module and the UI module may be added to the graph database. This data may be used for further analysis, for example, to monitor the performance of the algorithm before it is actually deployed. This is sometimes referred to as "ghost mode." A medical report in this case may be generated based on a subset of the nodes in the graph.

[0108] In one variation, the output of the algorithmic module may not be displayed to the user, and the output of the UI module may be used to train the algorithmic module. Random decisions may be made to determine when the output of the algorithmic module is displayed and when it is hidden while the UI module is in use. A random decision may be made when to enter "ghost mode," such as for quality control and post-market surveillance of the algorithmic module. "Ghost mode" may be selectively applied to only some of the program modules or simultaneously to all of the program modules. The difference between the output of the algorithmic module and the UI module selected at the same time may be stored and used for quality control and post-market surveillance. This difference may be used to trigger a warning to the user that the algorithmic module training or input appears unlikely.

[0109] For example, a graph that may be continuously filled in can be used to continuously learn and improve existing algorithmic (e.g., AI) modules.

[0110] The graph and / or graph database may be used to create new algorithmic modules. For example, there may be a "DISEASE_SELECTION" UI module that radiologists use to begin with. The output provided by this module may be used in conjunction with input CT image(s) to train an algorithm that performs image-based disease classification. Such an approach allows any of the UI modules to be replaced (e.g., incrementally) with one or more algorithmic modules.

[0111] The graph may be used to train meta-models (e.g., AI selection modules) on how to trigger program modules. The graph may be used for statistical analysis (e.g., inferring population statistics or patient similarity searches). The graph data may be converted into one or more different output formats (e.g., written medical reports, speech, etc.). Portions of the graph may be written back to other databases, e.g., RIS.

[0112] Some of the nodes described herein (e.g., nodes labeled "ROI," "local findings," "global findings," "anatomical site," or "impression") may also be referred to as "medical findings." That is, a graph may include multiple medical findings. Accordingly, a medical report may be generated based on the graph by including one or more of the medical findings of the graph in the patient's medical report. A medical report may be a collection of the nodes of the graph. The medical report may be generated by selecting several nodes from the graph and generating a dataset or file containing the properties of the selected nodes (e.g., links thereto). Nodes may be selected based on their labels, for example, using a predetermined list of labels to select from. The nodes "CT" and "ROI" in graph 600 are examples of medical image-related data of a patient. The textual representation of a medical finding may correspond to or be based on a property of the node representing the medical finding (e.g., the property "label" of the node labeled "local findings" shown in FIG. 4).

[0113] 7 is a schematic diagram illustrating a method according to the present disclosure and components associated therewith. The method may include some or all of method steps 202-206 and 302-310. The method may be performed by apparatus 100. The method steps may be performed in any of three areas: on a (e.g., graphical) user interface, labeled "UI layer," in a content database, labeled "content DB," or in an application layer, labeled "application layer." The content database may include a graph database. The content database may further include a node specification database and / or an edge specification database. The content database may be a graph database.

[0114] The computed CT image cCT may be obtained, for example, from a PACS or by loading an image based on a user command. A node 702 labeled "cCT" may be created in the content database and connected to the node "Patient" via an edge that conforms to the constraints defined in the edge specification database 500, thereby extending the graph. The cCT image may be included as a property of the node labeled "cCT" or linked to the node 702 by a property of the node 702 (e.g., the property "Address of Image Location"). In a next step, at least one of the program modules stored in the module database may be selected. For this purpose, the node 702 (e.g., its properties) may be provided to the application layer in step 704. The selected at least one program module 706, 708 may have input requirements that match the node 702. Each of the selected at least one program module may be used to obtain different medical findings in steps 710, 712. In the given example, each of the selected at least one program module 706, 708 may be invoked in steps 710, 712. Module 706 may provide a "local finding" 716 as a medical finding in step 714 based on the input.

[0115] In step 718, the medical finding may be written to the content database, thereby updating the graph by including the medical finding as an additional node 716. In the example given, the updated graph includes node 716 connected to node 702 via edges 720 and 722, and node 724 labeled "ROI." Node 724 is connected to node 716 via edge 720 and to node 702 via edge 722. In other words, the medical finding represented by the node labeled "local finding" is associated with the region represented by the node labeled "ROI," which in turn is associated with the medical image represented by the node labeled "cCT." That is, a medical finding stored as a first node of the graph can be associated with a region, such as an ROI, stored as a second node of the graph by an edge connecting the first and second nodes. A region may itself be a medical finding. A medical finding stored as a first node of the graph may be associated with another medical finding stored as a second or third node of the graph by an edge connecting the first node to the second or third node, each of which may be determined by the selected at least one program module 706 in step 714.

[0116] A medical report may be generated based on the updated graph, including the newly added medical finding. A text representation 726 of one or more medical findings included in the medical report may be displayed in the second portion of the display. Clicking on the text representation 726 may cause an indicator 728 of the ROI represented by the node 724 to be displayed in the cCT image 727 in the first portion of the display. Alternatively, or additionally, the text representation 726 may be displayed when the user clicks on the indicator 728. In this example, the first and second portions of the display may be part of separate display units 108, 110 included in or connected to the device 100.

[0117] The user may define other regions within the medical image in step 730. In response, a new node "ROI" 732 conforming to the requirements set forth in the node specification database 400 may be added to the graph, connected to node 702 via an edge 734 conforming to the constraints set forth in the edge specification database 500. The user may also define other medical findings, such as "local findings," associated with the ROI. A node 736 with the label "local findings" may be added to the graph via an appropriate edge 738 connecting nodes 736 and 732. The creation of node 736 may trigger another selection of at least one program module, this time with input requirements consistent with the newly added node 736. In the illustrated example, program modules 740-744 "ATLAS_AI," "VOLUME_METER," and "DENSITY_METER" are selected. Each of these selected modules 740-744 may provide (e.g., different) medical findings in response to being selected. In the illustrated example, each of the modules may be invoked in steps 746-750 and provided as input data to node 736 and / or node 732. Modules 742-744 output positions, volumes, and densities based on the input data in steps 752-756.

[0118] Again, as in step 718, these medical findings obtained in steps 752-756 are written to the graph database as nodes 758-762. In the given example, location, volume, and density are added to the graph as additional medical findings, as shown by nodes 748-752, respectively. Again, the medical report may be updated based on the newly added nodes. In the illustrated example, a textual representation 764 of the newly included medical finding may be displayed to the user on the display (e.g., in the second portion).

[0119] That is, step 730 may trigger the selection of an appropriate program module, which may then trigger the selected module to output a medical finding based on the user input of step 730. The medical report may then be updated to include the determined medical finding, and a visualization of the medical report may be provided to the user. The user may "jump" from one medical finding to another by selecting its textual representation, while being informed of the region associated with the selected medical finding by a dynamically displayed indicator of the selected region. Similarly, the user may select one of multiple regions in the medical image and provide a textual representation of all medical findings associated with the selected region, which may be generated based on or included in the medical report.

[0120] In step 766, the user may provide user input in text form. The user input may define medical findings. The information entered by the user may be displayed on the display (758) and then stored as an additional node 770 in the graph database (e.g., as a property thereof). The medical report may then be updated again, for example, by including the text input as an additional medical finding. Steps 730 and / or 766 may be performed upon request of the UI module.

[0121] FIG. 8a illustrates an exemplary display according to the present disclosure. The display may be presented by the display unit 108. The display may correspond to the example shown at the bottom of FIG. 7. As can be seen, a medical image 727 may be displayed on a first portion of the display unit 108. An indicator 728 of an ROI represented by node 724 may also be displayed in the first portion. In a second portion of the display unit 108 shown on the right side of FIG. 8a, multiple textual representations of medical findings may be displayed. In the illustrated example, only treatment, clinical information, and past examinations may be displayed in text format. Then, when a user selects the exemplary indicator 728 (as indicated by the hand symbol in FIG. 8a), a textual representation of a medical finding represented by node 724 and associated with the region visualized by indicator 728 may be displayed. For example, a textual representation of a local finding represented by node 716 may be displayed on the second portion of the display in response to a user selecting the region visualized by indicator 728. In the illustrated example, a textual representation of the medical finding may be displayed under a title representing the type of medical finding (e.g., the title "Findings" or "Opinions" shown on the right side of FIG. 8a; in the illustrated example, such a textual representation is not yet shown, and therefore the display still displays "No Findings" or "No Impressions").

[0122] FIG. 8b shows an exemplary display after a user clicks on indicator 728, as described above with respect to FIG. 8a. It can be seen that under each of the headings "Findings" and "Sentiments," a textual representation of a medical finding may be displayed. The textual representation may be a representation of a medical finding associated with the region visualized by indicator 728, represented by node 724, previously selected by the user. The displayed textual representation may be a textual representation of a local finding, represented by node 716. The user may modify or add to the medical finding (e.g., described above) and / or the textual representation (e.g., displayed). For example, the user may define properties of the medical finding (e.g., properties of the node representing the medical finding) by selecting at least one property from a list of available properties, as shown on the left side of FIG. 8b. If indicator 728 was previously selected, the newly added medical finding may be associated with the region visualized by indicator 728. This allows the user to "enrich" the selected ROI in the medical image 727 with medical findings.

[0123] In other variations in which selection of a textual representation occurs rather than selection of an indicator on the medical image, the textual representation of the medical finding represented by node 716 may be displayed in the second portion of the display unit 108, e.g., with the title "Findings," and the medical image 727 may be initially displayed in the first portion without indicator 728. Additional textual representations of different medical findings may also be displayed in the second portion, e.g., textual representations 764 and / or 768. Indicator 728 may then be displayed on the medical image 727 in the first portion of the display unit 108 in response to a user selecting the textual representation of the medical finding represented by node 716.

[0124] FIG. 9 is a schematic diagram of an example method according to the present disclosure and components involved therein. The method may include method steps 202-206, 302-310, and some or all of the steps described above with reference to FIGS. 7 and 8. The method may be performed by apparatus 100. The method steps may be performed in one of three domains: on a (e.g., graphical) user interface labeled "UI layer," in a content database labeled "content DB," or in an application layer labeled "AI engine / application layer." The content database may include a graph database. The content database may further include a node specification database and / or an edge specification database. The content database may be a graph database. The layers may correspond to the layers shown in FIG. 7.

[0125] The program modules may be stored in a module database and perform functions in the "AI engine / application layer." Medical image-related data 802, such as cCT, may be acquired. An appropriate program module 804 having input requirements that match the data 802 may be selected and used to acquire medical findings. The medical findings may be of various types, such as a first type 806 (e.g., ROI), a second type 808 (e.g., local findings), and a third type 810 (e.g., volume of the ROI). The medical findings may be written to a content database. In other words, the graph may be extended with new nodes corresponding to the medical findings acquired using the module 804. The medical report may then be updated based on the extended graph.

[0126] Next, program modules having input requirements matching one of the first through third types 806-810 may be selected. In the illustrated example, these selected modules are modules 812-816. Each of these modules may again provide a different medical finding based on the medical finding previously determined using module 804. The types of medical findings provided by modules 812-816 may differ from each other. Also, in this case, after determining the new medical finding using modules 812-816, the graph and medical report may be updated. Another iteration of this procedure is shown in FIG. 9 for selected modules 818-822. The selected modules may have input requirements matching multiple previously obtained medical findings, as illustrated for modules 824 and 826. Modules 804, 812-826 may include or correspond to modules 706, 708, 740-744 described above with reference to FIG. 7.

[0127] The UI module 828 may provide having input requirements that match a particular (e.g., type) medical finding. The UI module 828 may rely on user input to provide the medical finding. For example, the UI module 828 may request that the user provide user input, as described above for step 730 or step 766.

[0128] All acquired medical findings may be stored in a content database (e.g., as nodes in a graph), and a medical report may be generated that includes at least the most recent of the acquired medical findings, e.g., all of the acquired medical findings stored in the content database.

[0129] Accordingly, the present disclosure also provides methods according to the following examples. [Example (A)] 1. A method for providing an interactive display of a medical image (727) based on a patient's medical report, the method comprising: (a) displaying (202) a medical image (727) of the patient on a first portion of a display; at least, (b) in response to a user selecting a region in the displayed medical image (727), the region relating to a medical finding included in the medical report, displaying (204) a textual representation (726, 764) of the medical finding in a second portion of the display; (c) displaying (206) a textual representation (726, 764) of a medical finding on a second portion of the display, the medical finding being included in the medical report and associated with a region within the medical image (727), and displaying an indicator (728) of the region on the first portion of the display in response to a user selecting the displayed textual representation (726, 764); The method includes one of the following: [Example (B)] and updating the medical report by adding medical findings based on user input in the medical report. The method described in Example (A). [Example (C)] The method of embodiment (B), wherein the added medical findings are associated with a region (730) defined by the user in the displayed medical image (727). [Example (D)] displaying a plurality of indicators of different regions on the first portion of the display, wherein the added medical findings are associated with a group of regions, the group of regions including the set of different regions selected by the user. The method described in Example (B). [Example (E)] The method of any one of embodiments (B) to (D), wherein the added medical findings (770) are defined (766) by the user. [Example (F)] The method of any one of claims (A) to (E), wherein at least one additional text representation of a different medical finding associated with the region is displayed along with the text representation (726, 764) of the medical finding in the second portion of the display, and the different medical finding is included in the medical report. [Example (G)] and updating the medical report in response to the user specifying one or more of the text representations (726, 764) and the at least one additional text representation by deleting from the medical report any of the medical findings represented by text representations not specified by the user. The method described in Example (F). [Example (H)] updating the medical report by adding medical findings in the medical report, the medical findings being determined by an artificial intelligence (AI) program module (706, 708, 740, 742, 744, 804, 812-826); The method of any one of Examples (A) to (G). [Example (I)] and further comprising using at least one of the designated text representation and the medical findings represented by the designated text representation as training data for training the AI ​​modules (706, 708, 740, 742, 744, 804, 812 to 826). The method described in Example (G) or (H). [Example (J)] and further comprising using at least one of the added medical findings and the textual representation of the added medical findings as training data for training the AI ​​modules (706, 708, 740, 742, 744, 804, 812-826). The method described in Example (H) or (I). [Example (K)] The method of any one of Examples (B) to (E), (G) to (J), or Example (F) when dependent on one of Examples (B) to (E), wherein at least one of steps (b) and (c) is repeated after updating the medical report. [Example (L)] The method of any one of claims (A) to (K), wherein the medical findings are stored as nodes of a graph in a graph database. [Example (M)] An apparatus (100) comprising at least one processor (102) and at least one memory (104), wherein the at least one memory (104) contains instructions executable by the at least one processor (102) such that the apparatus (100) is operable to perform the method of any one of embodiments (A) to (L). [Example (N)] A computer program product comprising program code portions for performing the method according to any one of embodiments (A) to (L) when the computer program product is executed on one or more processors (102). [Example (O)] A computer program product according to any one of the preceding embodiments, stored on one or more computer-readable recording media.

[0130] The advantages of the technology presented herein will be fully understood from the foregoing description, and it will be apparent that various changes can be made in the form, construction, and arrangement of exemplary embodiments thereof without departing from the scope of the disclosure or sacrificing all of its advantageous effects. Because the technology presented herein can be varied in many ways, it will be recognized that the disclosure should be limited only by the scope of the claims which follow. [Configuration 1] 1. A method for generating a medical report for a patient, the method comprising: (a) in response to acquiring medical image-related data of the patient, selecting (302) from a plurality of program modules (706, 708, 740, 742, 744, 804, 812-828) at least one program module having input requirements that match the medical image-related data of the patient; (b) using (304) the at least one program module selected in step (a) to obtain a medical finding based on the medical image-related data of the patient; (c) selecting (306) at least one program module from the plurality of program modules (706, 708, 740, 742, 744, 804, 812-826) having input requirements that match the acquired medical findings; (d) using (308) the at least one program module selected in step (c) to obtain a medical finding based on the previously obtained medical finding; (e) preparing the medical report for the patient, the medical report including at least one of the medical findings obtained; A method comprising: [Configuration 2] 10. The method of claim 1, wherein the medical image-related data of the patient includes at least one of a medical image (727) of the patient, a region of interest (ROI) in the medical image (727) of the patient, properties of the ROI, and medical findings derived from the medical image (727) of the patient. [Configuration 3] 3. The method of claim 1 or 2, wherein steps (c) and (d) are repeated at least once before generating said medical report in step (e). [Configuration 4] 4. The method of claim 3, wherein the at least one program module selected in step (c) has input requirements that match a predetermined subset of the acquired medical findings or all of the acquired medical findings. [Configuration 5] 5. The method of any one of configurations 1 to 4, wherein some or all of the selected at least one program module is used to obtain different types of medical findings. [Configuration 6] one or more of the at least one program modules (706, 708, 740, 742, 744, 804, 812-826) selected in step (a) are configured to autonomously determine the medical findings based on the acquired medical image-related data; and / or one or more of the at least one program modules (706, 708, 740, 742, 744, 804, 812-826) selected in step (c) are configured to autonomously determine the medical finding based on the previously obtained medical finding; 6. The method according to any one of claims 1 to 5. [Configuration 7] 7. The method of claim 6, wherein one or more of the at least one selected program modules (706, 708, 740, 742, 744, 804, 812-826) is an artificial intelligence (AI) module. [Configuration 8] In step (a) or (c), if multiple AI modules (706, 708, 740, 742, 744, 804, 812 to 826) that provide the same type of medical findings are selected, combining the multiple AI modules (706, 708, 740, 742, 744, 804, 812 to 826) into an ensemble and using the ensemble to obtain the medical findings; 8. The method of claim 7, further comprising: [Configuration 9] one or more of the at least one program modules selected in step (a) is a user interface (UI) module (828) that requests user input defining the medical findings in step (b); and / or and wherein one or more of the at least one program modules selected in step (c) is a user interface (UI) module (828) that requests user input defining the medical findings in step (d). 9. The method according to any one of claims 1 to 8. [Configuration 10] 10. The method of claim 9, wherein if the plurality of program modules (706, 708, 740, 742, 744, 804, 812-828) does not include a program module configured to autonomously determine the medical findings, then the at least one selected program module is the one or more UI modules (828). [Configuration 11] training an artificial intelligence (AI) module (706, 708, 740, 742, 744, 804, 812-828) of the plurality of program modules (706, 708, 740, 742, 744, 804, 812-826) with the medical findings obtained using the one or more UI modules (828); 11. The method of claim 9 or 10, further comprising: [Configuration 12] If in step (a) a module configured to autonomously determine the medical finding is selected, displaying visualization information of the medical finding determined by the selected module configured to autonomously determine the medical finding; hiding the visualization information if a selected UI module (828) that provides the same type of medical findings as the selected module configured to autonomously determine the medical findings is also selected in step (a); and / or If in step (c) a module configured to autonomously determine the medical finding is selected, displaying visualization information of the medical finding determined by the selected module configured to autonomously determine the medical finding; In step (c), hiding the visualization information if a UI module (828) that provides the same type of medical findings as the selected module and that is configured to autonomously determine the medical findings is also selected; 12. The method of any one of configurations 9 to 11, when dependent on any one of configurations 6 to 8, further comprising: [Configuration 13] 13. The method of claim 12, wherein the selected module configured to autonomously determine the medical finding is an artificial intelligence (AI) module (706, 708, 740, 742, 744, 804, 812-826), and the method further comprises training the selected AI module (706, 708, 740, 742, 744, 804, 812-826) using the medical findings obtained by the selected UI module (828) that provides the same type of medical finding as the selected AI module (706, 708, 740, 742, 744, 804, 812-826). [Configuration 14] 14. The method of any one of configurations 1 to 13, wherein the selection of the at least one program module in steps (a) and / or (c) is performed by an artificial intelligence (AI) selection module. [Configuration 15] training the AI ​​selection module using some or all of the obtained medical findings; 15. The method of claim 14, further comprising: [Configuration 16] storing each of the obtained medical findings as a node of a graph in a graph database, based on which the medical report is generated; 16. The method of any one of claims 1 to 15, further comprising: [Configuration 17] An apparatus (100) including at least one processor (102) and at least one memory (104), wherein the at least one memory (104) includes instructions executable by the at least one processor (102) such that the apparatus (100) is operable to perform the method of any one of configurations 1 to 16. [Configuration 18] 17. A computer program product comprising program code portions for performing the method of any one of configurations 1 to 16 when the computer program product is executed on one or more processors (102). [Configuration 19] 19. The computer program product of configuration 18 stored on one or more computer-readable storage media. < / ddmmyyyy> < / ddmmyyy> < / name>

Claims

1. A processor-implemented method for generating a patient medical report, the method comprising: (a) in response to acquiring medical image-related data of the patient, selecting (302) from a plurality of program modules (706, 708, 740, 742, 744, 804, 812-828) at least one program module having input requirements consistent with the medical image-related data of the patient; (b) using the at least one program module selected in step (a) (304) to obtain a medical finding based on the medical image-related data of the patient; (c) selecting (306) at least one program module from the plurality of program modules (706, 708, 740, 742, 744, 804, 812-826) having input requirements that match the obtained medical findings; (d) using (308) the at least one program module selected in step (c) to obtain a medical finding based on the previously obtained medical finding; (e) preparing the medical report for the patient, the medical report including at least one of the medical findings obtained; A method comprising:

2. 2. The method of claim 1, wherein the medical image-related data of the patient includes at least one of a medical image (727) of the patient, a region of interest (ROI) in the medical image (727) of the patient, properties of the ROI, and medical findings derived from the medical image (727) of the patient.

3. 3. The method of claim 1 or 2, wherein steps (c) and (d) are repeated at least once before generating the medical report in step (e).

4. 4. The method of claim 3, wherein the at least one program module selected in step (c) has input requirements that match a predetermined subset of the acquired medical findings or all of the acquired medical findings.

5. 5. The method of claim 1, wherein some or all of the selected at least one program module are used to obtain different types of medical findings.

6. one or more of the at least one program modules (706, 708, 740, 742, 744, 804, 812-826) selected in step (a) are configured to autonomously determine the medical findings based on the acquired medical image-related data; and / or one or more (706, 708, 740, 742, 744, 804, 812-826) of the at least one program module selected in step (c) are configured to autonomously determine the medical finding based on the previously obtained medical finding; 6. The method according to any one of claims 1 to 5.

7. The method of claim 6, wherein one or more of the selected at least one program module (706, 708, 740, 742, 744, 804, 812-826) is an artificial intelligence (AI) module.

8. If in step (a) or (c) multiple AI modules (706, 708, 740, 742, 744, 804, 812-826) that provide the same type of medical finding are selected, combining the multiple AI modules (706, 708, 740, 742, 744, 804, 812-826) in an ensemble and using the ensemble to obtain the medical finding; The method of claim 7 further comprising:

9. one or more of the at least one program modules selected in step (a) is a user interface (UI) module (828) that requests user input defining the medical findings in step (b); and / or and one or more of the at least one program modules selected in step (c) is a user interface (UI) module (828) that requests user input defining the medical findings in step (d).

9. The method according to any one of claims 1 to 8.

10. 10. The method of claim 9, wherein if the plurality of program modules (706, 708, 740, 742, 744, 804, 812-828) does not include a program module configured to autonomously determine the medical finding, the selected at least one program module is the one or more UI modules (828).

11. training an artificial intelligence (AI) module (706, 708, 740, 742, 744, 804, 812-828) of the plurality of program modules (706, 708, 740, 742, 744, 804, 812-826) with the medical findings obtained using the one or more UI modules (828); 11. The method of claim 9 or 10, further comprising:

12. If in step (a) a module configured to autonomously determine the medical finding is selected, displaying visualization information of the medical finding determined by the selected module configured to autonomously determine the medical finding; hiding the visualization information if a UI module (828) that provides the same type of medical findings as the selected module configured to autonomously determine the medical findings is also selected in step (a); and / or If in step (c) a module configured to autonomously determine the medical finding is selected, displaying visualization information of the medical finding determined by the selected module configured to autonomously determine the medical finding; In step (c), hiding the visualization information if a UI module (828) that provides the same type of medical findings as the selected module and that is configured to autonomously determine the medical findings is also selected; 12. The method of any one of claims 9 to 11 when dependent on any one of claims 6 to 8, further comprising:

13. 13. The method of claim 12, wherein the selected module configured to autonomously determine the medical finding is an artificial intelligence (AI) module (706, 708, 740, 742, 744, 804, 812-826), and the method further comprises training the selected AI module (706, 708, 740, 742, 744, 804, 812-826) using the medical findings obtained in a selected UI module (828) that provides the same type of medical finding as the selected AI module (706, 708, 740, 742, 744, 804, 812-826).

14. 14. The method of claim 1, wherein the selection of the at least one program module in steps (a) and / or (c) is performed by an artificial intelligence (AI) selection module.

15. training the AI ​​selection module using some or all of the obtained medical findings; 15. The method of claim 14, further comprising:

16. storing each of the obtained medical findings as a node of a graph in a graph database, based on which the medical report is generated; 16. The method of claim 1, further comprising:

17. 17. An apparatus (100) comprising at least one processor (102) and at least one memory (104), wherein the at least one memory (104) comprises instructions executable by the at least one processor (102) such that the apparatus (100) is operable to perform the method of any one of claims 1 to 16.

18. A computer program comprising program code portions for performing the method according to any one of claims 1 to 16, when the computer program is run on one or more processors (102).

19. 20. The computer program of claim 18 stored on one or more computer readable storage media.

Images