Method and system for providing text summary for medical image datasets

Abstract

Systems and methods are provided for providing a textual summary for a medical image dataset. In particular, the following steps are proposed: receiving (S10) a medical image data set (MIDS) of a patient; identifying (S20) at least one compartment (ORG) in the medical image dataset (MIDS); accessing (S30) supplementary information (SI) associated with the patient in the medical information system (40); providing (S40) a text generation function (LLM), the text generation function (LLM) configured to provide, for a particular compartment, a natural language text summary (TS) summarizing medical information relating to the particular compartment; applying (S50) a text generation function (LLM) to the supplemental information (SI) to generate a text summary (TS) for the at least one compartment (ORG); and providing (S60) the text summary (TS) to the user (U) via the user interface (10).

Description

Technical Field

[0001] Embodiments of the present invention relate to systems and methods in the field of automatically or semi-automatically generating natural language text. Specifically, embodiments of the present invention relate to systems and methods for providing text summaries for medical image datasets. More specifically, embodiments of the present invention relate to systems and methods for summarizing findings for various compartments such as organs or anatomical structures visible in medical image datasets in natural language text. Background Art

[0002] In order to create reliable medical reports and produce correct diagnoses, findings and conclusions, radiologists need to integrate a large amount of information from different sources. Typically, radiologists have to perform several tasks in parallel when evaluating medical image data and preparing reports. First, they have to analyze the medical images and summarize their observations and impressions in the radiology report. In addition, they have to take into account additional information about the patient. This information can, for example, come from images of different modalities or measurement protocols, as well as information from patient health records, laboratory findings, previous images, etc. Therefore, the type and number of individual reporting tasks to be performed depends on the available interdisciplinary data, i.e., medical images and other available patient-specific medical data.

[0003] Furthermore, a wealth of information is hidden in plain text (previous reports, doctor's notes, lab reports, etc.). Reading and understanding the entirety of this information can take a significant amount of time and slow down the radiologist. As a result, some radiologists do not even read their patients' previous reports and other documentation. This, in turn, can reduce the quality of the radiology work product.

[0004] To improve this aspect, it has been proposed to automatically access and filter the available information of the patient and present this information to the radiologist. One problem is that it is difficult for the radiologist to understand why a certain piece of information is retrieved. This leads to the radiologist being unsure whether the information is relevant and complete.

[0005] This is especially true if the information is provided by an AI module or AI chatbot like ChatGPT and other large language models. Often these algorithms are treated as black boxes. Furthermore, each case is different, and information relevant to one case may not be relevant to another, simply because the image data that the radiologist has to analyze shows different anatomical structures.

[0006] Despite the impressive performance of large language models, methods and systems that allow for the retrieval of information specifically tailored to the medical image dataset under review remain prevalent. Summary of the invention

[0007] In this context, an object of embodiments of the present invention is to provide the following systems and methods: for processing the available data of a patient to retrieve and further process supplementary information and transform this information into actionable results, which optimally supplement the medical image dataset and / or the performance of diagnostic tasks based on the medical image dataset.

[0008] In the following, the technical solution according to the invention is described with respect to the claimed system and with respect to the claimed method. Features, advantages or alternative embodiments described herein may likewise be assigned to other claimed objects and vice versa. In other words, the claims for the method of the invention may be improved by features described or claimed with respect to the system. In this case, for example, the functional features of the method are embodied by the target unit or target element of the system.

[0009] Features and alternative forms of embodiments of the data structures and / or functions of the method and system for providing can be transferred to similar data structures and / or functions of the method and system for providing trained functions. Similar data structures can be identified in particular by the use of the prefix "training". In addition, the trained functions used in the method and system for providing text summaries can be adjusted and / or trained and / or provided in particular by the method and system for adjusting the trained functions.

[0010] According to one aspect, a computer-implemented method for providing a text summary for a medical image dataset is provided. The method comprises a plurality of steps. One step is to receive a medical image dataset of a patient. Another step is to identify at least one compartment in the medical image dataset. Another step is to access information associated with the patient in a medical information system. Another step is to provide a text generation function configured to provide, for a particular compartment, a natural language text summary summarizing medical information related to the particular compartment. Another step is to apply the text generation function to supplemental information to generate a text summary for at least one compartment. Another step is to provide the text summary to a user via a user interface.

[0011] The medical image dataset may be a two-dimensional image. Furthermore, the medical image dataset may be a three-dimensional image. Furthermore, the medical image may be a four-dimensional image, where there are three spatial dimensions and one dimension like time. Furthermore, the medical image dataset may include a plurality of individual medical images.

[0012] Medical image data sets include image data in the form of a two-dimensional or three-dimensional array of pixels or voxels, for example. Such an array of pixels or voxels may represent color, intensity, absorption or other parameters that vary with two-dimensional or three-dimensional position and may be obtained, for example, by appropriate processing of measurement signals obtained by a medical imaging modality or an image scanning device.

[0013] A medical image dataset may be a radiological image dataset depicting a body part of a patient. Thus, it may contain two-dimensional or three-dimensional image data of a body part of a patient. The medical image may represent an image volume or a cross-section through an image volume. A body part of a patient may be included in the image volume.

[0014] Medical imaging modalities correspond to systems for generating or producing medical image data. For example, medical imaging modalities may be computed tomography systems (CT systems), magnetic resonance systems (MR systems), angiography (or C-arm X-ray) systems, positron emission tomography systems (PET systems), etc. Specifically, computed tomography is a widely used imaging method, and utilizes X-rays generated and detected by a spatially rotating instrument. The resulting attenuation data (also referred to as raw data) is processed by computer analysis software, which generates detailed images of the internal structures of the patient's body parts. The resulting image set is referred to as a CT scan, which may constitute a plurality of series of continuous images, presenting the internal anatomical structures in cross-sections perpendicular to the axis of the human body. Another example to be provided, magnetic resonance imaging (MRI), is an advanced medical imaging technique that utilizes the effects of an effective magnetic field on the movement of protons. In an MRI machine, the detector is an antenna, and the signal is analyzed by a computer to create a detailed image of the internal structure in any section of the human body.

[0015] Therefore, the depicted body part of the patient will usually include multiple anatomical structures and / or organs. Taking a chest image as an example, a medical image may show lung tissue, a chest cavity, lymph nodes, etc.

[0016] The medical image dataset may include a plurality of images or image slices. The slices respectively show a cross-sectional view of an image volume. The slices may include a two-dimensional array of pixels or voxels as image data. The arrangement of the slices in the medical image dataset may be determined by the imaging modality or by any post-processing scheme used. Furthermore, the slices may be artificially defined in the imaging volume spanned by the medical image dataset. Optionally, this may occur based on the image data included in the medical image dataset in order to optimally pre-process the medical image dataset for a subsequent diagnostic workflow.

[0017] Medical images may be stored in a standard image format such as the Digital Imaging and Communications in Medicine (DICOM) format and stored in a memory or computer storage system such as a Picture Archiving and Communication System (PACS), a Radiology Information System (RIS), etc. Whenever DICOM is mentioned herein, it should be understood that DICOM refers to the "Digital Imaging and Communications in Medicine" (DICOM) standard, for example, according to the DICOM PS3.12020c standard (or any later or earlier version of the standard).

[0018] According to some examples, the supplemental information may be non-image data of the patient. According to some examples, the supplemental information includes structured and / or unstructured natural language text.

[0019] According to some examples, the supplemental information includes one or more of the following elements:

[0020] - The patient's previous medical reports, and / or

[0021] - prior knowledge of at least one type of medical problem that the patient is suspected of having, and / or

[0022] - an indication of a diagnostic task to be performed based on a medical image dataset about the patient, and / or

[0023] - Medical guidelines applicable to the patient, and / or

[0024] -The patient's electronic health record.

[0025] According to some examples, the electronic health record may include a patient's medical history, including any pre-existing conditions, comorbidities, risk factors, referral letters, demographic information such as age or gender, and the like.

[0026] Obtaining the supplemental information may include querying a healthcare information system, such as a HIS (hospital information system), a LIS (laboratory information system), an EMR system (electronic medical record system), etc., to obtain the patient's supplemental information. For example, such supplemental information may be obtained in the form of one or more EMR files (electronic medical record files). In addition, querying the healthcare information system may be based on a patient identifier that electronically identifies the patient in the system, such as an ID or a patient name.

[0027] A compartment may generally relate to an entity of a patient's body. For example, a compartment may relate to an organ, an organ part, an organ function, an anatomical structure, an anatomical body, a functional unit of a patient's body, etc.

[0028] The text generation function can be configured to recognize and / or understand natural language and in particular individual items such as words in the input containing natural language, and is configured to convert the items into text output. The natural language processing algorithm can be based on trained or machine-learned functions. As an alternative, the natural language processing algorithm can be rule-based.

[0029] According to some examples, the text generation function is configured to identify data elements (also represented as excerpts) in the supplemental information that are relevant to the specific compartment, and generate a text summary for the specific compartment based on the identified data elements. According to some examples, the text generation function is configured to filter the data elements to obtain relevant information, and generate a text summary for the specific compartment based on the relevant information.

[0030] According to some examples, the text generation function is configured to process or change the data elements and generate a text summary based on the changed data elements. According to some examples, the change can include language processing, such as rewriting, translation, shortening, etc.

[0031] Providing the text generation function may include keeping the function available in a suitable storage device accessible by the computing unit executing the method. Furthermore, providing the function may include downloading the function by the computing unit.

[0032] According to some examples, the text summary may include structured or unstructured natural language text. According to some examples, the text summary may have a predefined format and / or text length. According to some examples, the text summary includes an excerpt of the supplementary information. According to some examples, the format may include one or more main points with information related to the compartment. According to some examples, the text length may include a predefined number of sentences, such as one to ten sentences, in particular two, three or four sentences. According to some examples, the text summary has a shorter text length than the text included in the supplementary information. According to some examples, the text summary may include one or more medical findings for the compartment.

[0033] According to some examples, the text summary may be displayed to the user in a user interface. According to some examples, the text summary may be displayed in a manner that may be attributed to the potential compartment by the user. According to some examples, the text summary may be shown in the user interface as an overlay over the compartment in the visualization (or representation) of the medical image dataset. According to some examples, the display of the text summary may be activated / started by the user, for example, by hovering over the compartment in the visualization of the medical image dataset with a mouse cursor.

[0034] According to some examples, providing may include highlighting that the text summary is available for the identified compartment in a visualization (or representation) of the medical image dataset showing the identified compartment.

[0035] By generating a text summary for a specific compartment, the user is provided with a quick overview of clinically relevant information about the compartment. The generation of the text summary can be considered as an automatic processing of the medical information (i.e., supplementary information) to arrive at a medical diagnosis. This saves the user from having to search for relevant information in the patient's file. Furthermore, since the text generation is based on what is actually shown in the medical image dataset, the processing optimally complements the diagnostic task that the user has to perform based on the medical image dataset.

[0036] According to some examples, the step of identifying compartments may include subjecting the medical image data set to an image processing step configured to identify one or more compartments in the medical image data.

[0037] According to some examples, the image processing step may include an image segmentation step. Typically, as mentioned, any depicted body part will include multiple compartments, such as, for example, a chest image will include lung lobes, bone compartments (e.g., including rib structures or spine), the heart, hardware such as implants, pacemakers or catheters, etc. According to some examples, the step of segmentation may include segmenting multiple (different) compartments to provide image data respectively related to the segmented compartments.

[0038] In principle, for medical image segmentation, a variety of functions and methods are known (all of which can be implemented in the segmentation step described herein). For example, see US 7,822,274 B2, US 9,760,807 B2, US 11,393,229 B2, the contents of which are hereby incorporated by reference in their entirety.

[0039] According to some examples, the image processing step may include an image recognition or classification step configured to classify the image data according to its affiliation with a particular compartment. For example, an image classifier may be extracted and compared to signatures of different compartments. For example, reference is made to US 2023 / 0 041 553A1, the contents of which are incorporated herein by reference in their entirety.

[0040] According to some examples, the supplemental information includes at least one medical report of the patient including structured and / or unstructured text.

[0041] According to some examples, the supplemental information includes a plurality of different medical reports of the patient including structured and / or unstructured text, and the text summary summarizes the plurality of different medical reports.

[0042] In other words, according to the above example, the text summary can be considered as a compartment-specific text summary of one or more medical reports of the patient. Thus, the user does not have to read the individual reports, but is automatically provided with a summary constructed from the content of the medical image dataset.

[0043] According to some examples, the step of identifying at least one compartment includes: obtaining an indication of a disease pattern of the patient, performing a lookup operation in an association linking the compartment to a particular disease pattern to determine at least one compartment. According to some examples, the step of identifying may also include identifying the determined compartment in the medical image dataset (e.g., by applying a corresponding segmentation algorithm).

[0044] According to some examples, the indication may include one or more of the following: a reason why the medical image dataset has been acquired, a diagnostic task to be performed based on the medical image dataset, one or more suspected diagnoses, one or more confirmed diagnoses, and the like.

[0045] According to some examples, the indication may be obtained based on direct input from a user. According to other examples, the indication may be automatically identified in the supplemental information and / or the medical image dataset. According to some examples, this may include applying a text generation function to the supplemental information and / or the medical image dataset, which may then additionally be configured to extract an indication of a disease pattern from the supplemental information and / or the medical image dataset.

[0046] According to some examples, the (electronic) association may include one or more potentially affected compartments for at least one disease pattern. In other words, the association links the disease pattern to a potentially affected organ. This may be based on clinical experience, for example outlining metastatic pathways. Thus, potentially relevant compartments may be automatically identified, which enables targeted generation of a text summary.

[0047] According to some examples, the association may be based on or include one or more clinical ontologies such as RadLex or SNOMED. According to some examples, the association may include a medical guideline.

[0048] According to some examples, the text generation function is configured to filter the supplementary information and / or data elements according to the creation time (i.e., the time when the corresponding supplementary information and / or data elements are created). Specifically, according to some examples, the text generation function can be configured to filter the supplementary information and / or data elements to obtain elements created at a specific time interval. According to some examples, the time interval can extend from a first time point to a second time point. According to some examples, the second time point can be now, and the first time point can be a predetermined time point in the past.

[0049] Therefore, the acquisition time of a certain piece of information can be used as a parameter for retrieving relevant information from the supplementary information. Thus, outdated information can be automatically discarded.

[0050] According to one aspect, the text generation function includes a transformer network and / or a large language model.

[0051] Transformer networks are neural network architectures that typically include an encoder, a decoder, or both an encoder and a decoder. In some cases, the encoder and / or decoder are composed of several corresponding encoding layers and decoding layers, respectively. There is an attention mechanism in each encoding layer and decoding layer. The attention mechanism (sometimes referred to as self-attention) links a data item (e.g., a word) in a series of data items with other data items in the series. For example, the self-attention mechanism allows the model to examine a phrase in a sentence and determine the relative importance of other phrases in the sentence to the examined word.

[0052] In particular, the encoder can be configured to transform the input (text) into a digital representation. The digital representation can include a vector for each input token (e.g., each word). The encoder can be configured to implement an attention mechanism so that each vector of a token is affected by other tokens in the input. In particular, the encoder can be configured to parse the representation for the desired output of the transformer network.

[0053] In particular, the decoder can be configured to transform the input into a sequence of output tokens. In particular, the decoder can be configured to implement a masked self-attention mechanism so that each vector of a token is only influenced by other tokens on one side of the sequence. Furthermore, the decoder can be autoregressive, meaning that intermediate results (e.g., a sequence of previously predicted tokens) are fed back.

[0054] According to some examples, the output of the encoder is input into a decoder.

[0055] Additionally, the transformer network may include a classification module or a unit configured to map the output of an encoder or decoder to a set of learned outputs, such as a text summary.

[0056] The training of the transformer model according to some examples can occur in two stages - a pre-training stage and a fine-tuning stage. In the pre-training stage, the transformer model can be pre-trained on a large data corpus to learn the potential semantics of the problem. Such a pre-trained transformer model can be used for different languages. For some applications described herein, fine-tuning can include further training the transformer network with medical text and / or medical ontologies (e.g., RADLEX and / or SNOMED) with expert annotated meanings. In particular, using the latter, the transformer model according to some examples can learn typical relationships and synonyms of medical expressions.

[0057] For a review of transformer networks, see Vaswani et al., “Attention Is All You Need”, arXiv:1706.03762, June 12, 2017, which is incorporated herein by reference in its entirety.

[0058] The advantages of transformer networks are that they can efficiently handle long-range dependencies in the input data due to the attention mechanism. In addition, the encoder used in transformer networks is able to process data in parallel, which saves computational resources in inference. Moreover, due to autoregression, the decoder of the transformer network is able to iteratively generate a series of output tokens with great confidence.

[0059] According to one aspect, the step of identifying includes identifying a plurality of different compartments in the medical image dataset, and the step of applying includes applying a text generation function to the supplemental information to generate a corresponding text summary for each of the plurality of different compartments.

[0060] According to some examples, the method further comprises generating (rendering) a representation of the medical image dataset for display to a user via a user interface, and displaying the representation to the user via the user interface, wherein a corresponding text summary is generated before displaying the representation.

[0061] The representation may be a two-dimensional representation image rendered from a medical image dataset. The representation may include a plurality of image pixels. In particular, the representation may be a two-dimensional rendering of the medical image. Typically, the two-dimensional rendering may rely on known rendering processes, such as ray casting, ray tracing, texture rendering, etc. According to some examples, the rendering may cause the image data in combination with the medical image to display the identified compartment.

[0062] By creating text summaries for multiple compartments, relevant information can be kept available to the user, which she or he can access / display on a request basis. In particular, by already generating the text summary before displaying the rendering of the medical image study, the text summary is already available before the user views the case. Thus, the method is seamlessly integrated into the workflow.

[0063] According to one aspect, the method further includes receiving a request for an overall text summary of the medical image dataset, generating an overall text summary by applying a text generation function to the corresponding text summary, wherein the overall text summary includes a summary of the corresponding text summary.

[0064] In other words, a "summary of summaries" is generated. This provides the user with an overview of the entire case and she or he can trace back to the individual text summaries for the individual compartments. Furthermore, by basing the overall summary on the text summaries of the individual compartments, the overall summary is structured in nature, which makes it more accessible to the user.

[0065] According to some examples, the step of generating an overall text summary includes determining a weight for each of the text summaries, wherein the overall summary reflects the weight. According to some examples, the weight includes a weight according to a relevance level (optionally according to a relevance level as described below), wherein the relevance level can indicate the clinical relevance of the corresponding text summary.

[0066] According to one aspect, the method also includes: classifying a plurality of different compartments based on a text summary (a step of generating corresponding text summaries) according to at least two relevance levels (or criticality levels) to provide a classification result for each of the plurality of compartments, optionally wherein one of the relevance levels indicates that no medically relevant findings are present in the corresponding compartment; and providing the classification result to a user via a user interface.

[0067] In other words, a classification of natural language text based on text summaries is proposed (rather than image data based on, for example, a medical image dataset). According to some examples, the step of classification can be performed by a suitably configured text generation function. According to other examples, the step of classification can be performed by a separate classification function configured to classify text into different relevance levels.

[0068] The classification step may be based on identifying predefined trigger words in the corresponding text excerpt, which may indicate a particular relevance or criticality of the findings included in the corresponding text excerpt.

[0069] According to some examples, in the step of classifying, the compartments may be classified according to two correlation levels. According to some examples, the two correlation levels may include "normal" and "abnormal".

[0070] According to other examples, the compartments may be classified according to more than two relevance levels and in particular according to continuous relevance levels. For example, in the step of classification, a number, for example between zero and one, may be assigned to each text excerpt, wherein the number indicates the respective relevance or criticality level.

[0071] According to some examples, "normal" may mean that based on the corresponding text summary, the compartment is medically unnoticeable. According to some examples, "normal" may mean that the text summary does not include (or indicate) any medical findings (or medical abnormalities).

[0072] A medical finding may indicate a certain condition or pathology of the patient that is relevant to the patient's diagnosis and requires the user's attention. A medical finding may be an anatomical structure that distinguishes the patient from other patients. A medical finding may be a medical abnormality. A medical finding may be located within different organs of the patient (e.g., within the patient's lungs or within the patient's liver) or between the patient's organs. In particular, a medical finding may be a foreign body. In particular, a medical finding may be a vegetation (also referred to as a "tumor"), in particular a benign vegetation, an in situ vegetation, a malignant vegetation, and / or a vegetation of uncertain / unknown behavior. In particular, a medical finding may be a nodule, in particular a lung nodule. In particular, a medical finding may be a lesion, in particular a lung lesion.

[0073] According to some examples, where medical findings or anomalies are actively mentioned in the text summary, those compartments are classified as "non-normal".

[0074] According to some examples, "normal" may mean that the text summary does not include (or indicate) any actionable medical abnormalities. This may mean that there are no findings at all, or only non-actionable findings. Non-actionable findings or abnormalities are findings that do not require any immediate action by the user. Examples of non-actionable abnormalities may be, for example, anatomical variations, degenerative changes (e.g., bone changes in elderly patients), healing fragments, or tiny lung nodules that do not require follow-up. Other examples of such non-actionable findings that may still classify a medical image dataset as normal are "hardware" that has been imaged, such as surgical clips, (neck) straps, implants, ECG leads, etc. Such findings may be assigned a lower level of relevance or criticality.

[0075] According to some examples, the classification result provided in the classification step indicates whether the corresponding text summary generally indicates that the potential compartment is normal. According to some examples, some compartments may be normal while other compartments are abnormal.

[0076] According to some examples, the relevance levels may be provided as an overlay above the representation of the medical image dataset. In particular, in the overlay, the relevance levels may be encoded with markers or color coding. According to some examples, providing the relevance levels may include selectively showing a text summary of one or more relevance levels in a user interface, and hiding the text summaries of other relevance levels. For example, only text summaries classified as abnormal may be shown.

[0077] By providing classification results based on the text summary, additional levels of summaries can be easily and reproducibly provided. In other words, a higher level of medical diagnosis is provided based on further processing of the text summary. Thus, further assistance is provided to the user to focus her or his attention to provide a medical diagnosis.

[0078] According to some examples, the method further includes generating a ranking of the text summaries, optionally based on the relevance levels, and providing the ranking.

[0079] According to some examples, the sorting can be based on clinical relevance (e.g., level of relevance) or criticality. According to some examples, the sorting can be provided to the user in the form of an ordered list (from which the user can select). By sorting, an indication is provided to the user as to which of the text summaries is the most relevant text summaries.

[0080] According to some examples, the ranking can be generated based on predetermined rules or criticality identifiers, such as the number of mentions of a particular finding in a compartment, changes (deterioration) in findings in a compartment over time, relevance / criticality of the compartment, relevance / criticality of the findings, etc.

[0081] According to some examples, the providing step includes obtaining a compartment of interest of a user, selecting a text summary related to the compartment of interest from corresponding text summaries, and providing the selected text summary to the user via a user interface.

[0082] The compartment of interest may be a compartment to which the user currently focuses her or his attention. In addition, the compartment of interest may be a compartment that the user should consider. The compartment of interest may be obtained by monitoring the user. In addition, the compartment of interest may be obtained based on supplementary information. For example, if the supplementary information indicates that the user wants to check a specific compartment (e.g., in the form of a diagnostic task), the compartment may be identified as a compartment of interest. According to some examples, the step of providing the selected text summary may again include displaying the selected text summary in the user interface.

[0083] By identifying the compartments of interest and showing the corresponding text summaries, relevant information can be automatically provided to the user. In other words, instead of providing all text summaries, only those text summaries that the user is interested in are provided. Thus, the user can be more assured when screening complex cases.

[0084] According to some examples, the method further comprises obtaining at least one imaging parameter of the medical image dataset, the imaging parameter being related to settings used during acquisition and / or pre-processing or post-processing of the medical image dataset, wherein the step of obtaining the compartment of interest is also based on the imaging parameter.

[0085] According to some examples, the imaging parameters include one or more of the following parameters:

[0086] - patient position during the image acquisition process,

[0087] - an image acquisition protocol for acquiring raw data, based on which a medical image dataset is generated, and / or

[0088] -Image reconstruction protocols for generating medical image datasets based on raw data.

[0089] The acquisition protocol may relate to the type of medical imaging modality used to acquire the medical image data or potentially raw data. For example, the acquisition protocol may specify whether an MRI system or a CT system has been used. In addition, the acquisition protocol may relate to the settings used for the medical imaging modality during acquisition. Taking an MRI system as an example, this may include the MR pulse sequence used.

[0090] The reconstruction protocol may be related to corresponding settings and reconstruction algorithms for processing the acquired raw data to provide a medical image data set. Taking CT imaging processing as an example, the reconstruction protocol may specify the kernel (or convolution algorithm) used. A kernel refers to a process for modifying the frequency content of the projection data prior to back-projection during image reconstruction in a CT scanner. The process corrects the image by reducing blur. The kernel affects the appearance of image structures by sharpening the image. Different kernels have been developed for specific anatomical applications, including soft tissue (standard kernels) and bone (bone kernels).

[0091] Considering imaging parameters when deriving compartments of interest is beneficial because imaging parameters provide additional insights into organ types, anatomical structures, and medical findings that medical image datasets can indicate. Thus, the prediction of compartments of interest and, thereby, the associated textual overview can be improved.

[0092] According to some examples, obtaining the compartment of interest includes receiving a user input into a user interface, and determining the compartment of interest based on the user input.

[0093] The user input may be any input directed to the compartment of interest. The user input may include voice commands or any other input, in particular manual input to a user interface, in particular a graphical user interface. For example, the user may use an input device such as a computer mouse, a trackball device, a smart pen, a touchpad, a touch-sensitive display, etc. Furthermore, the user input may be captured by eye tracking or by tracking gestures.

[0094] According to some implementations, the user input may include multiple separate user interactions with the user interface (eg, user input regarding representations, display settings, general settings, measurements, etc.).

[0095] By predicting the compartments of interest based on the user input, an appropriate text summary is automatically provided to the user. Thus, the user is automatically provided with actionable results that he can immediately use in the downstream reporting workflow. Therefore, reviewing medical images and compiling structured medical reports based thereon can be rendered quite efficient.

[0096] According to some examples, receiving user input may include recording an audio and / or video signal of the user's environment (e.g., recorded using an appropriate recording device), and identifying user input in the signal. According to some examples, the signal includes a recording of a conversation of multiple users (e.g., in the case of a joint tumor discussion meeting), and the step of identifying includes identifying one or more user inputs in the conversation. In other words, the method can "listen" to the user or a group of users and automatically select an appropriate text summary based on the current topic of the conversation, etc. Thereby, the method interactively supports the user even if the user does not actively input into the user interface.

[0097] According to some examples, the method further comprises generating a representation of the medical image dataset for display to a user via a user interface, displaying the representation to the user via the user interface, wherein in the step of receiving user input, the user input is directed to a compartment of interest visible in the representation.

[0098] According to some examples, the user input comprises an interaction with the representation. In particular, the user input may comprise specifying the compartment of interest directly in the representation, for example by clicking at a specific location in the representation, drawing an outline, or invoking a measurement tool to specify the compartment of interest. The user input may be such that it completely lines out the compartment of interest, or such that it indicates only a portion of the representation or even only a point in the representation, which is then automatically associated with the compartment of interest.

[0099] By providing representations and recognizing user interactions in the form of interactions with the representations, continuous human-computer interaction is achieved, which is seamlessly integrated in the reporting workflow.

[0100] According to some examples, the method further comprises: determining the anatomical location to which the user input is directed based on the user input and the representation and / or the medical image dataset. Furthermore, the step of determining the compartment of interest is based on the anatomical location.

[0101] According to some examples, the anatomical location includes one or more of the following elements:

[0102] - an indication of the organ or organ part to which the user input is directed,

[0103] - a slice in a medical image dataset corresponding to the representation, and / or

[0104] - User input's relationship to one or more anatomical landmarks.

[0105] Using the anatomical location, information indicative of the anatomical context of the user input and thereby the compartment of interest can be retrieved. Thereby, the compartment of interest can be identified, leading to better results in the prediction stage as well as in downstream reporting processing.

[0106] According to an aspect, the method further comprises obtaining one or more display settings applied by a user for the representation when displaying the representation, and the step of obtaining the compartment of interest is also based on the display settings.

[0107] According to one aspect, the display settings include one or more of the following settings:

[0108] - Organ segmentation applied to representation,

[0109] - the intensity window to apply to the representation,

[0110] - Contrast and / or brightness adjustments applied to the representation

[0111] - a lookup table applied to the representation,

[0112] - the self-observation settings applied to the representation,

[0113] - the viewing plane to be applied to the representation, and / or

[0114] - The zoom level or pan to apply to the representation.

[0115] Taking CT image data as an example, the representation can be intensity filtered. Each voxel of a CT image data set typically has an intensity value that represents the attenuation of X-rays at the corresponding position in the volume determined based on CT measurements (typically measured in Hounsfield or HU units). Due to this relationship, a certain segmentation can be performed based solely on a threshold of the voxel intensity values. For example, all voxels with intensity values ​​within a certain range can be considered to represent bones. Other windows can be associated with demarcating soft tissue or lung parts. In the art, these windows are generally referred to as HU windows. For example, if a bone window is applied, it can be concluded that the compartment of interest may be associated with the patient's bone structure.

[0116] The self-observation settings may be specifically related to settings for a specific diagnostic task or a specific organ. The self-observation settings may be provided by a user interface for selection by the user. Since the self-observation settings are specific to a certain diagnostic task or anatomical structure, their knowledge may provide insights into compartments that may be of interest.

[0117] In particular, the viewing plane may be selected from an axial or sagittal or coronal viewing plane. Generally speaking, it relates to the viewing plane under which the medical image dataset (or possible image volume) is viewed. In a diagnostic workflow, the viewing direction is already an indication of the compartment of interest to the user.

[0118] Generally, it is beneficial to consider the display or viewing setup when obtaining a compartment of interest, as such a display setup provides valuable insights into the types of organs, anatomical structures, and medical findings that are of interest to the user.

[0119] According to some implementations, the step of determining the anatomical location can also be based on a display setting.

[0120] According to an aspect, the user input is directed to generating measurements of image features depicted in the representation, and the step of obtaining the compartment of interest is also based on the measurements.

[0121] For example, the measurement may be a volume, surface, angle, or distance in the representation. Furthermore, the measurement may involve defining a region of interest in the representation. Furthermore, the measurement may be based on image data associated with the user input. For example, the intensity distribution of such image data (e.g., measured in Hounsfield units) may indicate whether a certain medical finding is associated with a cyst or a malignancy.

[0122] Since measurements can be indicative of compartments, considering measurements when predicting compartments of interest can improve results.

[0123] According to an aspect, generating the measurements involves selecting, by a user, an image measurement tool from a plurality of available image measurement tools, and the step of predicting the compartment of interest is further based on the measurement tool selected by the user.

[0124] Typically, there will be various measurement tools available to the user. For example, such tools may be geometric measurement tools, volume measurement tools, image processing tools, and / or computer-aided detection tools. Therefore, the kind of tool invoked by the user also indicates the compartment of interest. Therefore, using this information in the prediction step may further improve the results.

[0125] According to one aspect, the text summary has a predetermined text length, and the providing step includes: receiving a request from a user to provide additional details, and extending the text summary to text of a text length longer than the predetermined text length based on the supplemental information using a text generation function.

[0126] In other words, upon request, additional information may be extracted from the supplementary information and integrated into an extended version of the text summary. The additional information may be information that was previously ignored for a more compressed initial version of the text summary. For example, if the text summary contains a certain number of points and / or sentences, the extended text summary may include additional points or sentences. Thus, the additional information may be provided to the user only if he or she explicitly requests it. Thus, different levels of detail may be provided in a continuous human-computer interaction without overburdening the user with information overflow.

[0127] According to one aspect, the method also includes: receiving a natural language query about a text summary from a user; inputting the natural language query into a text generation function to generate a natural language answer to the natural language query by applying the text generation function to supplemental information and / or general information; and outputting the natural language answer to the user (e.g., via a user interface).

[0128] According to some examples, general (medical) information may be non-patient-specific medical information. The general information may be valid for multiple patients and / or patient groups. According to some examples, the general information may include one or more medical guidelines and / or one or more electronic textbooks.

[0129] The user query may be a free text query that a user may enter into the user interface, for example, by typing the query into an appropriate input field in the user interface or by voice command. The query may ask for additional information from the patient's EMR. In addition, the query may ask for more general information that can be answered by considering the general information. For example, the user query may be of the following kind: "Show me additional information about finding XY", "What was the size of finding XY in a previous exam", "What kind of therapy has been prescribed", "What kind of therapy is being questioned", "Who is the referring doctor", etc.

[0130] By enabling free text queries from users, users can easily request additional information without requiring the user to interact directly with the data. This simplifies the workflow and reduces the user's workload.

[0131] According to one aspect, the step of providing the text summary includes: obtaining, for at least one item in the text summary, an electronic link to a source of the item in the supplemental information, wherein the link is configured to enable retrieval of the source in a user interface; and including the link in the text summary.

[0132] A source may be a data element included in the supplemental information, such as a document (eg, a medical report), an image dataset, a non-image dataset (eg, laboratory values), and the like.

[0133] Providing links enables quick retrieval of the source documents that form the basis of a certain term in the text summary. This, in turn, enables the user to verify a given element in the text summary and to gather additional information by inspecting the source. Furthermore, this provides an explanation as to why a certain term was included in the text summary.

[0134] According to some examples, the method further comprises displaying a representation of the source in the user interface. According to some examples, displaying the representation of the source comprises: receiving a user input directed to display the source, and displaying the representation of the source in response to the user input. The representation of the source may be a rendering of the source, such as a rendering of a medical image dataset or a document.

[0135] According to an aspect, the text generation function is further configured to detect changes in one or more terms of the text summary over time, and the step of providing the text summary includes indicating the changes to the user.

[0136] By detecting this change, worsening or improvement of certain conditions of the patient can be identified.

[0137] According to some examples, changes may be determined based on timestamps of various elements in the supplemental information. For example, a reference text summary may be generated based on data elements generated before a certain deadline, i.e., a text generation function may be applied only to such data elements. Then, the text generation function may be applied to all data elements to generate a text summary. The most recent changes may then be determined by comparing the text summary to the reference text summary.

[0138] According to one aspect, a system for providing a text summary is provided. The system includes an interface unit and a computing unit. The interface unit is configured to: receive a medical image dataset of a patient; access supplementary information associated with the patient in a medical information system; and provide a text summary to a user. The computing unit is configured to: identify at least one compartment in the medical image dataset; provide a text generation function configured to provide, for a specific compartment, a natural language text summary summarizing medical information related to the specific compartment; and apply the text generation function to the supplementary information to generate a text summary for the at least one compartment.

[0139] The computing unit can be implemented as a data processing system or a part of a data processing system. Such a data processing system can, for example, include a cloud computing system, a computer network, a computer, a tablet computer, a smart phone, etc. The computing unit can include hardware and / or software. The hardware can include, for example, one or more processors, one or more memories and combinations thereof. One or more memories can store instructions for performing method steps according to the present invention. The hardware can be configured and / or operated by software. Typically, all units, subunits or modules can, for example, be connected via a network or corresponding interfaces at least temporarily exchange data with each other. Therefore, each unit can be positioned to be separated from each other.

[0140] The interface unit may comprise an interface for data exchange with a local server or a central web server via an internet connection to receive the medical image data set. The interface unit may also be adapted to interface with one or more users of the system, for example by displaying the processing results of the computing unit to the user (e.g., in a graphical user interface) or by allowing the user to adjust parameters for data processing or visualization. In other words, the interface unit may comprise a user interface.

[0141] According to other aspects, the present invention also relates to an integrated data management system comprising the above system and a medical information system. In addition, such an integrated data management system may include an image archiving system configured to acquire, store and / or forward medical images. Thus, the interface unit may be configured to receive a medical image dataset from the image archiving system. According to some examples, the image archiving system may be implemented as a cloud storage device or a local storage device or an extended storage device, for example, as a PACS (Picture Archiving and Communication System).

[0142] According to other aspects, the system is adapted to implement the method for providing candidate medical findings of the present invention in its various aspects.The advantages described in conjunction with the method aspects may also be achieved by correspondingly configured system components.

[0143] According to another aspect, the present invention relates to a computer program product comprising program elements which, when loaded into a memory of a computing unit of a system configured to provide text summarization for a medical image data set, cause the computing unit to perform the steps according to one or more of the method aspects described above.

[0144] According to another aspect, the present invention relates to a computer-readable medium having program elements stored thereon, which can be read and executed by a computing unit of a system for providing text summaries for a medical image data set to perform steps according to one or more method aspects when the program elements are executed by the computing unit.

[0145] Implementing the invention by means of a computer program product and / or a computer readable medium has the advantage that already existing provisioning systems can be easily adapted by means of a software update to work as proposed by the invention.

[0146] A computer program product may for example be a computer program, or comprise another element adjacent to such a computer program. The other element may be hardware, such as a storage device on which the computer program is stored, a hardware key for using the computer program, etc., and / or software, such as documentation or a software key for using the computer program. The computer program product may also comprise development material, a runtime system and / or a database or library. The computer program product may be distributed among several computer instances. BRIEF DESCRIPTION OF THE DRAWINGS

[0147] The characteristics, features and advantages of the above invention and the ways to achieve them become clearer and easier to understand according to the following description of the embodiments, which will be described in detail with reference to the accompanying drawings. The following description does not limit the invention to the embodiments included. The same parts, parts or steps may be marked with the same reference numerals in different figures. Generally, the drawings are not drawn to scale. In the following drawings:

[0148] Figure 1 schematically depicts a system for providing text summarization according to an embodiment,

[0149] Figure 2 schematically depicts a method for providing a text summary according to an embodiment,

[0150] Figure 3 The data flow in the method for providing text summarization according to the embodiment is schematically depicted,

[0151] Figure 4 schematically depicts optional method steps in a method for providing a text summary according to an embodiment,

[0152] Figure 5 schematically depicts a data flow in a method for providing a summary according to an embodiment, and

[0153] Figure 6 A text generation function according to an embodiment is schematically depicted. DETAILED DESCRIPTION

[0154] Figure 1 A system 1 for providing a text summary TS for a medical image dataset MIDS to a user U is depicted. In this regard, the system 1 is adapted to perform a method according to one or more embodiments, for example, as described with reference to Figures 2 to 5 Further described.

[0155] According to some examples, a user U of the system 1 (in the sense of an operator controlling the system 1 ) may typically relate to a healthcare professional, such as a physician, clinician, technician, radiologist, or the like.

[0156] The system 1 includes a user interface 10 (as part of an interface unit) and a processing system 20 (as part of a computing unit). In addition, the system 1 may include a medical information system 40 or be connected to a medical information system 40. The medical information system 40 may generally be configured to acquire and / or store and / or forward a medical image data set MIDS and supplementary information SI. For example, the medical information system 40 may include one or more archive / review stations (not shown) for the medical image data set MIDS. The archive / review station may be implemented by one or more databases. In particular, the archive / review station may be implemented in the form of one or more cloud storage modules. Alternatively, the archive / review station may be implemented as a local storage device or an extended storage device, for example, as a PACS (picture archiving and communication system). According to some examples, the medical information system 40 may also include one or more medical imaging modalities (not shown), such as a computed tomography system, a magnetic resonance system, an angiography (or C-arm X-ray) system, a positron emission tomography system, a mammography system, an X-ray system, and the like.

[0157] The medical image dataset MIDS may be a three-dimensional image dataset acquired, for example, using an X-ray system, a computer tomography system, or a magnetic resonance imaging system or other system. The image information may be encoded in a three-dimensional array of m×n×p voxels. The medical image dataset MIDS may include a plurality of image slices stacked in a stacking direction to span an image volume covered by the medical image dataset MIDS.

[0158] Furthermore, the medical image dataset MIDS may include two-dimensional medical image data, wherein image information is encoded in an array of m×n pixels. According to some examples, these two-dimensional medical images may be extracted from the three-dimensional medical image dataset MIDS.

[0159] A collection of voxels or pixels may be designated as image data of the corresponding medical dataset MIDS below. Generally, the medical dataset MIDS may show a body part or an anatomical region or an anatomical object of a patient, which may include various anatomical structures and organs.

[0160] The medical image dataset MIDS may be formatted according to the DICOM format. DICOM (= Digital Imaging and Communications in Medicine) is an open standard for the communication and management of medical imaging information and related data in healthcare informatics. DICOM can be used to store and transmit medical images and associated information, enabling the integration of medical imaging devices such as scanners, servers, workstations, printers, network hardware, and Picture Archiving and Communication Systems (PACS). DICOM is widely adopted by clinical syndicates, hospitals, and for smaller applications such as doctors' offices or clinics. DICOM data objects consist of multiple attributes, including items such as patient name, ID, etc., and also include special attributes containing image pixel data and metadata extracted from the image data.

[0161] The supplementary information SI (or associated data) may be any data that provides additional information related to the patient and / or the medical image dataset MIDS. The supplementary information SI may include image data, such as other medical image studies of the patient, such as acquired at an earlier point in time than the medical image dataset MIDS under consideration. Furthermore, the supplementary information SI may include non-image data or data with mixed-type content, which includes medical images and non-image content such as text. The non-image data may relate to non-image examination results, such as laboratory data, vital signs records (including, for example, ECG data, blood pressure values, ventilation parameters, oxygen saturation levels), etc. Furthermore, the supplementary information SI may include structured and unstructured medical text reports MTR related to a previous examination or a current examination of the patient. Furthermore, the non-image data may include personal information of the patient, such as gender, age, weight, insurance details, etc.

[0162] The supplementary information SI may be obtained in the form of one or more electronic medical records (EMR) of the patient. The supplementary information SI may be stored in the healthcare information system 40. For example, the supplementary information SI may be stored in a dedicated database of the healthcare information system 40, such as a laboratory information system (LIS) or an electronic health / medical record database.

[0163] The user interface 10 may include a display unit and an input unit. The user interface 10 may be implemented by a mobile device such as a smart phone or a tablet computer. In addition, the user interface 10 may be implemented as a workstation in the form of a desktop PC or a laptop computer. The input unit may be integrated in the display unit, for example, in the form of a touch screen. As an alternative or in addition, the input unit may include a keyboard, a mouse or a digital pen and any combination thereof. The display unit may be configured to display a representation R of the medical image dataset MIDS, to display one or more text summaries TS, and to receive any user input INP, for example for specifying a compartment of interest COI in the medical image dataset MIDS.

[0164] The user interface 10 may also include an interface computing unit, which is configured to execute at least one software component for serving a display unit and an input unit, to provide a graphical user interface for allowing a user U to select a case of a target patient to be reviewed and to perform various inputs. In addition, the interface computing unit may be configured to communicate with a medical information system 40 or a processing system 20 to receive a medical data set MIDS, any supplementary information IS, and a text summary TS. The user U may activate the software component via the user interface 10, and may obtain the software component, for example, by downloading the software component from an Internet application store. According to an example, the software component may also be a client-server computer program in the form of a web application running in a web browser. The interface computing unit may be a general-purpose processor, a central processing unit, a control processor, a graphics processing unit, a digital signal processor, a three-dimensional rendering processor, an image processor, an application-specific integrated circuit, a field programmable gate array, a digital circuit, an analog circuit, a combination thereof, or other now known devices for processing image data. The user interface 10 may also be implemented as a client.

[0165] The processing system 20 may comprise sub-units 21 to 24 configured to process the medical image dataset MIDS and the supplementary information SI to provide a text summary TS for a specific compartment CP shown in the medical image dataset MIDS.

[0166] The processing system 20 may be a processor. The processor may be a general purpose processor, a central processing unit, a control processor, a graphics processing unit, a digital signal processor, a three-dimensional rendering processor, an image processor, an application specific integrated circuit, a field programmable gate array, a digital circuit, an analog circuit, a combination thereof, or other currently known devices for processing image data. The processor may be a single device or multiple devices operating in series, in parallel or individually. The processor may be the main processor of a computer such as a laptop or a desktop computer, or may be a processor for processing some tasks in a larger system such as a medical information system or a server. The processor is configured by instructions, design, hardware and / or software to perform the steps discussed herein. The processing system 20 may be included in the user interface 10. Alternatively, the processing system 20 may include a real or virtual computer group, such as a so-called "cluster" or "cloud". Such a server system may be a central server, such as a cloud server; or a local server, such as a server located at a hospital or a radiation site. In addition, the processing system 20 may include a memory such as a RAM for temporarily loading the medical image data set MIDS. According to some examples, such a memory may also be included in the user interface 10.

[0167] The subunit 21 is a data retrieval module or unit. The subunit 21 is configured to access and search the medical information system 40 for the medical image dataset MIDS and the supplementary information SI. Specifically, the subunit 21 can be configured to formulate a search query and parse it to the medical information system 40. According to some examples, the search query can be based on the electronic patient identifier of the patient in the medical information system 40.

[0168] The subunit 22 may be considered as a segmentation module or unit. The subunit 22 may be configured to identify a plurality of compartments CP in the medical image dataset MIDS. To this end, the segmentation unit 22 may be configured to run one or more appropriate detection or segmentation algorithms.

[0169] The subunit 23 may be configured as a user interaction module or unit. The subunit 23 may be configured to provide a representation R for display to the user U via the user interface 10. The representation R may be in the form of a rendering of the medical image data set MIDS. In addition, the subunit 23 may be configured to identify the compartment of interest COI in the representation R (and thereby in the medical image data set MIDS) based on the corresponding input INP of the user U. For example, the subunit 23 may be configured to provide a corresponding tool, which the user U may activate via the user interface 10. After the tool has been activated, user input INP such as speech, gesture, eye movement, processing of an input device such as a computer mouse, etc. may be evaluated to derive the compartment of interest COI currently being reviewed by the user U. According to an example, such user input INP may specify a point or a set of points in the representation R, which are then further processed to define the corresponding compartment of interest ROI. According to other examples, the tool may be a measurement tool, with which the user U may obtain measurements from the representation R. The measurements may involve volumes, surfaces, angles, distances, etc.

[0170] Furthermore, the subunit 23 may be configured to provide the user U with a text summary TS corresponding to the identified compartment CP, and in particular the compartment of interest COI, for example in the form of a pop-up window above the representation R at the location of the corresponding compartment CP.

[0171] In addition, the subunit 23 may also be configured to receive and process other kinds of user inputs to control the method and allow continuous and guided human-computer interaction.Such user inputs may involve queries for additional information based on the text summary TS or other queries.

[0172] The subunit 24 may be considered as a text generation module or unit. The subunit 24 is configured to filter the supplementary information SI for information related to a specific compartment CP and to generate a text summary TS for the specific compartment on this basis. In other words, the subunit 24 may be configured to extract information from the supplementary information SI and convert the information into a text suitable for a concise text summary TS in a compartment-specific manner. To this end, the submodule 24 may run a correspondingly configured text generation function LLM.

[0173] The designation of the different subunits 21 to 24 will be explained by way of example and not limitation. Thus, the subunits 21 to 24 may be integrated to form a single unit (e.g., in the form of a "computing unit") or may be implemented by computer code segments configured to execute corresponding method steps running on a processor or the like of the processing system 20. The same is true for the interface computing unit. Each subunit 21 to 24 and the interface computing unit may be individually connected to other subunits and / or other components of the system 1 that require data exchange to perform method steps.

[0174] The processing system 20 and the interface computing unit can together constitute the computing unit of the system 1. It is worth noting that the layout of the computing unit, that is, the physical distribution of the interface computing unit and the subunits 21 to 24, is arbitrary in principle. Specifically, the processing system 20 can also be integrated in the user interface 10. As already mentioned, the processing system 20 can alternatively be implemented as a server system such as a cloud server, or a local server such as a server located at a hospital or a radiation site. According to such an implementation, the user interface 10 can be designated as a "front end" or "client" facing the user U, and the processing system 20 can be considered as a "back end" or server. For example, the https protocol can be used to implement the communication between the user interface 10 and the processing system 20. The computing power of the system can be distributed between the server and the client (i.e., the user interface 10). In a "thin client" system, most of the computing power exists at the server. In a "fat client" system, more computing power and possible data exist on the client.

[0175] The various components of the system 1 can be connected to each other at least temporarily for data transmission and / or exchange. The user interface 10 communicates with the processing system 20 via a (data) interface 26 to exchange, for example, a medical image data set MID, a representation R, a text summary TS, or any user input INP made. For example, the processing system 20 can be activated based on a request, wherein the request is sent by the user interface 10. In addition, the processing system 20 can communicate with the medical information system 40 to retrieve the case of the target patient. As an alternative or in addition, the user interface 10 can communicate directly with the medical information system 40. The medical information system 40 can also be activated based on a request, wherein the request is sent by the processing system 20 and / or the user interface 10. The data interface 26 for data exchange can be implemented as a hardware interface or a software interface, such as a PCI bus, USB or FireWire. Data transmission can be implemented using a network connection. The network can be implemented as a local area network (LAN), such as an intranet or a wide area network (WAN). The network connection is preferably wireless, such as a wireless LAN (WLAN or Wi-Fi). In addition, the network can include a combination of different network examples. The interface 26 for data exchange and the components for interfacing with the user U are regarded as constituting an interface unit of the system 1 .

[0176] Figure 2 A method for providing a text summary TS according to an embodiment is described. The corresponding data flow is as follows Figure 3 As shown. The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but may also vary between different embodiments of the invention. In addition, individual steps or a series of steps may be repeated.

[0177] In a first step S10, a medical image dataset MIDS is received. This may involve selecting the medical image dataset MIDS from a plurality of cases stored, for example, in the medical information system 40. The selection may be performed manually by the user U, for example by selecting appropriate image data in a graphical user interface running in the user interface 10. Alternatively, the medical image dataset MIDS may be provided to the computing unit by the user U by uploading the medical image dataset MIDS to the computing unit.

[0178] In step S20, one or more compartments CP in the medical image dataset MIDS are identified. The identified compartments CP may be organs, anatomical structures or functional units of the patient's body at least partially shown in the medical image dataset MIDS. According to some examples, all or at least all relevant compartments CP shown in the medical image dataset MIDS may be identified at step S20.

[0179] In order to identify the compartment CP, step S20 may involve applying one or more appropriately configured detection algorithms to the medical image dataset MIDS. The detection algorithm may include a medical image segmentation algorithm configured to identify and segment a specific compartment in the medical image data. In addition, the detection algorithm may include an image classification algorithm configured to identify a specific compartment based on image features extracted from the medical image dataset MIDS. Therefore, step S20 may include applying a plurality of different detection algorithms to the medical image dataset MIDS. In addition, step S20 may include selecting one or more detection algorithms from a plurality of available detection algorithms and applying the selected detection algorithms to the medical image dataset MIDS. According to some examples, the selection may be based on an indication of the body region shown in the medical image dataset MIDS or which compartments CP may be of particular interest. The latter may be based on user input in the system 1 or based on a concept of the reason for the examination (i.e., the medical image dataset MIDS).

[0180] At step S30, the patient's supplementary information SI is accessed. The supplementary information SI may be all available information of the patient in the medical information system 40. The supplementary information SI may include non-image data, and in particular one or more structured or unstructured (text) documents generated at different points in the past.

[0181] The supplementary information SI may be obtained by querying the medical information system 40 , for example, based on the electronic patient identifier of the patient in the medical information system 40 .

[0182] At step S40 , the text generation function LLM is provided. For example, the text generation function LLM may be provided, for example as an executable computer code, by keeping it available in a memory of the processing system 20 .

[0183] The text generation function LLM is configured to parse the patient information (e.g., supplementary information SI) to obtain information related to a specific compartment CP. In addition, the text generation function LLM is configured to compress the information thus retrieved into a text summary text TS, which may have a predefined maximum text length, such as a certain number of bullet points.

[0184] Typically, the text generation function LLM may be a trained function that is configured in a dedicated training process for the task of generating a text summary TS. According to some examples, the text generation function LLM may be trained using a compartment-specific text summary TS generated by an expert. Specifically, in the training process, the text summary TS generated by the text generation function LLM may be compared with the expert text summary TS. Based on the comparison, the parameters of the text generation function LLM may be adaptively adjusted so that the text generation function LLM better reflects the expected behavior as specified by the expert text summary TS. In this way, the text generation function LLM may learn which data items in the supplementary information SI are related to which compartment CP, and which data items are more important than other data items and should be prioritized for the text summary TS. In combination Figure 6 A more specific example of the text generation function LLM is provided.

[0185] According to some examples, the prioritization of information for text summarization TS can be based on the time point of data generation (new data is preferred over old data) and / or the medical criticality of the data (more critical findings are preferred over less critical findings).

[0186] At step S50, the supplementary information SI and the compartment CP identified at step S20 are provided to the text generation function LLM, and the text generation function LLM is caused to generate a text summary TS according to the identified compartment CP. According to the setting of the text generation function LLM, the command of the text generation function LLM can be formulated as a prompt.

[0187] The content of the text summary TS is not based on actual image data of the medical image dataset MIDS but on supplementary information SI outside the medical image dataset MIDS. With the help of the shown compartments, the image data of the medical image dataset MIDS only serve as a hint of what to look for in the (usually large) supplementary information SI.

[0188] At step S60, the text summary TS thus generated may be provided. This may involve showing the text summary TS in a user interface 10, such as a suitable graphical user interface. According to some examples, the text summary TS may be shown as an overlay above the representation R rendered according to the medical image dataset MIDS. In addition, individual text summaries TS may be shown based on a request (i.e., in response to a corresponding request of the user U). For example, if the user U hovers over a compartment CP in the representation R with a mouse cursor or the like, the text summary TS may be automatically shown.

[0189] At optional step S70, an overall text summary can be generated. The overall text summary can be configured to summarize each compartment-specific text summary TS. Thus, the main findings related to the medical image data set MIDS can be concentrated in one summary. The overall text summary can be generated by applying a text generation function LLM (or another text generation function) to each text summary TS. In order to decide which aspects are relevant enough for the overall summary, in principle, the same prioritization concept discussed in conjunction with steps S40 and S50 can be relied on. According to some examples, the overall text summary can be generated based on a request from a user U.

[0190] At optional step S80, the text summary TS may be classified according to at least two relevance levels. In other words, a classification result for each of the plurality of compartments CP based on the text summary TS may be provided, the classification result indicating the relevance of the previously collected information to the corresponding compartment CP.

[0191] According to some examples, there may be only two relevance levels, which may be: no finding (or "normal") and finding (or "not normal"). Thus, the user U may get a quick overview of compartments CP that were not previously found in the records in the supplementary information.

[0192] Furthermore, there may be additional relevance levels or even a continuum of relevance levels to indicate how critical the finding is from a medical perspective. For example, a healed fracture may be assigned a lower level of criticality as a result of the classification than a previously reported lung nodule.

[0193] At optional step S81, the classification result may be provided. According to some examples, the text summary TS displayed in the user interface 10 may be color-coded according to the classification result and / or provided with a visual mark indicating the classification result. In addition, the classification result may also be provided in such a way that only text summaries TS having a specific relevance level / above a specific relevance level are automatically provided to the user U, and thereby attract the user's attention.

[0194] Figure 4Describes the optional method steps related to providing a text summary TS. The corresponding data flow is as follows Figure 5 As shown. The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but may also vary between different embodiments of the invention. In addition, individual steps or a series of steps may be repeated.

[0195] At the optional step S61, the compartment of interest COI is obtained, and a text summary TS is specifically generated for the compartment of interest COI and / or provided to the user U. According to some embodiments, the step of obtaining the compartment of interest COI is part of step S60, that is, this step is performed to decide which one (which ones) of the generated text summaries TS are actually provided to the user U. According to other embodiments, step S61 may also be performed independently of step S60.

[0196] According to some examples, the compartment of interest COI may be determined based on the interaction of the user U with the medical image dataset MIDS. An example of such a similar process is described below in conjunction with optional steps D10 to D30.

[0197] At step D10, a representation R of the medical image dataset MIDS is rendered. The representation R may relate to a slice of the medical image dataset MIDS currently being viewed by the user U. Furthermore, the representation R may be a rendering of a three-dimensional view of the medical image dataset MIDS. The rendering of the representation R may be based on one or more imaging parameters set by the user U, such as viewing angle, filter, contrast and image enhancement settings.

[0198] At step D20 , the representation R is displayed to the user U in a graphical user interface, for example running in the user interface 10 .

[0199] Next, at an optional step D30, if a user input INP about the representation R points to a specific compartment CP within the medical image dataset MIDS, the user input INP is received and analyzed. The specific compartment can then be identified as the compartment of interest COI. For example, the user input INP may include: moving an input device such as a mouse cursor to the compartment CP shown in the representation R, activating a measurement tool specific to the compartment CP, clicking on the compartment CP shown in the representation R, etc.

[0200] At optional step S62, a text summary TS corresponding to the compartment of interest COI may be selected from the available text summary TS as generated in step S50. Alternatively, for the case where no pre-generated text summary TS is available, a text summary TS may be requested specifically for the compartment of interest COI.

[0201] Subsequently, a text summary TS of the compartment of interest COI may be provided. This may be done in much the same way as described in connection with step S60.

[0202] As mentioned, the text summary TS is intended to have a concise format focusing on the information that is most valuable to the user U. However, if the user U wants to get more information for a specific compartment, the user U may request an extended version of the text summary TS according to optional modifications.

[0203] Specifically, after receiving a corresponding request to extend a specific text summary TS (optional step D40), an extended version of the original text summary TS can be generated using the text generation function LLM. For example, the extended version may include additional information from the supplementary information SI that is not considered sufficiently relevant to the initial text summary TS. In addition, the extended text summary may include additional explanations and / or background information related to those information already included in the initial text summary TS.

[0204] As another optional extension of the workflow, it may be possible for the user U to enter a natural language query which is then answered based on the text summary TS and / or supplementary information SI or even general information not specifically related to the patient.

[0205] Such a query may be received at optional step D50. The query may directly relate to the text summary TS (e.g., "show only information added to the file after the last radiological examination", "show only text summaries related to disease XY", etc.). Additionally or alternatively, the query may also more generally relate to supplementary information SI (e.g., "when was the last blood test", "is the patient a smoker", etc.) or more general aspects (e.g., "according to our internal guidelines, which examination is recommended to ensure the diagnosis of disease XY").

[0206] At optional step S65, answers to these queries are obtained.Thus, the natural language understanding capabilities of the text generation function LLM can be exploited to firstly understand the queries and secondly translate the queries into machine understandable instructions to adapt the text summary TS or to find additional information.

[0207] One problem with the text generation function LLM and in particular the language model is that it is difficult for the user U to verify the correctness of the information provided. Optional steps S66 and S67 are intended to support the user U in this regard.

[0208] Specifically, it is proposed to obtain a link LNK to the source of an item or information in the text summary TS. The source may be a data element in the supplementary information SI from which the corresponding information in the text summary TS is extracted. Depending on the situation, the source may be a document such as a medical report, a referral letter, a patient interview protocol, a medical history questionnaire, etc.

[0209] The link LNK may be an electronic link to an item of supplementary information SI, with which the item may be directly retrieved from the medical information system 40 .

[0210] At optional step S67, a link LNK is provided. According to some examples, the link LNK may be directly included in the text summary TS. When the text summary TS is displayed, the link LNK may be visualized as an action field or button in the text summary TS. When the link LNK is activated, the source document may be retrieved and displayed to the user U. According to some examples, this may include automatically highlighting those paragraphs in the source document that are relevant to the corresponding information in the text summary TS.

[0211] In optional step S68, another optional extension process is based on the automatic change assessment of the generated text summary TS. For example, the changes can be systematically assessed by time filtering the supplementary information SI. Thus, the recent development can be contrasted with the situation at an earlier time point. This makes it possible to draw the attention of the user U to the recent changes.

[0212] exist Figure 6 In, a schematic representation of a text generation function LLM according to an embodiment is shown. The text generation function LLM according to the embodiment includes a transformer architecture. The transformer architecture follows an encoder-decoder structure and includes an encoder ENC and a decoder DEC. In short, the task of the encoder ENC is to map the input INPT to a series of continuous representations, which are then fed into the decoder DEC. The decoder DEC receives the output of the encoder ENC and the decoder output OUTR at the previous iteration to generate an output OUT.

[0213] Specifically, the input INPT may be supplementary information SI and an indication of a compartment CP for which a text summary TS is to be generated. The output OUT is a text summary TS for the compartment CP based on the supplementary information SI.

[0214] The encoder ENC of this embodiment may include a stack of N=6 identical layers. For ease of reference, only one layer xN is shown in the figure. In addition, depending on the corresponding task, N can also be set to different values, and in particular to a value greater than N=6. Each layer xN in the encoder ENC includes two sublayers L1 and L3. The first sublayer L1 implements the so-called multi-head self-attention mechanism. Specifically, the first sublayer L1 can be configured to determine the relevance of a particular word relative to other words in the input INPT. This can be represented as an attention vector. Thus, it can be determined whether a particular paragraph in the supplementary information is related to the compartment CP for which the text summary TS is to be generated. In order to avoid any deviation, multiple attention vectors for each word can be generated and fed to a weighted average to calculate the final attention vector for each word. The second sublayer L3 is a fully connected feedforward network, which can, for example, include two linear transformations, with a rectified linear unit (ReLU) activation between the two linear transformations. The N=6 layers of the encoder ENC apply the same linear transformation to all words in the input INPT, but each layer uses different weights and bias parameters to do so. Each sub-layer L1, L3 is followed by a normalization layer L2, which normalizes the sum calculated between the input fed to the corresponding sub-layer L1, L3 and the output generated by the corresponding sub-layer L1, L3 itself. In order to capture information about the relative position of the words in the input INPT, a position encoding PE is generated based on the input embedding INPT-E, and then fed to the layer xN. The position encoding PE has the same dimension as the input embedding INPT-E and can be generated using sine and cosine functions of different frequencies. The position encoding PE can then be simply added to the input embedding INPT-E to inject the position information PE. In general, the input embedding INPT-E can be a representation of each word in the input INPT, usually in the form of a real-valued vector that encodes the meaning of the word so that words that are closer in the vector space are expected to be similar in meaning. According to some examples, a neural network can be used to generate the input embedding INPT-E.

[0215] The decoder DEC of this embodiment may also include a stack of N=6 identical layers xN, each layer including three sublayers L4, L1, and L3, as described in conjunction with the encoder ENC, and the sublayers L4, L1, and L3 may be followed by a normalization layer L2. For ease of reference, only one layer xN of the decoder DEC is shown in the figure. In addition, N may also be set differently according to the corresponding task, and in particular is greater than N=6. Although the sublayers L1 and L3 of the decoder DEC correspond to the corresponding sublayers L1 and L3 of the encoder ENC in their functions, the sublayer L4 receives the previous output OUTR of the decoder DEC (if the output is a series of words, it is optionally transformed into a corresponding embedding and enhanced with position information), and implements multi-head self-attention on it, thereby weighting the importance of each element of the previous output vector OUTR. Subsequently, the value from the first sublayer L4 of the decoder DEC is input into the L1 sublayer of the decoder DEC. This sublayer L1 of the decoder DEC implements a multi-head self-attention mechanism similar to that implemented in the first sublayer L1 of the encoder ENC. On the decoder side, the multi-head mechanism receives the values ​​from the previous decoder sublayer L4 and the output of the encoder ENC. This enables the decoder to process all words in parallel. As with the encoder ENC part, the output of the L1 sublayer is passed to the feedforward layer L2, which will form the output vector into a form that is easily accepted by another decoder block or linear layer. After processing all layers xN of the decoder DEC, the intermediate results are fed to the linear layer L5, which can be another feedforward layer. The linear layer L5 is used to expand the dimension to the format expected for calculating the output vector OUT, which in this case is the precursor of the text summary TS. Subsequently, the result is passed through the Softmax layer L6, which converts the result into a text summary TS.

[0216] The various embodiments or their various aspects and features may be combined or exchanged with one another whenever it makes sense, without limiting or expanding the scope of the invention. Where applicable, the advantages described with respect to one embodiment of the invention are also advantageous for other embodiments of the invention. Independent of the grammatical term usage, individuals with male or female identities are also included in the terms.

Claims

1. A computer-implemented method for providing a text summary (TS) for a medical image dataset (MIDS), the method comprising the following steps: - receiving (S10) said medical image dataset (MIDS) of a patient, - identifying (S20) at least one compartment (ORG) in said medical image data set (MIDS), - accessing (S30) supplementary information (SI) associated with the patient in a medical information system (40), - providing (S40) a text generation function (LLM) configured to provide, for a specific compartment (ORG), a natural language text summary (TS) summarizing medical information related to the specific compartment, - applying (S50) said text generation function (LLM) to said supplementary information (SI) so as to generate a text summary (TS) for said at least one compartment (ORG), and - providing (S60) said text summary (TS) to a user (U) via a user interface (10).

2. The method according to claim 1, wherein: The text generation function (LLM) includes a transformer network and / or a large language model.

3. A method according to any one of the preceding claims, wherein: - the step of identifying (S20) comprises identifying a plurality of different compartments (ORG) in said medical image dataset (MIDS), and - the step of applying (S40) comprises applying (S50) said text generation function (LLM) to said supplementary information (SI) to generate a respective text summary (TS) for each of said plurality of compartments (ORG).

4. The method according to claim 3, further comprising: - receiving (S71) a request from said user (U) for a whole text summary (H-TS) of said medical image dataset (MIDS), - generating (S70) a whole text summary (H-TS) by applying said text generation function (LLM) to said corresponding text summary (TS), wherein said whole text summary (H-TS) comprises a summary of said corresponding text summary (TS).

5. The method according to any one of claims 3 or 4, further comprising: - classifying (S80) the plurality of different compartments based on the corresponding text summaries (TS) according to at least two relevance levels to provide a classification result for each of the plurality of compartments (ORG), optionally wherein one of the relevance levels indicates that no medically relevant findings are present in the corresponding compartment (ORG), - Providing (S81) the classification result to the user (U) via the user interface (10).

6. The method according to any one of claims 3 to 5, wherein: The step of providing (S60) comprises: - obtaining (S61) a compartment of interest (COI) of said user (U), - selecting (S62) a text summary (TS) related to said compartment of interest (COI) from said corresponding text summaries (TS), and - providing (S63) the selected text summary (TS) to the user (U) via the user interface (10).

7. The method according to claim 6, wherein: The step of obtaining (S61) the compartment of interest (COI) comprises: - receiving (D30) user input (INP) into said user interface (10), and - determining said compartment of interest (COI) based on said user input (INP).

8. The method according to claim 7, further comprising: - generating (D10) a representation (R) of the medical image dataset (MIDS) for display to the user (U) via the user interface (10), - displaying (D20) the representation (R) to the user (U) via the user interface (10), wherein - In the step of receiving (D30) said user input (INP), said user input (INP) is directed to said representation (R).

9. A method according to any one of the preceding claims, wherein: - the text summary (TS) has a predetermined text length, and - The step of providing (S60) comprises: - receiving (D40) a request (REQ) from said user (U) to provide additional details, - using the text generation function (LLM) to expand (S64) the text summary (TS) into a text having a text length longer than the predetermined text length based on the supplementary information (SI).

10. The method according to any one of the preceding claims, further comprising: - receiving (D50) a natural language query about said text summary (TS) from said user (U), - inputting (S65) the natural language query into the text generation function (LLM) to generate a natural language answer to the natural language query by applying the text generation function (LLM) to the supplementary information (SI), - outputting the natural language answer via the user interface (10).

11. A method according to any one of the preceding claims, wherein: The step of providing (S60) the text summary (TS) comprises: - obtaining (S66) for at least one item in the text summary (TS) an electronic link (LNK) to a source of said item in the supplementary information (SI), wherein said link (LNK) is configured to enable retrieval of said source in said user interface (10), - Including (S67) said link (LNK) in said text summary (TS).

12. A method according to any one of the preceding claims, wherein: - the text generation function (LLM) is further configured to determine a change in the text summary (TS) according to a change in the data state of the supplementary information (SI) over time, and - the step of providing said text summary (TS) comprises indicating (S68) said changes in said user interface (10) for said user (U).

13. A system (1) for providing a text summary (TS) for a medical image data set (MIDS), the system (1) comprising an interface unit and a computing unit, wherein: - the interface unit is configured to: - receiving (S10) said medical image dataset (MIDS) of a patient, - accessing (S30) supplementary information (SI) associated with the patient in a medical information system (40), and - providing (S60) said text summary (TS) to a user (U), and - the computing unit is configured to: - identifying (S20) at least one compartment (ORG) in said medical image data set (MIDS), - providing (S40) a text generation function (LLM), the text generation function (LLM) configured to provide, for a specific compartment, a natural language text summary (TS) summarizing medical information related to the specific compartment, and - applying (S50) said text generation function (LLM) to said supplementary information (SI), To generate a text summary (TS) for the at least one compartment (ORG).

14. A computer program product comprising program elements which, when loaded into the memory of a computing unit (20) of a system (1) for providing text summaries (TS) for medical image data sets (MIDS), cause the computing unit (20) to perform the steps of a method according to any one of claims 1 to 12.

15. A computer-readable medium having program elements stored thereon, which are capable of being read and executed by a computing unit (20) of a system (1) for providing a text summary (TS) for a medical image data set (MIDS) to perform the steps of a method according to any one of claims 1 to 12 when the program elements are executed by the computing unit (20).

Images