Method for providing medical imaging decision support data

Abstract

Embodiments of the present disclosure relate to a method for providing medical imaging decision support data. In one aspect, the present invention relates to a computer-implemented method for providing medical imaging decision support data, the method comprising: receiving natural language data, the natural language data comprising patient-specific clinical information, generating structured information by applying a large language model to the natural language data, the structured information comprising the patient-specific clinical information in a structured format, computing medical imaging decision support data based on the structured information, and providing the medical imaging decision support data.

Description

Technical Field

[0001] In one aspect, the present invention relates to a computer-implemented method for providing medical imaging decision support data. In other aspects, the present invention relates to a data processing system, a medical imaging apparatus, a computer program product and a computer-readable storage medium. Background Art

[0002] In clinical applications, it is crucial to make fast and accurate decisions. Due to practical requirements, such as timeline and medical equipment needs, typical clinical evaluations are not necessarily performed on patients, but are usually specialty-centric. That is, patients are assigned to different specialties based on the indications derived from the medical history so far.

[0003] Therefore, patient-specific clinical information must be consistently transferred between different diagnostic institutions or between clinical specialties within the same institution to the next point of care. Despite the various unified medical vocabularies, treatment and disease codes, protocols, etc., information directly related to treatment may be lost in the medical assessment, and many patients will receive delayed or interrupted treatment and / or face additional risks of harm.

[0004] Therefore, utilizing predefined and well-structured templates that give a blueprint for the required data is key to ensuring the availability of all required clinical information. While specialized techniques exist for some specific diseases, such as TNM staging and RECIST schemes for cancer diseases, RADS variants (e.g., CAD-RADS or bone RADS), etc., there is often little uniform and structured reporting of information used in a clinical setting. And even when appropriate schemes exist, as in the examples above, the way this information is actually presented can still vary significantly in practice.

[0005] The main reason for this is the lack of unified software solutions for communicating clinical information and the resulting reliance on various communication means. Clinical IT environments tend to be highly diverse, and despite the existence of unified communication protocols such as FHIR and DICOM, solutions are often vendor-specific or contain vendor-specific extensions and therefore cannot be universally used within the clinic. Furthermore, additional restrictions may be imposed due to data security and data privacy laws.

[0006] In a typical scanner workflow, a radiologist decides which imaging examination to perform based on free-text clinical indications. After reading the scan, the radiologist documents the findings in an unstructured report, which is later used as the basis for disease diagnosis. So far, the related steps, such as the selection of the scan protocol, the reading workflow, and report generation, have only been partially supported by software, as they are usually unable to process natural language.

[0007] As a result, clinical information is often communicated in an unstructured manner, most notably in unstructured clinical assessment reports or brief unstructured reports. Although clinicians are trained to use highly descriptive, concise, and compact language, these unstructured reports neither inherently (i.e., due to the technical procedures used to create them) require clinicians to include all relevant information nor necessarily serve as a precise source of information for further evaluation. Therefore, the completeness of the information in unstructured reports is primarily a result of clinical experience and adherence to guidance or regulations, and is also susceptible to the significance of the disease, practitioner fatigue, and time constraints.

[0008] It is noteworthy that unstructured text-based information plays a vital role not only between clinical institutions and departments, but also during the scanning and post-processing workflow itself. To name just a few examples, every decision regarding the choice of scanning protocol, the selected image reconstruction, and the automatically applied post-processing algorithm is directly and immediately based on the clinical indication, i.e., the reason for the examination, as well as other potentially clinically relevant information about the patient. It is noteworthy that the reason for the examination itself may be influenced by previous patient examinations. Therefore, relevant sources of information for the scanning and post-processing workflow also include previous diagnostic reports. Typically, this data is mostly unstructured and must be transformed, i.e., translated, mapped, and protocolized to form suitable imaging and post-processing procedures. This transformation can be performed manually by radiologists and technicians. As cutting-edge technologies such as photon counting CT become available, the number of available options is expected to increase steadily, and automated support for these tasks is of the highest clinical importance.

[0009] EP 3 451 211 A1 discloses a concept for controlling a medical imaging system based on indication information using an indication ontology.

[0010] There have been efforts to create automated solutions to extract key information from unstructured reports for use in workflow automation. Such approaches are typically built on keyword searches using lexical ontologies such as RadLex. Due to their nature, these approaches often suffer from incorrect information parsing because they fail to fully consider the context. Since these models have no actual understanding of the environment and instead process reports in symbolic terms, they have particular difficulty handling negations and inversions, abbreviations, additional information, and other content that disrupts the linear language structure. Furthermore, these approaches must be designed for specific clinical tasks, i.e., they can only be applied to a single or a small number of diseases, as the base ontology must be augmented with a large number of additional rules to handle the aforementioned issues.

[0011] Recent initiatives, such as the introduction of DICOM TID 1500 to provide a vendor-neutral structure for coding diagnostic findings, exist but have not been widely adopted. Furthermore, they lack a default way to fill in data and appear obtrusive in the daily clinical workflow. In clinical practice, beyond billing, coding of findings and indications, i.e. mapping to structured, predefined medical codes, does not appear to have been widely adopted. Especially for the scanning and post-processing workflow itself, many sites still rely heavily on unstructured text-based information, which has inherent shortcomings.

[0012] The technical problem underlying the present invention is to facilitate medical imaging decision support, in particular improved in natural language processing. This problem is solved by the subject matter of the independent claims. The dependent claims relate to further aspects of the present invention. Summary of the invention

[0013] In one aspect, the present invention relates to a computer-implemented method for providing medical imaging decision support data, the method comprising:

[0014] - receiving natural language data including clinical information specific to a patient,

[0015] - Generate structured information by applying large language models to natural language data, which includes patient-specific clinical information in a structured format,

[0016] - Computing medical imaging decision support data based on structured information, and

[0017] -Provide medical imaging decision support data.

[0018] In particular, the natural language data may include patient-specific clinical information in an unstructured format. For example, the natural language data may include an unstructured report including the patient-specific clinical information. The patient-specific clinical information may include, for example, information about demographic parameters, in particular gender, weight, height and / or age, physiological parameters, previous investigation results, previous treatments, clinical history and / or medical history of the patient. The patient-specific clinical information may include, for example, an indication and / or reason for a referral request and / or a medical imaging examination.

[0019] The large language model can be configured to recognize and / or understand natural language, and in particular individual items, such as words, in an input containing natural language, and convert these items into a text output. The large language model can be trained and / or machine learned. For example, the large language model can be provided by storing the large language model in a suitable storage device accessible to a computing unit performing the method and / or downloading the large language model by the computing unit.

[0020] Large language models can be used to transform unstructured information into a structured format and vice versa. Typical large language models have specific advantages in summarizing and presenting information in the manner in which the user prompts the large language model. They have an inherent ability to extract information from text. Large language models can be configured for natural language processing (NLP), in particular for extracting structured information based on NLP from natural language data.

[0021] For example, the structured information may include a structured report including patient-specific clinical information in a structured format. The structured format may be, for example, a predefined and / or computer-readable format. The natural language data and / or the structured information may be text-based.

[0022] In a structured format, patient-specific clinical information can be easily extracted and efficiently communicated in a machine-readable format using a predefined automatic communication protocol. Notably, by leveraging a large language model that is capable of understanding language rather than just parsing, this approach avoids various major problems that arise from using only parsing-based methods, such as the mentioned inversion problem. In particular, the structured information can be easily parsed for downstream applications, including integrating additional classical algorithms, or prompting the user to provide missing information when needed. The structured information can include a set of structured information items. For example, these can be represented by codes from commonly used coding systems (ICD-10, LOINC, CPT, etc.) and / or customized task-specific representations. In particular, the structured information can include codes, such as ICD-10 or CPT codes, even though these codes are not present in the natural language data.

[0023] For example, the medical imaging decision support data may be computed by applying a function for computing medical imaging decision support data to the structured information. The function for computing medical imaging decision support data may be trained, in particular machine learning. For example, a large language model may include the function for computing medical imaging decision support data. Alternatively, the function for computing medical imaging decision support data may be rule-based.

[0024] A specific advantage of the proposed approach is its strong comprehensibility. Since the information is presented in a highly structured format, it is easy to spot inconsistent information by humans and thus reduce new sources of errors as they may be caused by ontology-based approaches. Moreover, the reconfigurability of the large language model allows for easy adaptation in case of updated report templates, new tasks and diseases, and ultimately updating of previously created structured reports, since the reconfiguration corresponds to a change in the query rather than a change in the architecture of the large language model.

[0025] Large language models can include transformer networks. A transformer network is a neural network architecture that typically includes an encoder, a decoder, or both an encoder and a decoder. In some cases, the encoder and / or decoder are composed of several corresponding encoding layers and decoding layers, respectively. Within each encoding and decoding layer there is an attention mechanism. An attention mechanism, sometimes called self-attention, relates a data item (such as a word or a pixel) within a series of data items to other data items within the series. For example, a self-attention mechanism allows a model to examine a word within a sentence and determine the relative importance of another word within the sentence to the examined word.

[0026] In particular, the encoder can be configured to transform the input (medical image or text) into a digital representation. The digital representation may include a vector for each input token (e.g., each word). The encoder can be configured to implement an attention mechanism so that each vector of a token is affected by other tokens in the input. In particular, the encoder can be configured to represent the desired output of the parsing transformer network. In particular, the decoder can be configured to transform the input into an output token sequence. In particular, the decoder can be configured to implement a masked self-attention mechanism so that each vector of a token is only affected by other tokens on one side of the sequence. In addition, the decoder can be autoregressive in the sense that intermediate results (such as previously predicted token sequences) are fed back. According to some examples, the output of the encoder is input into the decoder. In addition, the transformer network can include a classification module or unit that is configured to map the output of the encoder or decoder to a set of learning outputs, such as a text summary.

[0027] According to some examples, the training of the transformer model can occur in two stages, namely pre-training and fine-tuning stages. In the pre-training stage, the transformer model can be trained on a large amount of corpus data to learn the underlying semantics of the problem. Such pre-trained transformer models can be used for different languages. For some applications described herein, fine-tuning may include further training the transformer network with medical texts with expert annotated meanings and / or medical ontologies (such as RADLEX and / or SNOMED). In particular, the transformer model according to some examples can learn typical relationships and synonyms of medical expressions.

[0028] Large language models can include, for example, transformer architectures. Transformer networks can use previously inserted text to find logical continuations by querying the information already provided, extracting key information, and assigning values ​​to each. By doing this process layer by layer, large language models predict one token (i.e., word or word part) at a time, where each token is based on the information provided and the world knowledge implicitly represented in the network weights.

[0029] A query may be received, the query comprising natural language data and further query data, wherein the structured information is generated by applying a large language model to the natural language data and the further query data. The query may include, for example, a user query. The user query may be a free text query that a user may enter into a user interface, for example, by typing the user query in an appropriate input populated in the user interface or by voice command.

[0030] Further query data may include template data indicating a structured format. In particular, a large language model may be provided with a dynamically configurable template and queried to transform the natural language data so as to fill in relevant information into the provided template.

[0031] Reference data may be received, the reference data indicating rules and / or examples related to generating structured information and / or related to computing medical imaging decision support data, wherein a large language model is tuned based on the reference data, in particular self-tuned and / or fine-tuned based on the reference data.

[0032] The large language model can be configured to consider reference data as an additional source of information, in particular in the form of general medical guidance, site-specific rules and / or preferences and / or patient history. These examples may relate to different patients and / or examinations. In particular, the reference data may include corresponding preferred medical imaging decisions for different sets of exemplary patient-specific clinical information. In particular, further query data may include reference data. In particular, reference data may be automatically injected as part of the query. Based on the reference data, the large language model can identify patterns and / or reasons why certain recommendations for new patients and / or examinations are desirable, thereby allowing, for example, learning on the fly.

[0033] Explanation data may be computed by a large language model, the explanation data indicating an explanation and / or reason for generating the structured information and / or computing the medical imaging decision support data, wherein the explanation data is provided.

[0034] Large language models may include functions for explanation (particularly interactive explanation) and / or inference (particularly interactive inference), particularly about structured information and / or about medical imaging decision support data. As a key capability, typical large language models are inherently able to explain their decisions about the context in which they are applied. The explanation data may further indicate explanations and / or reasons for tuning the large language model based on reference data. The explanation data may be automatically recorded and / or directly accessible to the user, for example as a hover overlay, decision log and / or information button and / or through interactive queries, which allows cross-checking of automated reasoning, for example in the form of a natural language dialogue.

[0035] For example, the interpretation data may include references to content from natural language data, particularly from unstructured reports, as a verifiable source of generated structured information and / or computed medical imaging decision support data. For each reference to content from natural language data, the interpretation data may include reference-specific related information and / or reference-specific contextual information. The contextual information may include, for example, references to guidance, rules, preferences, patient history, and / or examples (if any), particularly together with an overview of the relevance of each reference to a given situation.

[0036] Examination request data may be received, the examination request data indicating a type of medical imaging examination, the medical imaging decision support data indicating the appropriateness of the type of medical imaging examination in view of clinical information specific to the patient.

[0037] When a referring physician orders a scan, it is necessary to check whether the requested scan type is appropriate for the given indication. Given a text-based referrer request and a description of an appropriate scan for the indication, a large language model can perform this check with only minimal interaction, which is particularly relevant if the user disagrees with the assessment. Furthermore, the large language model's inference about this decision can be queried in order to quickly determine if there was a human oversight or if this was likely an erroneous conclusion by the large language model, e.g. due to missing information.

[0038] Based on the medical imaging decision support data, medical imaging examination workflow steps, options and / or parameters may be suggested, selected and / or triggered, in particular automatically.

[0039] The medical imaging decision support data may indicate values ​​and / or value changes of scan parameters of a scan protocol used for performing a medical imaging examination on a patient by a medical imaging device. The method for providing medical imaging decision support data may further comprise: - setting the scan parameters and / or performing a medical imaging examination based on the scan parameters.

[0040] The medical imaging decision support data may indicate values ​​and / or value changes of reconstruction parameters of a reconstruction algorithm used to reconstruct a medical image based on the medical imaging data. The method for providing medical imaging decision support data may further comprise: - setting reconstruction parameters and / or reconstructing a medical image based on the medical imaging data.

[0041] The medical imaging decision support data may indicate a value and / or a value change of an image processing parameter of an image processing algorithm used to process the medical image. The method of providing medical imaging decision support data may further comprise: setting the image processing parameter and / or processing the medical image. The image processing algorithm may be configured, for example, for post-processing and / or analysis of the medical image. Post-processing of the medical image may comprise, for example, computing a representation of an anatomical structure based on the medical image.

[0042] For example, structured information can be used to enhance and / or automate scanning workflows and / or post-processing workflows. To this end, information from structured and unstructured reports and referral requests can be automatically structured with respect to a clinical question related to the medical imaging examination to be performed, while including all specific aspects surrounding this question. For example, based on given textual information, large language models can be used to quickly answer relevant questions such as "Can the patient receive contrast agents?", "Is high-resolution reconstruction recommended for a given indication?", "In which organs should metastases be automatically searched?", etc.

[0043] Structured information, particularly about patient demographics, prior investigations, or comorbidities, can be used, for example, to select and / or trigger appropriate workflow steps, for selecting appropriate parameters for scanning, reconstruction, and post-processing, and as prior information for any machine learning-based image processing algorithms run on the resulting image dataset.

[0044] When large language models are trained to take this additional input into account, their accuracy is expected to improve, and results can also be automatically checked to see if they are consistent with available information, for example, to reduce the likelihood of false positive findings.

[0045] The method can be implemented in the form of a technical assistant (Tech Co-pilot), which provides support in the daily work of technicians. The method allows the relevant protocol refinement steps, i.e., the reasons for the examination, to be automatically determined given previous reports and clinical indications. These data can be obtained by utilizing open standards (such as DICOM modality worksheets and FHIR standards). However, using these data, the proposed technical assistant will allow automatic protocol refinement, including scanner options such as image overlay, reconstruction and reformatting, multi-energy, spectral and multi-phase acquisition, kernel selection, dose and exposure control (CarekeV), bolus application, timing and tracking, inhalation or exhalation scanning, (quantum) iterative reconstruction and ADMIRE parameterization, metal and motion artifact reduction, and subsequent CAD applications.

[0046] For patients with a prior written medical history and given screening instructions, this approach can be used to identify which patients are eligible and could benefit most from lung screening. If the clinician makes a decision to perform screening, this information will appear in the referral text as the “reason for the examination.” It can therefore be interpreted as such by a large language model.

[0047] Based on this information, an appropriate scanning protocol can be automatically determined based on a method for providing medical imaging decision support data, i.e., screening using a specific ultra-low dose protocol in order to avoid excessive radiation exposure of potentially healthy patients, and automatically triggering an appropriate lung CAD algorithm that detects potential lung nodules in the acquired images. In the case of scanners that allow the acquisition of spectral information (such as photon counting scanners), a quantitative assessment of the composition of any detected lesions can also be automatically triggered for further characterization. Finally, a large language model is used to summarize the results and findings, and / or elaborate on them in a structured report.

[0048] Cancer patients undergo regular staging and follow-up scans to determine the progression of the disease and the effectiveness of treatment. Therefore, several previous diagnostic reports are usually available, detailing the primary cancer type, the identified target lesions, and ultimately all known metastatic loci. This information can be forwarded to a large language model. The method for providing medical imaging decision support data can further include at least one of the following methods

[0049] - summarizes the disease progression up to the current time point, making it easy to use as a reference for current examinations,

[0050] - Identification of the most appropriate baseline scan and the relevant target lesions therein by comparing it with the current exam using lesion detection and segmentation algorithms (which themselves may have information about known lesion locations to increase confidence in their results), and automatically supporting the calculation of quantitative measures (such as RECIST),

[0051] - triggering the generation of appropriate visualizations, e.g., in the case of a pancreatic cancer patient, a high-resolution stack in a standard orientation, including an iodine overlay and an unwrapped image of the organ and its surrounding vessels,

[0052] - Identify organs that may be affected by metastatic lesions and automatically trigger appropriate lesion detection algorithms,

[0053] - Summarize current findings in structured report data and compare them to previous reports, and / or

[0054] - Summarize clinical guidance for staging of corresponding types of cancer and answer specific questions that should be asked about how staging is performed.

[0055] The invention further relates to a data processing system for providing medical imaging decision support data, the data processing system comprising an interface unit and a computing unit,

[0056] - wherein the interface unit is configured to receive natural language data, the natural language data comprising clinical information specific to the patient,

[0057] - wherein the computing unit is configured to generate structured information by applying a large language model to natural language data, the structured information comprising patient-specific clinical information in a structured format,

[0058] - wherein the computing unit is further configured to compute medical imaging decision support data based on the structured information, and

[0059] - wherein the interface unit is further configured to provide medical imaging decision support data.

[0060] The data processing system may be configured to execute the method according to one of the aspects of the invention.

[0061] The data processing system may include, for example, at least one of a cloud computing system, a distributed computing system, a computer network, a computer, a tablet computer, a smart phone, etc. The data processing system may include hardware and / or software. The hardware may be, for example, a processor system, a memory system, and a combination thereof. The hardware may be configured and / or operated by software. The calculations for performing the actions of the method may be performed in a processor.

[0062] The computing unit can be implemented as a data processing system or a part of a data processing system. For example, such a data processing system may include a cloud computing system, a computer network, a computer, a tablet computer, a smart phone, etc. The computing unit may include hardware and / or software. The hardware may include, for example, one or more processors, one or more memories, and combinations thereof. One or more memories may store instructions for performing the method steps according to the present invention. The hardware may be configured and / or operated by software. Typically, all units, subunits, or modules may at least temporarily exchange data with each other, for example, via a network connection or a corresponding interface. Therefore, the individual units may be separated from each other.

[0063] The interface unit may comprise an interface for data exchange with a local server or a central network server via an internet connection for receiving the medical image dataset. The interface unit may further be adapted to interface with one or more users of the system, for example by displaying the processing results of the computing unit to the user (e.g. in a graphical user interface), or by allowing the user to adjust parameters for data processing or visualization. In other words, the interface unit may comprise a user interface.

[0064] Data, in particular natural language data, queries, further query data, reference data and examination request data may be received, in particular via the interface unit, for example by receiving a signal carrying the data and / or by reading the data from a computer memory and / or by manual user input, for example via a graphical user interface. Data, in particular medical imaging decision support data and / or interpretation data may be provided, in particular via the interface unit, for example by transmitting a signal carrying the data and / or by writing the data into a computer memory and / or by displaying the data on a display.

[0065] Any of the algorithms, functions and / or models mentioned in this article can be based on one or more of the following architectures: deep convolutional neural networks, deep belief networks, random forests, deep residual learning, deep reinforcement learning, recurrent neural networks, Siamese networks, generative adversarial networks, or autoencoders.

[0066] The present invention further relates to a medical imaging device, comprising a data processing system according to one of the aspects of the present invention. The medical imaging device can be configured to perform a medical imaging examination on a patient based on a scanning protocol for the medical imaging examination. The medical imaging device can be, for example, a computer tomography device, a magnetic resonance imaging device, an ultrasound imaging device, a PET imaging device, a SPECT imaging device, an X-ray imaging device, a cone beam CT device and / or a combination thereof.

[0067] The invention further relates to a computer program product comprising instructions which, when executed by a computer, cause the computer to perform a method according to one of the aspects of the invention. The invention further relates to a computer readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform a method according to one of the aspects of the invention. An advantage of implementing the invention by means of a computer program product and / or a computer readable storage medium is that existing computers can be easily adapted by software updates to work as proposed by the invention.

[0068] The computer program product can be, for example, a computer program or include another element in addition to the computer program. The other element can be hardware, such as a memory device storing the computer program, a hardware key for using the computer program, etc., and / or software, such as a document or software key for using the computer program. The computer program product can further include development materials, a runtime system and / or a database or library. The computer program product can be distributed in several computer instances. The computer readable storage medium can be embodied as a non-permanent main memory (e.g., random access memory) or a permanent mass storage device (e.g., a hard disk, a USB disk, an SD card, a solid state disk).

[0069] Leveraging large language models that can understand language, rather than just parse it, offers significant advantages over symbolic ontology approaches, which typically have difficulty handling negations, inversions, abbreviations, typos, and additional information that is either “hard-coded” in parsing and is therefore very limited, or cannot handle that information at all. In contrast, language understanding, information extraction, and content summarization are key capabilities of typical large language models, allowing these problems to be avoided. Specifically, while negation and inversion are an integral part of natural language and are therefore richly expressed, world knowledge, as well as explicitly given query information, can be used to account for abbreviations and typos. Additional information can be explicitly queried and summarized, such as in the form of easily digestible bullet points, which themselves may reference the original report.

[0070] The use of large language models may therefore revolutionize the way structured information is created from unstructured natural language data, an important step in the digital transformation of clinics, thereby improving throughput, accelerating treatment, serving as a double check to avoid treatment errors, and ultimately reducing the workload of the parties involved. With a special focus on automation of scanning and post-processing workflows, such as is addressed by technical assistants, the advantages are similarly multifaceted, with the main aspects including time savings through smart scanners, automatic pre-selection of the correct workflows and steps, and triggering only relevant and specifically tailored algorithms in order to create relevant (AI-based) image analysis results, standardization and improved consistency of examinations. In summary, this allows for highly reproducible, guidance-based workflow recommendations and gives rise to a strong potential for reducing errors by serving as an automated "second pair of eyes" that automatically parses available text-based sources for any relevant information.

[0071] In the context of the present invention, the expression "based on" can be particularly understood as "especially using". In particular, the wording according to which the first feature is calculated (or generated, determined, etc.) based on the second feature does not exclude the possibility of calculating (or generating, determining, etc.) the first feature based on the third feature.

[0072] Note that the described method and the described system are only preferred exemplary embodiments of the invention and that a person skilled in the art may vary the invention without departing from the scope of the invention as defined in the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The invention will be explained below using exemplary embodiments with reference to the accompanying drawings. The illustrations in the figures are schematic and highly simplified and are not necessarily drawn to scale.

[0074] Figure 1 A flow chart of a computer-implemented method for providing medical imaging decision support data is shown.

[0075] Figure 2 A data processing system is shown.

[0076] Figure 3 A diagram showing the flow of data through an information system.

[0077] Figure 4 A medical imaging workflow environment is shown.

[0078] Figure 5 A table showing a structured report. DETAILED DESCRIPTION

[0079] Figure 1 A flow chart illustrating a computer-implemented method for providing medical imaging decision support data, the method comprising:

[0080] - receiving S1 natural language data U, the natural language data U comprising clinical information specific to the patient,

[0081] - generating S2 structured information N comprising patient-specific clinical information in a structured format by applying a large language model to the natural language data U,

[0082] - Compute S3 medical imaging decision support data based on structured information N, and

[0083] - Provide S4 medical imaging decision support data.

[0084] Figure 2 A data processing system P for providing medical imaging decision support data is shown, the data processing system P comprising an interface unit PI and a computing unit PC,

[0085] - wherein the interface unit PI is configured to receive S1 natural language data U comprising patient-specific clinical information,

[0086] - wherein the computing unit PC is configured to generate S2 structured information N comprising patient-specific clinical information in a structured format by applying a large language model to the natural language data U,

[0087] - wherein the computing unit PC is further configured to compute S3 medical imaging decision support data based on the structured information N, and

[0088] - wherein the interface unit PI is further configured to provide S4 medical imaging decision support data.

[0089] Figure 3 A data flow diagram through an information system M is shown in which a large language model is implemented. The information system M may be, for example, a computer information system M. Thus, the information system M is able to gracefully handle nearly all of the idiosyncrasies of natural language used in daily clinical routines.

[0090] The information system M includes a component 1 for transforming unstructured information into structured information, a component 2 for automating and enhancing medical imaging workflows, a component 3 for learning on the fly, and a component 4 for interpretable processing. Each of these components uses a large language model (LLM)-based process to infer results. The large language model of the information system M includes a function L1 for generating S2 structured information N based on natural language data U and a function L2 for calculating S3 medical imaging decision support data based on the structured information N. The large language model of the information system M further includes a function 4R for interpretation and / or reasoning. Component 4 includes a user interface 4Q for interactive query.

[0091] Component 2 includes a module 2A for suggesting, selecting and / or triggering medical imaging examination workflow steps, options and / or parameters related to acquisition (particularly related to scanning protocols) and / or related to reconstruction (particularly related to reconstruction algorithms). Component 2 further includes a module 2B for suggesting, selecting and / or triggering medical imaging examination workflow steps, options and / or parameters related to image processing (particularly related to image processing algorithms). For example, module 2B may include an interface for injecting structured information as a priori information input into the image processing algorithm.

[0092] Reference data is received, the reference data indicating rules 3G and / or examples 3E related to generating S2 structured information N and / or to computing S3 medical imaging decision support data, wherein a large language model is tuned based on the reference data. Component 3 is configured to tune the large language model based on the reference data, the reference data indicating rules 3G, in particular in the form of general medical guidance, site-specific rules and / or preferences, and / or indicating examples 3E. Component 3 can further be configured to tune the large language model based on patient medical history data 5. Thus, as described above, the information system M is able to learn from contextual information and previous examples on the fly and reason with them.

[0093] Figure 4 A medical imaging workflow environment 8 is shown. An information system M may be used as a technical assistant 80 for a technical expert 83 in the medical imaging workflow environment 8. Given a referral from a physician 81, which includes an imaging request to solve a clinical problem (indication), a radiologist 82 determines an appropriate underlying protocol. As a communication platform 91 for the physician 81 and the radiologist 82, an electronic medical record (EMR) and / or a radiology information system (RIS) may be used.

[0094] The appropriate base protocol is recorded, for example, in a DICOM modality worksheet 92 and forwarded to the medical imaging device 93 in the form of a request procedure 74 with only technical terms such as "CT chest w / o contrast". In addition, background information about the protocol selection 71, such as "Long nodule follow-up CT", can be forwarded directly from the physician 81 and / or the radiologist 82 to the technical expert 83. The reason for the examination 73, such as "Lung nodule follow-up" is provided to the technical assistant 80. Receiving additional information such as previous diagnostic reports 72 through a communication interface such as FHIR, the technical assistant 80 infers and suggests relevant protocol refinements 75 for the technical expert 83, thereby significantly reducing the workload of adjusting the scanning, reconstruction and / or post-processing protocols. The medical imaging device 93 includes a data processing system P.

[0095] Figure 5Table T of the structured report is shown. For example, the transformer-based ChatGPT large language model is able to generate a structured report based on the following query, which includes unstructured natural language data and a report template.

[0096] “Extract findings from the following radiology reports and summarize the findings in the following template:

[0097] Name:

[0098] age:

[0099] gender:

[0100] Medical history:

[0101] <Key Points>

[0102] Findings:

[0103] <Table with three columns - Abnormality, Location of abnormality, Corresponding ICD-10 code>

[0104] Summarize:

[0105] <Key Points>

[0106] Clinical History: Jane Doe, a 68-year-old white male, was recently diagnosed with adenocarcinoma by sputum cytology. Abnormal chest radiograph showed right middle lobe infiltration and collapse. The patient required contrast staging CT of the chest. Physical examination revealed right supraclavicular and anterior subcervical gland disease.

[0107] Technique: Multiple cross-sectional images are obtained in 10 mm slices through the chest. Contrast is injected intravenously.

[0108] Findings: A large lymph node measuring 3 x 4 cm is seen in the right supraclavicular region. A large right paratracheal lymph node is best appreciated on image #16 and measures 3 x 2 cm. The subcortical lymph nodes are also enlarged. It measures 6 x 2 cm. Multiple pulmonary nodules are seen along the posterior margin of the visceral and parietal pleura. A pleural mass is seen within the right hemithoracic groove and the right crus of the diaphragm. Soft tissue density adjacent to the underside of the right lobe of the liver is also best appreciated on image #36, which most likely also represents metastatic deposits. The liver parenchyma is normal with no evidence of any significant mass. The right kidney demonstrates a solitary cyst in the right mid-renal pole.

[0109] impression:

[0110] 1. The upper right side shows more than 20 pulmonary nodules, including nodules in the parietal pleura and various visceral pleura, with adjacent consolidations that most likely represent lung tumors.

[0111] 2. Extensive mediastinal adenopathy as described above.

[0112] 3. No lesions were found in the left lung at this time.

[0113] 4. Supraclavicular gland disease.

[0114] The following is the response to the above query.

[0115] "Name: Anonymous

[0116] Age: 68

[0117] Gender: Male

[0118] Medical history:

[0119] - Recently diagnosed adenocarcinoma by sputum cytology

[0120] Findings:

[0121] Table T Figure 5 shown.

[0122] Summarize:

[0123] -Presence of more than 20 pulmonary nodules on the right side, including nodules within the parietal and visceral pleura, with adjacent consolidations that most likely represent pulmonary tumors.

[0124] -Extensive mediastinal adenopathy with lymphadenopathy in the right supraclavicular, right paratracheal, and subcarinal regions.

[0125] -No lesions were observed in the left lung at this time.

[0126] - Supraclavicular disease. "

Claims

1. A computer-implemented method for providing medical imaging decision support data, the method comprising: - receiving (S1) natural language data (U), said natural language data (U) comprising clinical information specific to a patient, - generating (S2) structured information (N) by applying a large language model to said natural language data (U), said structured information (N) comprising said patient-specific clinical information in a structured format, - calculating (S3) said medical imaging decision support data based on said structured information (N), and - providing (S4) said medical imaging decision support data.

2. The method according to claim 1, - wherein the large language model comprises a transformer network.

3. The method according to claim 1 or 2, - wherein a query is received, said query comprising said natural language data (U) and further query data, - wherein said structured information (N) is generated by applying said large language model to said natural language data (U) and said further query data.

4. The method according to claim 3, - said further query data comprises template data, said template data being indicative of said structured format.

5. The method according to one of claims 1 to 4, - wherein reference data is received, the reference data indicating rules (3G) and / or examples (3E) related to generating (S2) the structured information (N) and / or related to calculating (S3) the medical imaging decision support data, - wherein said large language model is tuned based on said reference data.

6. The method according to one of claims 1 to 5, - wherein explanation data is calculated by the large language model, the explanation data indicating explanations and / or reasons for generating (S2) the structured information (N) and / or calculating (S3) the medical imaging decision support data, - wherein said interpretation data is provided.

7. The method according to one of claims 1 to 6, - wherein examination request data is received, the examination request data indicating a type of medical imaging examination, - the medical imaging decision support data indicates the appropriateness of the type of the medical imaging examination taking into account the patient-specific clinical information.

8. The method according to one of claims 1 to 7, - wherein medical imaging examination workflow steps, options and / or parameters are suggested, selected and / or triggered based on said medical imaging decision support data.

9. The method according to one of claims 1 to 8, - the medical imaging decision support data are indicative of values ​​and / or value changes of scan parameters of a scan protocol for a medical imaging examination of a patient by a medical imaging device.

10. The method according to one of claims 1 to 9, - the medical imaging decision support data are indicative of values ​​and / or value changes of reconstruction parameters of a reconstruction algorithm used for reconstructing a medical image based on the medical imaging data.

11. The method according to one of claims 1 to 10, - the medical imaging decision support data are indicative of values ​​and / or value changes of image processing parameters of an image processing algorithm for processing the medical image.

12. A data processing system (P) for providing medical imaging decision support data, the data processing system (P) comprising an interface unit (PI) and a computing unit (PC), - wherein the interface unit (PI) is configured to receive (S1) natural language data (U), the natural language data (U) comprising clinical information specific to the patient, - wherein said computing unit (PC) is configured to generate (S2) structured information (N) by applying a large language model to said natural language data (U), said structured information (N) comprising said patient-specific clinical information in a structured format, - wherein the computing unit (PC) is further configured to compute (S3) the medical imaging decision support data based on the structured information (N), and - wherein said interface unit (PI) is further configured to provide (S4) said medical imaging decision support data.

13. A medical imaging device (93) comprising a data processing system (P) according to claim 12.

14. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to one of claims 1 to 11.

15. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to one of claims 1 to 11.

Images