Method for providing medical imaging workflow support data

Abstract

Embodiments of the present disclosure relate to a method for providing medical imaging workflow support data. In one aspect, the invention relates to a computer-implemented method for providing medical imaging workflow support data, the method comprising: receiving (S1) natural language data (U), the natural language data (U) comprising clinical information, generating (S2) structured information (N) by applying a large language model (L) to the natural language data (U), the structured information (N) comprising the clinical information in a structured format according to at least one input node (31, 32) of an inference graph (G), computing (S3) medical imaging workflow support data by applying at least one rule of the inference graph (G) to the structured information (N), and providing (S4) the medical imaging workflow support data.

Description

Technical Field

[0001] In one aspect, the present invention relates to a computer-implemented method for providing support data for medical imaging workflows. In other aspects, the present invention relates to data processing systems, medical imaging systems, computer program products, and computer-readable storage media. Background Technology

[0002] In clinical IT systems, patient information is often stored in segments, widely distributed across various IT systems, and typically presented only in the form of unstructured reports. As recent research has shown, this can lead to significant loss of critical information, directly impacting timely and accurate patient treatment, and thus posing a significant obstacle to effective workflow automation and accurate patient care.

[0003] At the same time, the complexity of scanning is increasing, primarily due to technological advancements, the shift from a traditional one-size-fits-all approach to personalized medicine, and, significantly, increased legal requirements. This situation is exacerbated by a growing staff shortage in healthcare facilities, leading to a widening gap between workload and available personnel.

[0004] In these situations, missing information often translates into delays or disruptions in care for most patients, which can significantly increase risk and impair treatment outcomes. While standardized vocabularies and protocols (such as ICD-10, FHIR, etc.) exist, their practical application in various clinical IT environments is far from ubiquitous, particularly due to data warehouses, vendor-specific solutions, and the inherent ambiguity of natural language in medical documentation if non-standardized formats are used.

[0005] Therefore, critical details are often lost during the conversion from unstructured reports to structured formats, which, if any, can impact downstream processes and increase the risk of treatment delays or interruptions. Furthermore, clinicians must manually decipher complex reports, map findings, and adjust protocols, diverting valuable time and effort from patient care. Moreover, with the increasing complexity of image acquisition technologies, the process of mapping clinical problems into acquisition protocols and their manual refinement to achieve patient-tailored imaging adds workload and cognitive burden, leading to a decrease in patient focus.

[0006] Due to the imperfections in available information, decisions are often intuitive and / or based on the substantial experience of the therapists. This process may lack transparency, interpretability, and / or traceability, making it challenging to understand the underlying principles and update workflows in response to evolving clinical practices and technological advancements.

[0007] In many cases, scanners typically only receive an abstract description of the process to be performed via a DICOM Modal Work List (DMWL), which is often insufficient for advanced workflow automation. For example, based on decision trees and / or manual settings of parameters and steps to be performed, the immediate adaptation of protocols can be limited to user-provided answers to predetermined questions. The process relies entirely on the expertise of the scanner operator and often requires time-consuming additional communication with radiologists and referring physicians when information is missing or unclear.

[0008] Furthermore, the common scanning and post-processing standard operating procedures (SOPs) are still largely modeled as separate, proprietary protocols. This can result in large and complex sets of protocols that are difficult to maintain in practice and require expert knowledge to set up correctly via the corresponding configuration UI. This is typically done by the chief technical expert based on the requirements of the radiologist. If they need support in this process, including the best options for a specific scanner and how to properly configure both, they are often limited to consulting manuals or asking application experts, who may not be familiar with every relevant aspect due to the abundance of options.

[0009] Creating and maintaining scanning protocols at the scanner site is an expensive and time-consuming process. This is primarily due to the numerous specialized protocols required to meet clinical needs, and the ever-increasing capabilities and configuration parameters of scanners. The different scanners with varying capabilities and software versions further complicate the representation of scanning protocols. Furthermore, field scanning protocols are typically adapted and therefore specific to each clinic, making it impossible to provide all relevant configurations at the factory. Therefore, each site maintains its own protocols, for example, in a protocol book, and ensures that they are accurately reflected on each scanner.

[0010] For example, CT scan protocols can be exported and imported from the scanner as XML files using a proprietary internal schema. However, this schema is not explicitly defined and can only be understood through its implementation. Due to the large and complex nature of the protocol files, modifying them requires extensive knowledge and understanding. Furthermore, each new version of the scanner software may require separate specifications and implementations to transmit earlier protocols or protocol versions. Complementary aspects, such as the clinical decision tree associated with the protocol, can be specified in separate files with similar formats.

[0011] Given an abstract high-level protocol specification for a specific patient, currently, scanner-specific protocols must be selected and configured individually for each scanner because acquisition and post-processing parameters differ with each scanner and version. Therefore, managing and maintaining scanning protocols is often challenging.

[0012] The typical process of creating and maintaining scanning protocols is manual and heavily reliant on the expertise of the protocol administrator. This labor-intensive task requires a thorough understanding of the protocol to ensure its accurate implementation and ongoing maintenance on each scanner, both in the central documentation (protocol manual) and directly through the user interface used to check the protocol's specifications. This manual process is costly and requires extensive knowledge of the different scanners, software versions, and protocols used at each site. This becomes even more critical for photon counting computed tomography (PCT), which offers many additional capabilities such as always-on spectral acquisition and ultra-high-resolution reconstruction.

[0013] For example, prior art related to this technical field is disclosed in EP4462445A1.

[0014] A potential technical problem of the present invention is to provide alternatives and / or improvements to known solutions for supporting medical imaging workflows. This problem is addressed by the subject matter of the independent claims. Dependent claims relate to other aspects of the invention. Nouns and pronouns relating to persons in this patent application generally do not specify a particular gender. Summary of the Invention

[0015] In one aspect, the present invention relates to a computer-implemented method for providing medical imaging workflow support data, the method comprising: - Receives natural language data, including clinical information. - By applying a large language model to natural language data, structured information is generated, including clinical information in a structured format according to at least one input node of the inference graph. - Compute medical imaging workflow support data by applying at least one rule of the inference graph to structured information, and - Provides data to support medical imaging workflows. Attached Figure Description

[0016] Features relating to the present invention or its technical background will now be described with reference to the accompanying drawings. The illustrations in the drawings are schematic and highly simplified, and are not necessarily drawn to scale.

[0017] Figure 1 A flowchart is shown of a computer-implemented method for providing support data for medical imaging workflows.

[0018] Figure 2 The data processing system is shown.

[0019] Figure 3 The representation of the reasoning graph is shown.

[0020] Figure 4The user interface for modifying the inference graph is shown.

[0021] Figure 5 A flowchart is shown of a computer-implemented method for protocol modification. Detailed Implementation

[0022] Using reasoning graphs, rules can be implemented in a meaning-based, easily understood format, making the entire process traceable, editable, and interpretable, even for complex rule sets, such as those from clinical standard operating procedure (SOP) guidelines. Reasoning graphs can allow radiologists and technicians to modify optimized workflows by leveraging discovery-defined natural language interactions based on natural language (NL). Reasoning graphs can also enable rule-driven derivation based on protocols of structured information items.

[0023] Therefore, configurable rule set components can be provided, specifically allowing clinicians and technicians to tailor workflows to their unique requirements, adhering to SOPs defined by individual clinics, even when using only natural language. Furthermore, the proposed solution allows for the elimination of dependence on steadily increasing context size to ensure constant learning capacity and introduces a novel data-driven learning mechanism that allows for the automatic derivation of SOPs using an available set of paired images and indicator / report data provided by clinical RIS.

[0024] Clinical information can be, for example, patient-specific clinical information. In particular, natural language data can include clinical information in unstructured formats. For example, natural language data can include unstructured reports containing clinical information. Clinical information can include, for example, information about demographic parameters, particularly sex, weight, height and / or age, physiological parameters, previous findings, previous treatments, clinical history and / or patient history. Clinical information can also include, for example, instructions and / or reasons for referral requests and / or medical imaging examinations.

[0025] Large language models can be configured to recognize and / or understand natural language, and in particular, to recognize and / or understand individual items such as words in an input containing natural language, and to send these items to a text output. Large language models can be trained and / or machine learning-enabled. For example, a large language model can be provided by keeping it in suitable memory accessible to the computing unit performing the method and / or by downloading it by the computing unit.

[0026] Large language models can be used to transform unstructured information into structured formats and / or vice versa. Typical large language models have a particular strength in summarizing and presenting information in a way that is user-initiated. They possess an inherent ability to extract information from text. Large language models can be configured for Natural Language Processing (NLP), particularly for extracting structured information from natural language data using NLP principles.

[0027] For example, structured information may include structured reports, which contain clinical information in a structured format. The structured format may be, for example, a predefined and / or computer-readable format. Natural language data and / or structured information may be text-based.

[0028] In structured formats, clinical information can be easily extracted and efficiently communicated in a machine-readable format using predetermined automated communication protocols. Notably, by using a large language model capable of understanding language rather than merely parsing, this approach avoids various major problems arising from parsing-only methods, such as the aforementioned inversion problem. Specifically, structured information can be readily parsed for downstream applications, including integration with additional classical algorithms, or, if needed, prompting the user for any missing information. Structured information can include sets of structured information items. For example, these can be represented by codes from common coding systems (ICD-10, LOINC, CPT, etc.) and / or custom, task-specific representations. In particular, structured information can include codes, such as ICD-10 or CPT codes, even though these codes are not present in the natural language data.

[0029] Medical imaging workflow support data can be, for example, medical imaging decision support data. For instance, medical imaging workflow support data can be computed by applying a function used to compute the data to structured information. The function used to compute the data can be at least one rule of a reasoning graph. The function used to compute the data can be trained, particularly through machine learning. For example, a large language model can include a function for computed to the data. Alternatively, the function used to compute the data can be rule-based.

[0030] A key advantage of the proposed method stems from its strong understandability. Because information is presented in a highly structured format, inconsistencies are easily detected by humans, reducing new sources of error that might arise from ontology-based approaches. Furthermore, the reconfigurability of the large language model allows for simple adaptation to updated report templates, new tasks, and diseases, as well as ultimately updating previously created structured reports, since the reconfiguration corresponds to changes in the query, rather than changes to the architecture of the large language model.

[0031] Large language models can include transformer networks. A transformer network is a neural network architecture that typically includes an encoder, a decoder, or both. In some instances, the encoder and / or decoder each consist of several corresponding encoding and decoding layers. Each encoding and decoding layer contains an attention mechanism. Attention mechanisms, sometimes called self-attention, associate data items (such as words or pixels) within a set of data items with other data items within that set. For example, a self-attention mechanism allows the model to examine a word in a sentence and determine the relative importance of another word in the sentence to the examined word.

[0032] Specifically, the encoder can be configured to convert the input (medical image or text) into a numerical representation. The numerical representation may include a vector for each input term (e.g., each word). The encoder can be configured to implement an attention mechanism such that each vector of a term is influenced by other terms in the input. Specifically, the encoder can be configured to represent the desired output of the parsing transformer network. Specifically, the decoder can be configured to transform the input into an output sequence of terms. Specifically, the decoder can be configured to implement a masked self-attention mechanism such that each vector of a term is influenced only by other terms on one side of the sequence. Furthermore, the decoder can be autoregressive, meaning that intermediate results (such as previously predicted sequences of terms) are fed back. According to some examples, the output of the encoder is fed into the decoder. Additionally, the transformer network may include a classification module or unit configured to map the output of the encoder or decoder to a set of learned outputs, such as text summaries.

[0033] Training of the transformer model, based on some examples, can occur in two phases: a pre-training phase and a fine-tuning phase. In the pre-training phase, the transformer model can be trained on a large corpus to learn the underlying semantics of the problem. This pre-trained transformer model can be used with different languages. For some applications described in this paper, fine-tuning may include further training the transformer network with medical text and / or medical ontologies such as RADLEX and / or SNOMED, ​​which have expert-annotated meanings. In particular, utilizing the latter, the transformer model, based on some examples, can learn typical relationships and synonyms of medical expressions.

[0034] Large language models can include, for example, transformer architectures. A transformer network can find logical continuity using previously inserted text by querying provided information, extracting key information, and assigning values ​​to each piece of information. By building this process layer by layer, the large language model predicts one lexical unit (i.e., a word or part of a word) at a time, each lexical unit based on the provided information and world knowledge implicitly represented in the network weights.

[0035] Query can be received, comprising natural language data and further query data, wherein structured information is generated by applying a large language model to the natural language data and further query data. Query can include, for example, user queries. User queries can be free-text queries that a user can input into a user interface, for example, by typing the user query into the appropriate input field in the user interface or via voice command. Further query data can include template data indicating a structured format. Specifically, dynamically configurable templates can be provided to the large language model, and queries can be used to transform the natural language data, thereby populating the provided templates with relevant information.

[0036] Reference data can be received, which indicates rules and / or examples related to the generation of structured information and / or to the data supporting computational medical imaging workflows, wherein large language models are adjusted based on the reference data, particularly self-adjusting and / or fine-tuning based on the reference data.

[0037] Large language models can be configured to treat reference data as an additional source of information, particularly in the form of general medical guidelines, site-specific rules and / or preferences and / or patient history. Examples can involve different patients and / or examinations. In particular, reference data can include corresponding preferred medical imaging decisions for different exemplary sets of clinical information. In particular, further query data can include reference data. In particular, reference data can be automatically injected as part of the query. Based on reference data, large language models can identify certain recommended patterns and / or reasons why new patients and / or desired examinations are needed, thereby allowing, for example, on-the-spot learning.

[0038] Structured information may include Structured Information Items (SIIs). A Structured Information Item may include clinical information in a structured format according to at least one input node of a reasoning graph. The Structured Information Item may, for example, be related to clinical findings and / or operational coding systems (ICD-10, LOINC, CPT). The Structured Information Item may also be modeled using description-based natural language variable constraints. In exemplary constraints, the Structured Information Item may be specified solely by natural language. Exemplary constraints would be, for example, “mention of abnormal chest XR,” “significant motion-induced artifacts,” or “tall people (>180cm).” The constraints of the Structured Information Item may also include possible target values ​​(e.g., true / false), document selectors, and potential FHIR links.

[0039] In particular, structured information items may include one or more of the following attributes.

[0040] Name: The name of the structured information item. This attribute can be used specifically in the reasoning graph described below.

[0041] Description: Natural language-based descriptions of structured information items.

[0042] Allowed values / value range: Limits on allowed values, which can be in the form of a list (e.g., [true, false]), a range (e.g., [1…10]), or a data type (e.g., bool / int / string / …).

[0043] Document selector: Used to select the document type from which this result can be extracted. This is especially important if structured information items should only be considered relevant if they originate from a specific source, such as previously confirmed inspection reports, laboratory results, etc.

[0044] Equivalent code: The equivalent code in classic coding schemes, such as FHIR, is used to link structured information to an existing information system (if any).

[0045] Structured information items can be interchanged with other information sources, such as FHIR information, or clinically approved findings in any format. Therefore, equivalent codes serve as a mechanism to link description-based items to specific (e.g., FHIR-based) qualifications, allowing approved findings to supersede NL-based findings, further enhancing reliability.

[0046] According to another aspect, the medical imaging workflow supports data corresponding to nodes in the inference workflow. According to another aspect, the inference graph is based on clinical procedure guideline data and / or sets of paired patient and workflow data. According to another aspect, reference data is received, wherein the inference graph is generated by applying a retrieval-enhanced generative model to the reference data.

[0047] Inference graphs can correspond to a set of functions and related parameters and / or to a graphical representation of the relationships (edges) between different pieces of information (nodes). They are used to model how conclusions (inferences) are drawn from given data based on logical connections and / or probabilistic reasoning. Inference graphs help to understand and visualize the flow of information and the logical steps taken to reach conclusions, which is particularly useful in complex NLP tasks such as question answering, information extraction, and reasoning.

[0048] The inference graph can be, for example, a trained inference graph model. The inference graph can be, for example, a dynamic inference graph, particularly a dynamically updated inference graph. Clinical procedure guideline data can be, for example, clinical standard operating procedure (SOP) guideline data. Reference data can include clinical procedure guideline data and / or sets of paired patient and workflow data. The retrieval-enhanced generative (RAG) model can first retrieve relevant information from the reference data and / or from the sets of paired patient and workflow data. The retrieval-enhanced generative model can then use a generative model to generate an inference graph based on the retrieved information. Therefore, the retrieval-enhanced generative model combines the strengths of retrieval-based and generation-based models.

[0049] Structured information items can be linked in the inference graph, particularly for deriving patient-customized data collection workflows. The inference graph can include rule sets and / or at least two different types of nodes, such as information nodes and workflow aspect nodes. Nodes can correspond to variables.

[0050] Information nodes can be abstract representations of any kind of information. Information nodes include a name, description, and a value during inference. Information nodes can be, for example, input nodes or intermediate nodes. In the inference graph, each structured information item is represented as an input node. The values ​​of input nodes can be populated at the start of inference, while the values ​​of intermediate nodes can be set by rule application. Information nodes may not directly correspond to acquisition settings (e.g., keV, mAs, pitch, etc.).

[0051] Workflow aspect nodes can directly correspond to imaging workflow settings and can represent entities such as scanning, reconstruction, or post-scan algorithms. Notably, workflow aspect nodes can cover multiple sub-settings, such as mAs values ​​during scanning, specific kernels for reconstruction, or parameters for specific algorithm selections. Specifically, workflow aspect nodes in the inference graph can be target nodes in the inference graph. These issues can be addressed by bridging the gap between unstructured clinical narratives and the structured data required for effective workflow automation. This introduces an LLM-driven paradigm shift by enabling radiologists and technicians to define Structured Information Items (SIIs) using only natural language and integrating them into rule-driven systems.

[0052] Natural language understanding can be used for seamless data extraction. Specifically, Large Language Models (LLMs) can be used to autonomously extract relevant clinical information from unstructured reports, translating it into a standardized, machine-readable format represented as Structured Information Items (SIIs) defined solely by natural language. This eliminates the bottleneck of manual mapping and ensures data integrity. Furthermore, rules can be created and managed intelligently. In particular, structured information items expressed in plain language can form the basis for adaptive rules used to create inference graphs and manage workflow processes in computed tomography imaging. This enables clinicians and technicians to modify acquisition workflows interactively via natural language, enhancing flexibility and reducing reliance on static rule sets.

[0053] LLM-driven chatbots can provide real-time support, suggest optimal scanning protocols, explain rule application, and proactively query EHR / EMR systems for missing patient context. This facilitates data-driven decision-making and minimizes the possibility of human error. Leveraging clinical standard operating procedures (SOPs) guidelines for scanning workflow configuration, and in conjunction with retrieval-enhanced generation (RAG), Large Language Models (LLMs) can be used to identify key decision criteria for scanning protocols used in clinics. This is particularly true for factory protocols, but also ensures adaptability to site-specific SOPs.

[0054] On the other hand, the representation of the inference graph and / or a subset thereof is visualized in a user interface, where user input, particularly manual and / or natural language user input, is received through the user interface, and the inference graph is modified based on the user input. A subset of the inference graph can include a subset of subgraphs of the inference graph and / or a subset of nodes and / or rules of the inference graph. Further refinement using graph-based editing is possible based on the generated initial rule set, greatly reducing the initial work required for workflow and protocol configuration. The graph can be modified using intuitive graphical editing methods. Typically, this process can be performed manually due to the option to visualize the graph itself. This provides a highly relevant and intuitive fallback solution to accelerate the graph modification process.

[0055] Clinical IT systems can aggregate constructed scans and their respective protocols, along with patient clinical data. Therefore, data mining methods, specifically LLM-based de-siloing of the data, can be applied to derive inference graphs from paired patient and examination data. This automatically extracts patient details and subsequently analyzes the identified patterns, allowing the derivation of general rules that can be represented using inference graph concepts. Notably, the data mining methods presented above can be similarly applied to analyze continuously changing patterns, as these are typical during the adoption of new technologies. Therefore, the proposed method allows for gradual adaptation to users as they become more familiar with the new technology, and thus further improves clinical workflows over time through continuous learning.

[0056] On the other hand, additional information is queried from medical information systems and / or from users, particularly in the form of AI-based assistive functions, especially chatbots. Specifically, inference graphs are modified based on this additional information, and / or medical imaging workflow support data is calculated based on it. Graphical editing interactions can be automated using AI-based assistive functions.

[0057] In addition to enriching the scan user interface to display and / or adapt derived workflows, suggestions, and their underlying rules and source data, AI-based, particularly LLM-based, assistance features, especially in the form of chatbots, can be used to guide and support the system user at any point in the scanning and post-processing workflow. This is especially important and beneficial given that making ideal choices for each patient during imaging and post-processing has become more complex due to the increased possibilities—particularly since the introduction of photon-counting CT with “always-on” spectral imaging and the possibility of acquiring and reconstructing ultra-high resolution (UHR) images.

[0058] The current scan indications and available structured and unstructured patient histories—that is, clinical context information that can be used as input for the aforementioned rule-based workflow automation, as well as for AI-based assistive functions—can also be accessed. Furthermore, it can access rule sets and ideal workflows derived from them based on clinical context information.

[0059] Technical and clinical knowledge about scanners and their functions can also be provided by the supplier (by default, such as scanner manuals, white papers, etc.) or by the user themselves (optional; the rule set may not directly cover or fully explain custom site-specific technologies and / or SOPs). In this way, accessibility features can be well equipped to reason and answer any questions about the patient, the ideal examination, and how to best achieve it using the scanner at hand (“Smart User Manual”).

[0060] Technological components that can make these documents and related information available to LLM include: direct injection of prompts (system prompts, enriched user prompts), and / or retrieval-enhanced generation systems, where the context window and capabilities of LLM allow for efficient and useful processing of appropriate amounts of text, where only relevant portions of the documents are dynamically searched and injected using, for example, a vector database storing pre-computed semantic embeddings of all relevant source documents. Where applicable, documents can exist in natural (e.g., diagnostic reports) and / or structured languages ​​(e.g., workflows deduced in formats like JSON or XML). Large language models may be well-suited for flexibly handling both.

[0061] In many cases, clinical context information needs to be retrieved from other systems within the hospital, such as EHR / EMR (Electronic Health / Medical Records) or HIS / RIS / LIS (Hospital / Radiology / Laboratory Information Systems). Integration with these systems at the technical (see Connectivity) and / or semantic (see Concept Mapping) levels allows the use of information, including that used herein for descriptive purposes. However, the LLM employed can also facilitate dynamic information retrieval. For example, if a piece of information is requested by a user (or identified as required input for a rule in a given rule set) and its accessibility is not yet known, but it may be available within the connected system, then an appropriate query (e.g., an FHIR query, i.e., a request for information resources as defined in the FHIR standard and established according to the FHIR communication protocol) can be automatically generated by the LLM and then executed automatically in the background to retrieve the corresponding information on demand (if it exists).

[0062] As mentioned above, AI-based assistance can support users in performing checks. This is not limited to answering questions and providing suggestions in plain text, but also includes the possibility of identifying and navigating to the source of the provided information and conveniently taking action on given suggestions. To this end, the LLM can be instructed to use a prescribed markup language when referencing any input documents or information snippets, as well as when referencing workflow parameters and their values. Specifically, the markup can include a combination of function descriptors ("Jump to document / information", "Display parameter value", "Set parameter value", "Display default / allowed value of parameter", etc.) and internally unique identifiers for the document / information and workflow parameters.

[0063] This allows surrounding systems to parse the LLM output of the tag and to call appropriate functions to provide UI and backend elements and interactions, as well as to display the tag accordingly in the chat (“function call”). Examples could include: any reference to information / documents in the text could be displayed as a link that allows the user to hover over it, displaying a pop-up with further information such as the name and timestamp of the source document. Clicking such a link would switch to a more detailed view of the information within the full document context with a timeline, allowing for easy verification, tracking, and tracing of the information.

[0064] Similar links for any mentioned workflow steps will allow the view to switch to the scanned UI section with that parameter set when clicked. When a specific parameter value is mentioned, clicking the link can immediately set the parameter value accordingly in the backend (possibly protected by a dialog window for confirmation to prevent accidental changes). Using LLM function calls, similar interactions can also be triggered directly by the LLM upon user request. Since the LLM already receives information about all parameters and potential values, it will be able to identify whether the user request involves a non-existent parameter or an unallowed value, and respond accordingly, given appropriate instructions. In any case, the corresponding backend functionality can be implemented in such a way that invalid requests are ignored even if the LLM incorrectly includes such a request in its response.

[0065] In a similar manner, AI-based assistance can also allow users to easily interact not only with the current patient's parameters and their values, but also with the underlying rule set itself. Note that in practice, this is often targeted at a more specific user role: the rule set may reflect the research center's SOPs, typically maintained by the chief technical expert based on the radiologist's requests and decisions.

[0066] Because the rule set is internally represented in a formal language as a dependency graph between descriptive information items and predetermined workflow steps (see above), natural language requests for new or updated rules can be translated by the LLM into an appropriate format, and the resulting modifications are applied to the current rule set representation. The specification of the formal language used to define the rules—that is, what types of rules can be expressed in what way—can be provided to the LLM as a hint to ensure correct results and the ability to identify infeasible or ambiguous requests that require further clarification from the user.

[0067] The scope of change can seamlessly range from modifying individual targets to the incremental creation of complete graphs or subgraphs based on iterative interactions and / or existing documents describing the desired rules in prose or structured formats such as spreadsheets. In this sense, as mentioned above, the transition to LLM-driven "rule mining" is incremental. Since direct editing of large graph structures—even visually accomplished using suitable GUI tools—can be cumbersome even for experts and exceeds the typical expertise and usual working methods of technical personnel, this solution promises to significantly lower the barriers to entry for using the entire system and increase acceptance and ease of use.

[0068] AI-based assistance features can be used for context-aware and / or acquisition-aware retrieval. These features can be able to comment on the inference graph and / or derivation of the currently configured workflow, particularly in the form of an interactive scan guide.

[0069] This is a list of example user / chatbot (“assistant”) interactions demonstrating the possibility of direct interaction with workflows. Note that for clarity, the markup is displayed as plain text here. In a real user interface, such markup could be presented, for example, as a hyperlink. User: Could you outline the most important changes you made to the protocol and why you made them? Assistant: This is related to the basic protocol.<jump_to_param =“baseprotocol”> Routine chest CT scan (I-)< / jump_to_param> The most important deviation in the default settings is recommended for use.<jump_to_information_item=cancerstaging> Lung cancer staging< / jump_to_information_item> : Change<jump_to_param=ScanProtocol::Scan1::pitch> Pitch< / jump_to_param> From 1 to 1.5, because faster scans can help avoid...<jump_to_information_item =motionartifacts> Motion artifacts< / jump_to_information_item> , its in<jump_to_source_document =Report_2024_04_01> Previous inspection in April of this year< / jump_to_source_document> It was observed in the middle. Open<jump_to_param=Visualization::RibUnfolding> Rib spread< / jump_to_param> To facilitate the detection of metastatic lesions, because patients have<jump_to_information_item="HistoryOfCancer"> Previous history of cancer< / jump_to_information_item> . User: Okay, can you set the super brittleness factor to 3? Assistant: I'm sorry, I don't believe "supercalifragiliticity" is a workflow parameter that I can modify. User: Okay, then please set the spacing to at least 100, that will make the scanning speed much faster! Assistant: Sorry, the allowed spacing value in this agreement is...<set_param=ScanProtocol::Scan1::Pitch|0.5> 0.5< / set_param> ,<set_param=ScanProtocol::Scan1::Pitch|1> 1< / set_param> ,<set_param=ScanProtocol::Scan1::Pitch|1.5> 1.5< / set_param> and<set_param=ScanProtocol::Scan1::Pitch|2> 2< / set_param> Do you want me to change one of them or change...?<jump_to_param=BaseProtocol> Basic Protocol< / jump_to_param> ? User: Then please change it to 2, and then... Assistant: Understood.<set_param=ScanProtocol::Scan1::Pitch|2> Please click here to confirm the changes.< / set_param> . User: In addition, I also need to see the spine, not just the ribs! Assistant: Do you wish to generate<jump_to_param=Visualization::SpineStraightening> Visualization of spinal straightening< / jump_to_param> ? User: Yes! Assistant: Of course. Please click here.<set_param=Visualization::SpinestRightening|True> confirm< / set_param> Or reply with "Confirm". User: Confirm Helper: Automatically calls the function associated with the above tags.

[0070] In another aspect, the present invention relates to a computer-implemented method for providing medical imaging workflow support data, the method comprising: - Receive reference data, - Generate medical imaging workflow support data by applying a retrieval-enhanced generative model to the reference data. - Provides data to support medical imaging workflows.

[0071] Large Language Models (LLMs) can be used to create and modify protocol files, particularly in the form of XML protocol files, and / or other reference files based on templates and examples, such as decision tree constraints, without requiring full constraints and specifications.

[0072] Retrieval Enhancement Generation (RAG) can be used to dynamically query protocols and sample databases to adapt or synthesize protocols from scratch. Additional documentation can be provided, such as protocol excerpts, scanner manuals, and globally or site-specific clinical guidelines that are dynamically and optimally selected for the required modifications.

[0073] Since LLMs can be very proficient in working with formal languages ​​and can learn from examples without modifying the underlying syntactic structure, they can intelligently fill in missing information based on given instructions and public knowledge derived from their training and example data by learning from a small number of samples in the context of example protocol files.

[0074] This method enables the creation and tailoring of scanner-compatible protocol files—the main protocol XML file and any other relevant files required to fully define the protocol—based on a more abstract representation of the desired workflow and scanning strategy. Therefore, formal languages ​​(e.g., JSON), protocol documents (typically available digitally, such as as text or tabular documents), and ultimately natural language interactions can be used to maintain a simple and high-level representation, as described regarding inference graphs.

[0075] Using this approach, scanner-specific protocols can be automatically derived by selecting a sample protocol definition for the appropriate scanner and / or by providing additional information about the scanner's capabilities (such as by including the scanner manual as context). Furthermore, the same mechanism can be employed to convert abstract, patient-customized protocol configurations created during examination sequencing, protocol, or scheduling in EHR and / or cloud-based environments into specific acquisition protocols for the selected scanner. This allows for fine-grained scan configuration from a UX-friendly queue-level perspective.

[0076] Hybrid approaches may be needed, especially if an LLM struggles to reliably perform these tasks with complex protocols given its size and contextual constraints. On the other hand, high-level document pre-segmentation can be applied using traditional domain-knowledge-driven methods. The overall structure and general semantic hierarchy of the scanned protocol document typically stabilize over time and are similar across different scanners. Relevant sub-sections are then provided to the LLM for further modification, making this approach feasible. Thus, while XML parsing can be used for meaningful pre-segmentation, detailed modifications and adjustments can be handled at the semantic level using an LLM.

[0077] The proposed solution allows for the automation of the process (automatic protocol maintenance, automatic protocol transfer). Optionally, dual-examination results and secondary target corrections can be performed manually and / or via natural language-based instructions. User interaction at the cloud or EHR level can initiate the automated transfer of generic or patient-customized protocols to different scanners or different versions of scanners. Direct natural language interaction may lead to the creation or modification of scanning protocols.

[0078] Based on medical imaging workflow support data, medical imaging examination workflow steps, options, and / or parameters can be suggested, selected, and / or triggered. Medical imaging workflow support data can indicate the values ​​and / or changes in values ​​of scan parameters used in the scanning protocol for medical imaging examinations performed on a patient by a medical imaging device. Methods for providing medical imaging workflow support data may also include: - setting scan parameters and / or constructing a medical imaging examination based on the scan parameters.

[0079] Medical imaging workflow support data can indicate the values ​​and / or changes in the values ​​of reconstruction parameters used in reconstruction algorithms to reconstruct medical images based on medical imaging data. Methods for providing medical imaging workflow support data may also include: - setting reconstruction parameters and / or reconstructing medical images based on medical imaging data. Medical imaging workflow support data can also indicate the values ​​and / or changes in the values ​​of image processing parameters used in image processing algorithms to process medical images. Methods for providing medical imaging workflow support data may also include: - setting image processing parameters and / or processing medical images. Image processing algorithms can be configured, for example, for post-processing and / or analysis of medical images. Post-processing of medical images may include, for example, calculating representations of anatomical structures based on medical images.

[0080] For example, structured information can be used to enhance and / or automate scanning and / or post-processing workflows. To this end, information from structured and unstructured reports, as well as referral requests, can be automatically structured relative to clinical questions related to the medical imaging examination to be performed, while including all specific aspects surrounding that concern. Exemplarily, based on given textual information, large language models can be used to quickly answer relevant questions such as “Is the patient suitable for contrast agents?”, “Is high-resolution reconstruction recommended for a given indication?”, “In which organs should an automated search for metastases be performed?”, and similar questions.

[0081] Structured information, particularly structured information about patient demographics, prior findings, or comorbidities, can be used, for example, to select and / or trigger appropriate workflow steps, to select appropriate parameters for scanning, reconstruction, and post-processing, and as prior information for any machine learning-based image processing algorithms running on the resulting image dataset.

[0082] On the other hand, the present invention relates to a data processing system for providing support data for medical imaging workflows, the data processing system comprising an interface unit and a computing unit. - The interface unit is configured to receive natural language data, including clinical information. - The computational unit is configured to generate structured information by applying a large language model to natural language data. The structured information includes clinical information in a structured format according to at least one input node of the inference graph. - The computing unit is further configured to compute medical imaging workflow support data by applying at least one rule of the inference graph to the structured information, and - The interface unit is also configured to provide medical imaging workflow support data.

[0083] A data processing system can be configured to perform a method according to one aspect of the invention. The data processing system may include at least one of, for example, a cloud computing system, a distributed computing system, a computer network, a computer, a tablet computer, a smartphone, etc. The data processing system may include hardware and / or software. The hardware may be, for example, a processor system, a memory system, or a combination thereof. The hardware may be configured by software and / or operated by software. Calculations for the actions of performing the method may be executed in the processor.

[0084] The computing unit can be implemented as a data processing system or a part of a data processing system. Such a data processing system may include, for example, a cloud computing system, a computer network, a computer, a tablet computer, a smartphone, etc. The computing unit may include hardware and / or software. The hardware may include, for example, one or more processors, one or more memories, and combinations thereof. The one or more memories may store instructions for performing the method steps according to the invention. The hardware may be configured and / or operated by software. Typically, all units, sub-units, or modules may exchange data with each other, at least temporarily, for example, via a network connection or a corresponding interface. Therefore, the individual units may be located separately from each other.

[0085] The interface unit may include an interface for exchanging data with a local server or a central network server via an Internet connection to receive intermediate image datasets. The interface unit may also be adapted to interface with one or more users of the system, for example, by displaying the processing results of the computing unit to the user (e.g., in a graphical user interface) or by allowing the user to adjust parameters used for data processing or visualization. In other words, the interface unit may include a user interface.

[0086] Data, particularly natural language data, queries, additional query data, reference data, and examination request data, can be received, especially through the interface unit, for example by receiving signals carrying data and / or by reading data from computer memory and / or by manual user input, such as through a graphical user interface. Data, particularly medical imaging workflow support data, can be provided, especially through the interface unit, for example by sending signals carrying data and / or by writing data to computer memory and / or by displaying data on a monitor.

[0087] Any algorithms, functions, and / or models mentioned in this article may be based on one or more of the following architectures: deep convolutional neural networks, deep belief networks, random forests, deep residual learning, deep reinforcement learning, recurrent neural networks, Siamese networks, generative adversarial networks, or autoencoders.

[0088] The present invention also relates to a medical imaging apparatus comprising a data processing system according to one aspect of the invention. The medical imaging apparatus can be configured to perform medical imaging examinations on a patient based on a scanning protocol for medical imaging examinations. The medical imaging apparatus can be, for example, a computed tomography (CT) apparatus, a magnetic resonance imaging (MRI) apparatus, an ultrasound imaging apparatus, a PET imaging apparatus, a SPECT imaging apparatus, an X-ray imaging apparatus, a cone-beam CT apparatus, and / or combinations thereof.

[0089] In another aspect, the present invention relates to a computer program product comprising instructions which, when executed by a computer, cause the computer to perform a method according to one aspect of the invention. In another aspect, the present invention relates to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform a method according to one aspect of the invention. An advantage of implementing the invention through a computer program product and / or a computer-readable storage medium is that existing computers can be easily adapted through software updates to operate as proposed by the invention.

[0090] A computer program product can be, for example, a computer program or include other elements besides a computer program. This other element can be hardware, such as a storage device on which the computer program is stored, hardware keys for using the computer program, and / or software, such as documentation or software keys for using the computer program. A computer program product may also include development materials, a runtime system, and / or a database or library. A computer program product can be distributed across multiple computer instances. Computer-readable storage media can be embodied as non-persistent main memory (e.g., random access memory) or persistent mass storage (e.g., hard disk, USB stick, SD card, solid-state drive).

[0091] Compared to symbolic ontology methods, using large language models that understand language rather than merely parse it offers significant advantages. Symbolic ontology methods often struggle with negation, inversion, abbreviations, typos, and extraneous information, or are "hard-coded" into parsing, thus being very limited or unable to handle this information at all. Conversely, language understanding, information extraction, and content summarization are key capabilities of typical large language models, allowing these problems to be avoided. Specifically, while negation and inversion are integral parts of natural language and are therefore richly represented, world knowledge, along with explicitly given query information, can be used to resolve abbreviations and typos. Extraneous information can be explicitly queried and summarized, such as in the form of easily understandable bullet points, which themselves can refer to the original report.

[0092] As a result, the use of large language models can revolutionize the way structured information is created from unstructured natural language data, becoming a major step in the digital transformation of clinics, increasing throughput, accelerating treatment, serving as a double check to avoid treatment errors, and ultimately reducing the workload for stakeholders. A particular focus is on automating scanning and post-processing workflows, as proposed by Technology Co-pilot, with advantages that are also multifaceted. Key aspects include saving time through smart scanners, automatically pre-selecting the correct workflows and steps, and triggering only relevant and specially tailored algorithms to create relevant (AI-based) image analysis results, standardization, and improved consistency of examinations. In summary, this allows for highly reproducible, guideline-based workflow recommendations and has the powerful potential to reduce errors by acting as an automated "second pair of eyes" that automatically parses any available text-based sources of relevant information.

[0093] In the context of this invention, the expression "based on" can be particularly understood to mean "especially used". Specifically, the wording on which the first feature is calculated (or generated, determined, etc.) based on the second feature does not preclude the possibility of calculating (or generating, determining, etc.) the first feature based on the third feature. Please take note of the fact that the described methods and systems are merely preferred exemplary embodiments of the invention, and that those skilled in the art can modify the invention without departing from the scope of the invention as defined by the claims.

[0094] Figure 1 A flowchart is shown of a computer-implemented method for providing medical imaging workflow support data, the method comprising:

[0095] - Receive natural language data U from S1, wherein the natural language data U includes clinical information.

[0096] - S2 structured information N is generated by applying a large language model L to natural language data U. The structured information N includes clinical information in a structured format according to at least one input node 31, 32 of the inference graph G.

[0097] - S3 medical imaging workflow support data is computed by applying at least one rule of the inference graph G to the structured information N, and

[0098] - Provides S4 medical imaging workflow support data.

[0099] Figure 2 A data processing system P is shown for providing support data for medical imaging workflows. The data processing system P includes an interface unit PI and a computing unit PC. - The interface unit PI is configured to receive S1 natural language data U, which includes clinical information. - The computing unit PC is configured to generate S2 structured information N by applying a large language model to natural language data U. The structured information N includes clinical information in a structured format according to at least one input node 31, 32 of the inference graph G. - The computing unit PC is further configured to compute the medical imaging workflow support data described in S3 by applying at least one rule of the inference graph G to the structured information N, and - The interface unit PI is also configured to provide S4 medical imaging workflow support data.

[0100] exist Figure 3 and Figure 4 An exemplary implementation of manually editing the inference graph G through the user interface 4 is described in the document.

[0101] Input node 31 represents the basic protocol conditions. Input node 32 represents the metal artifact conditions. Edges 31A, 31B, 31C, and 31D represent conventional chest (I+), conventional chest (I-), nodule follow-up (I-), and CT PE (I+), respectively, as potential values ​​for the basic protocol. Each connection in 3A, 3B, 3C, and 3D represents a logical connection (“AND”). Target nodes within set 33 are related to reconstruction. Nodes within subset 34 are related to specific reconstruction algorithms, particularly iterative reconstruction 34A and metal artifact reduction 34B. Nodes within subset 35 are related to soft thin reconstruction, particularly kernel 35A, slice increment 35B, and slice thickness 35C. Nodes within subset 36 are related to dual-energy computed tomography reconstruction, particularly PBV 36A, VMI 36B, and VNC 36C. Other nodes are related to lung thickness reconstruction 37 and lung thin reconstruction 38.

[0102] The medical imaging workflow supports data corresponding to workflow aspect nodes of the inference graph G. The representation of the inference graph G is visualized in user interface 4, through which user input, particularly manual and / or natural language user input, is received, and the inference graph G is modified based on this user input. Therefore, an intuitive graph editing method based on a visualization of a subset of the inference graph G can be used to modify the inference graph G.

[0103] Figure 5 A flowchart is shown of a computer-implemented method for protocol modification.

[0104] The protocol modification function 50 generates an output protocol file 58 based on the input protocol file 51 and prompts 53, whereby prompts 53 include descriptions / requests (e.g., "Recon 3 slice thickness should be halved" or equivalent codes) and / or links to protocol book data. The reference database 52 includes supplementary documents such as protocol excerpts, scanner manuals, and / or global and / or site-specific clinical guidelines, which are dynamically and optimally selected relative to the required modifications by retrieving and enhancing the generative model.

[0105] The protocol modification function 50 includes a function 54 for pre-segmenting the input protocol file 51, a function 55 for context-aware retrieval and retrieval enhancement generation, a function 56 for processing prompts and modifying the protocol file, and a function 57 for reassembling to obtain the output protocol file.

Claims

1. A computer-implemented method for providing medical imaging workflow support data, the method comprising: - Receive (S1) natural language data (U), the natural language data (U) including clinical information, - Structured information (N) is generated (S2) by applying a large language model (L) to the natural language data (U), the structured information (N) including the clinical information in a structured format according to at least one input node (31, 32) of the inference graph (G). - The medical imaging workflow support data is computed (S3) by applying at least one rule of the inference graph (G) to the structured information (N), and - Provides the medical imaging workflow support data described in (S4).

2. The method according to claim 1, - Wherein the medical imaging workflow support data corresponds to the workflow aspect nodes of the inference graph (G).

3. The method according to claim 1 or 2, - The inference graph (G) mentioned therein is based on clinical procedure guideline data and / or on a set of paired patient and workflow data.

4. The method according to any one of claims 1 to 3, - Reference data is received. - The inference graph (G) is generated by applying a retrieval-enhanced generative model to the reference data.

5. The method according to any one of claims 1 to 4, - The representation of the inference graph (G) and / or the representation of a subset of the inference graph (G) are visualized in the user interface (4). - User input is received through the user interface (4). - The inference graph (G) is modified based on the user input.

6. The method according to any one of claims 1 to 5, - The additional information is queried from medical information systems and / or from users, particularly in the form of AI-based assistive functions, especially in the form of chatbots. - Wherein the inference graph (G) is modified based on the additional information and / or wherein the medical imaging workflow support data is calculated based on the additional information.

7. A computer-implemented method for providing medical imaging workflow support data, the method comprising: - Receive reference data, - The medical imaging workflow support data is generated by applying a retrieval-enhanced generative model to the reference data. - Provides supporting data for the medical imaging workflow.

8. The method according to any one of claims 1 to 7, - The medical imaging workflow steps, options, and / or parameters are suggested, selected, and / or triggered based on the medical imaging workflow support data.

9. The method according to any one of claims 1 to 8, - The medical imaging workflow supports data indicating the values ​​and / or changes in the values ​​of scanning parameters for the scanning protocol used in medical imaging examinations performed on patients by medical imaging equipment.

10. The method according to any one of claims 1 to 9, - The medical imaging workflow supports data indication of the values ​​and / or value changes of reconstruction parameters used in reconstruction algorithms to reconstruct medical images based on medical imaging data.

11. The method according to any one of claims 1 to 10, - The medical imaging workflow supports data indication of the values ​​and / or value changes of image processing parameters used in the image processing algorithms for processing medical images.

12. A data processing system (P) for providing support data for medical imaging workflows, the data processing system (P) comprising an interface unit (PI) and a computing unit (PC). - The interface unit (PI) is configured to receive (S1) natural language data (U), which includes clinical information. - The computing unit (PC) is configured to generate (S2) structured information (N) by applying a large language model to the natural language data (U), the structured information (N) including the clinical information in a structured format according to at least one input node (31, 32) of the inference graph (G). - The computing unit (PC) is further configured to compute (S3) the medical imaging workflow support data by applying at least one rule of the inference graph (G) to the structured information (N), and - The interface unit (PI) is further configured to provide (S4) the medical imaging workflow support data.

13. A medical imaging device comprising the data processing system (P) according to claim 12.

14. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 11.

15. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 11.

Images