Method for providing medical imaging determination support data

Abstract

To provide a method for providing medical imaging determination support data, a program, and a computer-readable storage medium which make it easier to support medical imaging determination.SOLUTION: The method for providing medical imaging determination support data includes the steps of: receiving natural language data U including patient unique clinical information (S1); applying a large-scale language model to the natural language data U and generating structuring information N including the patient unique clinical information of the structuring format (S2); calculating the medical imaging determination support data on the basis of the structuring information N (S3); and providing medical imaging determination support data (S4).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] In one aspect, the invention relates to a computer implemented method for providing medical imaging decision support data, in other aspects the invention relates to a data processing system, a medical imaging device, a computer program product, and a computer readable storage medium. [Background technology]

[0002] In clinical applications, it is of utmost importance to make fast and accurate decisions. As a result of operational requirements, e.g. time schedules and needs regarding medical devices, routine clinical evaluations are not necessarily patient-centered, but more often expert-centered, i.e. the patient is referred to different experts as required according to the indications derived by the medical history up to that point.

[0003] As a result, patient-specific clinical information must be communicated consistently between different diagnostic laboratories or clinical specialists within the same institution to the next point of care. Although various standardized medical terminologies, treatment and disease codes, protocols, etc. exist, directly treatment-related information can be lost in the medical evaluation, leaving many patients at risk for delayed or interrupted treatment and / or additional harm.

[0004] Therefore, it is important to use predefined and well-structured templates that provide a blueprint of the required data to ensure that all the required clinical information is available. Although dedicated methods exist for some specific diseases, such as TNM classification for cancer diseases, RADS variants (e.g., CAD-RADS or Bone-RADS), the use of unified and structured reporting of information in clinical environments is usually rare. And even if a suitable scheme exists, as in the above example, the way in which the information is actually presented still varies considerably in practice.

[0005] The main reason is the lack of an integrated software solution for conveying clinical information and the resulting reliance on various means of communication. Clinical IT landscapes tend to be very diverse, and despite unified communication protocols such as FHIR and DICOM, solutions are often vendor-specific or contain vendor-specific extensions and therefore cannot be universally used within a clinic. Moreover, additional restrictions may apply in light of data security and data privacy regulations.

[0006] In a typical scanner workflow, a radiologist makes a decision about which imaging exam to perform based on free-text clinical indications. After reading the scans, the radiologist records the findings in an unstructured report that later forms the basis for disease diagnosis. The associated steps such as scanning protocol selection, reading workflow, and report generation to date can only be partially supported by software due to its usual inability to process natural language.

[0007] As a result, clinical information is often communicated in an unstructured manner, most notably in unstructured clinical evaluation reports, or short unstructured reports. Although clinicians are trained to use highly descriptive, concise, and compact language, these unstructured reports do not inherently (i.e., by the technical procedures used to create them) require clinicians to include all relevant information, and do not necessarily serve as an accurate source of information on what needs to be evaluated in further processes. Thus, the completeness of information in unstructured reports is primarily the result of clinical experience and compliance with guidelines or regulations, and is further subject to the characteristics of the disease, physician fatigue, and time constraints, etc.

[0008] It is worth noting that unstructured text-based information not only plays an integral role between clinical organizations and departments, but also between scanning and post-processing workflows. As just a few examples, each decision regarding the choice of scanning protocol, the selected image reconstruction, and the automatically applied post-processing algorithms is directly and immediately based on the clinical indication, i.e., the reason for the consultation, and possibly other clinically relevant information about the patient. In particular, the reason for the consultation may itself be influenced by previous patient visits. Thus, relevant information sources for the scanning and post-processing workflow also include previous diagnostic reports. Typically, most of these data are unstructured and must be transformed, i.e., translated, mapped, and protocolized, into the appropriate imaging and post-processing procedures. This transformation may be performed manually by radiologists and technicians. As with the availability of cutting-edge technologies such as photon-counting CT, the number of available options is expected to steadily increase, making automated support of these tasks of paramount clinical importance.

[0009] Patent Document 1 discloses a technical idea of ​​using an adaptation ontology to control a medical imaging system based on adaptation information.

[0010] Efforts have been made to create automated solutions to extract key information from unstructured reports for workflow automation. The methods have been largely built on keyword searches using lexicographic ontologies such as RadLex. By their nature, these methods often suffer from inaccurate parsing of information, as they are unable to fully take into account the context. Since these models process reports with a lexicon of symbolic representations, rather than with a real understanding of the environment, they must deal in particular with negation and inversion, abbreviations, redundant information, and other content that breaks the linear language structure. Furthermore, these methods must be designed for a specific clinical task, i.e., they can only be applied to one or a few diseases, because the underlying ontologies must be augmented with a wide variety of additional rules to handle the aforementioned issues.

[0011] There are recent steps such as the introduction of DICOM-TID-1500 to provide a vendor-neutral structure for encoding diagnostic findings, but they have not yet been widely adopted. Moreover, they do not have a default way of filling in the data, which is inconvenient in the everyday clinical workflow. It is clear that coding of findings and indications, with mapping to structured, predefined medical codes, apart from hierarchy, has not yet been widely adopted in clinical practice. Especially when it comes to the scanning and post-processing workflow itself, many sites still rely heavily on unstructured, text-based information, despite their shortcomings. [Prior art documents] [Patent documents]

[0012] [Patent Document 1] European Patent Application Publication EP3451211A1 Summary of the Invention

[0013] The underlying technical problem of the present invention is to facilitate improved medical imaging decision support, especially with regard to aspects of natural language processing. This problem is solved by the subject matter of the independent claims. The dependent claims relate to further aspects of the invention. Regardless of grammatical usage of the phrase, individuals of male, female and other gender identities are included in the phrase.

[0014] In one aspect, the present invention relates to a computer-implemented method for providing medical imaging decision support data, the method comprising: receiving natural language data including patient specific clinical information; generating structured information comprising the patient-specific clinical information in a structured format by applying a large-scale language model to the natural language data; calculating medical imaging decision support data based on said structured information; Providing said medical imaging decision support data.

[0015] In particular, the natural language data may include patient-specific clinical information in an unstructured format. For example, the natural language data may include an unstructured report containing the patient-specific clinical information. The patient-specific clinical information may include, for example, information regarding demographic parameters, in particular gender, weight, height and / or age, physiological parameters, previous findings, previous treatments, clinical history and / or medical history of the patient. The patient-specific clinical information may include, for example, a referral request and / or an indication and / or a reason for a medical imaging visit.

[0016] The large scale language model is configured to recognize and / or understand natural language and in particular individual items such as words in an input that includes natural language and transitions the items to a text output. The large scale language model is trained and / or machine learned. The large scale language model is provided, for example, by keeping the large scale language model available in a suitable storage accessible by a computing unit that executes the method and / or by downloading the model by said computing unit.

[0017] Large-scale language models are used to convert unstructured information into structured formats and / or vice versa. Exemplary large-scale language models have a particular strength in summarizing and presenting information in a way that a user encourages the large-scale language model to perform. Large-scale language models have an intrinsic ability to extract information from text. Large-scale language models are configured for natural language processing (NLP), and in particular for NLP-based extraction of structured information from natural language data.

[0018] For example, the structured information may include a structured report, which may include patient-specific clinical information in a structured format, e.g., a predefined and / or computer readable format, and the natural language data and / or the structured information may be text-based.

[0019] In a structured format, patient-specific clinical information can be easily extracted and efficiently communicated in a machine-readable format using predefined automated communication protocols. Notably, by utilizing a large-scale language model capable of language understanding rather than parsing only, the present approach avoids various major problems arising from the use of approaches based solely on parsing, such as the mentioned inverse problem. In particular, the structured information can be easily parsed into downstream applications, including, if necessary, the integration of additional standard algorithms or prompting the user to provide information that is still missing. The structured information includes sets of structured information items. These sets can be represented, for example, by codes from widely used coding systems (ICD-10, LOINC, CPT, ...) and / or custom task-specific representations. In particular, the structured information can include codes, for example ICD-10 or CPT codes, despite the absence of codes in the natural language data.

[0020] The medical imaging decision support data is calculated, for example, by applying a function for calculating the medical imaging decision support data to the structured information. The function for calculating the medical imaging decision support data is trained, in particular machine learned. For example, a large-scale language model can include the function for calculating the medical imaging decision support data. Alternatively, the function for calculating the medical imaging decision support data can be rule-based.

[0021] A particular advantage of the proposed method is due to its strong understandability (comprehensiveness). Since the information is presented in a highly structured format, it is easily possible for humans to find inconsistent information, thus reducing novel error sources that may arise from ontology-based approaches. Furthermore, the reconfigurability of the large-scale language model allows for easy adaptation in case of updated report templates, new tasks and diseases, and finally updates of previously created structured reports, since reconfiguration corresponds to a change in the query but not to a change in the architecture of the large-scale language model.

[0022] The large-scale language model may include a Transformer network. A Transformer network is a neural network architecture that typically includes an encoder, a decoder, or both an encoder and a decoder. In some examples, the encoder and / or the decoder are each composed of several corresponding encoding and decoding layers. In each of the encoding and decoding layers, there is an attention mechanism. An attention mechanism, sometimes called self-attention, associates each data item (such as a word or pixel) in a sequence of data items with other data items in the sequence. A self-attention mechanism, for example, allows the model to examine a word in a sentence and determine the relative importance of another word in the sentence to the examined word.

[0023] The encoder is specifically configured to convert the input (medical image or text) into a numerical representation. The numerical representation includes a vector for each input token (e.g., for each word). The encoder is configured to perform an attention mechanism such that each vector of tokens is influenced by other tokens in the input. In particular, the encoder is configured such that the representation determines the desired output of the Transformer network. The decoder is specifically configured to convert the input into a sequence of output tokens. In particular, the decoder is configured to perform a masked self-attention mechanism such that each vector of tokens is influenced only by other tokens to one side of the sequence. Furthermore, the decoder may be in an autoregressive sense in that intermediate results (such as sequences of previously predicted tokens) are fed back. According to some examples, the output of the encoder is input to the decoder. The Transformer network also includes a classification module or unit configured to map the output of the encoder or decoder to a set of learned outputs such as a text summary.

[0024] The training of the Transformer model according to some examples is performed in two stages: pre-training and fine-tuning. In the pre-training stage, the Transformer model is trained on a large corpus of data to learn the underlying semantics of the problem. Such pre-trained Transformer models are available for different languages. For the specific application described herein, fine-tuning may include further training the Transformer network using medical texts with expert annotated meanings and / or medical ontologies such as RADLEX and / or SNOMED. Using the latter, in particular, the Transformer model according to some examples can learn typical relations and synonyms of medical expressions.

[0025] Large-scale language models include, for example, the Transformer architecture. A Transformer network can use previously inserted text to find logical continuations by interrogating previously provided information, extracting key information, and assigning a value to each. By performing this process layer by layer, large-scale language models predict one token (i.e., word or word part) at a time, with each token based on the provided information plus knowledge of the world implicit in the network weights.

[0026] A query may be received, the query including natural language data and additional query data, and the structured information is generated by applying a large-scale language model to the natural language data and additional query data. The query may include, for example, a user query. The user query may be a free text query, and may be entered by a user through the user interface, for example by typing the user query into an appropriate input field in the user interface or by voice command.

[0027] The additional query data includes template data, the template data indicating a structured format. In particular, a large-scale language model can comprise dynamically configurable templates that can be queried to transform the natural language data and thereby fill in the relevant information in the comprised templates.

[0028] Reference data may be received, the reference data indicating rules and / or examples relevant to generating the structured information and / or relevant to computing the medical imaging decision support data, and the large scale language model is trained based on the reference data, in particular self-trained and / or fine-tuned based on the reference data.

[0029] The large-scale language model is configured to consider reference data as additional information sources, in particular in the form of generic medical guidelines, site-specific rules and / or preferences, and / or patient medical history. These examples may relate to different patients and / or consultations. In particular, the reference data includes corresponding preferred medical imaging decisions for another set of example patient-specific clinical information. In particular, additional query data may include the reference data. In particular, the reference data may be automatically injected as part of the query. Based on the reference data, the large-scale language model may identify patterns and / or reasons why a certain recommendation is desirable for a new patient and / or consultation, thereby enabling, for example, on-the-fly learning.

[0030] The explanatory data may be computed by a large scale language model, the explanatory data providing explanations and / or reasons for the generation of the structured information and / or the computation of the medical imaging decision support data, the explanatory data being provided.

[0031] The large-scale language model includes functions of explanation, particularly interactive explanation, and / or reasoning, particularly interactive reasoning, especially for structured information and / or for medical imaging decision support data. As a key capability, a typical large-scale language model is essentially capable of explaining its decisions in relation to the context in which it is applied. The explanation data may further provide explanations and / or reasons for the adjustment of the large-scale language model based on the reference data. The explanation data is automatically recorded and / or made directly accessible to the user, for example as overlay hovering, decision log, and / or information button and / or through interactive querying that allows cross-checking the automated reasoning, for example in the form of a natural language dialogue.

[0032] For example, the explanatory data may include citations of content from the natural language data, particularly from unstructured reports, as verifiable sources for the generated structured information and / or for the computed medical imaging decision support data. For each citation of content from the natural language data, the explanatory data may include citation-specific relevance information and / or citation-specific context information. The context information may include, for example, references to guidelines, rules, preferences, patient histories and / or examples (if any), particularly together with a summary of the relevance of each reference for a given case.

[0033] Consultation request data may be received, the consultation request data indicating a type of medical imaging visit, and the medical imaging decision support data indicating the appropriateness of the type of medical imaging visit given the patient-specific clinical information.

[0034] When a scan is ordered by a specialist physician, it is essential to check that the requested type of scan is appropriate for the determined indication. Given a text-based query request and a description of an appropriate scan for the indication, the large-scale language model can perform this check with minimal interaction, which is particularly relevant when the user does not agree with the assessment. Furthermore, the large-scale language model can be queried regarding the reasoning regarding this decision in order to quickly determine whether there was a human oversight or whether the decision is not an erroneous conclusion by the large-scale language model due to, for example, missing information.

[0035] Medical imaging consultation workflow steps, options and / or parameters may be suggested, selected and / or triggered, in particular automatically suggested, selected and / or triggered, based on the medical imaging decision support data.

[0036] The medical imaging decision support data may be indicative of values ​​and / or changes in values ​​of scanning parameters of a scanning protocol for a medical imaging examination of a patient with a medical imaging device. The method for providing medical imaging decision support data further comprises setting the scanning parameters and / or performing a medical imaging examination based on the scanning parameters.

[0037] The medical imaging decision support data may be indicative of values ​​and / or changes in values ​​of reconstruction parameters of a reconstruction algorithm for reconstructing a medical image based on the medical imaging data. The method for providing medical imaging decision support data further comprises setting the reconstruction parameters and / or reconstructing a medical image based on the medical imaging data.

[0038] The medical imaging decision support data may be indicative of values ​​and / or value changes of image processing parameters of an image processing algorithm for processing the medical image. The method for providing medical imaging decision support data further comprises setting the image processing parameters and / or processing the medical image. The image processing algorithm is, for example, configured to post-process and / or analyze the medical image. Post-processing of the medical image comprises, for example, computing a representation of an anatomical structure based on the medical image.

[0039] The structured information can be used, for example, to enhance and / or automate the scanning workflow and / or the post-processing workflow. For this purpose, the information from the structured and unstructured reports together with the referral request is automatically structured in relation to the clinical question regarding the medical imaging consultation to be performed, including all the specific aspects surrounding the concern. For example, based on the given text information, large-scale language models can be used to quickly answer relevant questions such as "Can the patient accept contrast media?", "What is the recommended high-resolution reconstruction for a given indication?", "In which organs should an automatic exploration of metastases be performed?", etc.

[0040] Structured information, in particular regarding patient demography, previous findings or comorbidities, can be used, for example, to select and / or trigger appropriate workflow steps, to choose appropriate parameters for scanning, reconstruction and post-processing, and as prior information for any machine learning based image processing algorithms performed on the resulting image dataset.

[0041] As large-scale language models are trained to take this additional input information into account, their accuracy is expected to improve, and results can also be automatically checked for consistency with the available information, for example reducing the likelihood of false positive findings.

[0042] The method may be implemented in the form of a Tech Co-pilot that assists technicians in their daily work. The method allows to automatically determine relevant protocol improvement steps given previous reports and clinical indications, i.e. reasons for consultation. These data may be obtained, for example, by utilizing open standards such as the DICOM Modality Worklist and the FHIR standard. However, using this data, the proposed Tech Co-pilot is envisaged to enable automatic protocol improvement, including scanner options such as image coverage, reconstruction and reformatting, multi-energy, spectral and multi-phase acquisition, kernel selection, dose and exposure control (Care keV), bolus administration, timing and tracking, inspiratory or expiratory scanning, (quantum) iterative reconstruction and ADMIRE parameterization, metal and motion artifact reduction, and subsequent CAD applications.

[0043] For patients with previously completed medical records and in the presence of screening guidelines, the method can be used to identify patients who are most suitable for and would benefit most from pulmonary screening. If the decision to perform a screening visit was made by a clinician, this information is present in the query text as "reason for visit" and can therefore be interpreted as such by a large-scale language model.

[0044] According to this information, an appropriate scan protocol is automatically determined based on the present method for providing medical imaging decision support data. That is, a specific ultra-low dose protocol is used for screening to avoid excessive radiation exposure of potentially healthy patients, and an appropriate lung CAD algorithm is automatically triggered to detect potential lung nodules in the acquired images. In case of scanners that allow acquisition of spectral information, such as photon counting scanners, a quantitative evaluation of the composition of each detected lesion for overlay characterization can also be automatically triggered. Finally, large-scale language models are used to summarize the results and findings and / or elaborate them in a structured report.

[0045] Cancer patients undergo regular follow-up scans for staging to determine disease progression and treatment effectiveness. Therefore, several previous diagnostic reports are usually available, detailing, among other things, the primary cancer type, identified target lesions, and finally all known metastatic loci. This information can be transferred to a large-scale language model. The method for providing medical imaging decision support data can further include at least one of the following steps: Summarize disease progression to date and make this readily available as a reference for ongoing care. Identifying optimal baseline scans and associated target lesions therein by comparing them with the current consultation using lesion detection and segmentation algorithms (which are provided with information about known lesion locations to increase the confidence in their results) and automatically assisting in the calculation of quantitative measures such as RECIST. Triggering the generation of appropriate visualizations, e.g. in the case of a pancreatic cancer patient, a high-resolution stack in standard orientation containing unfolded images of the organ and its surrounding vasculature together with a contrast overlay map. Determining which organs are likely to be affected by metastatic lesions and automatically triggering appropriate lesion detection algorithms. · Summarize current findings in structured report data and contrast this with previous reports. · Summarize clinical guidelines for staging of corresponding types of cancer and answer specific questions about how staging should be performed.

[0046] The invention further relates to a data processing system for providing medical imaging decision support data, the data processing system comprising an interface unit and a computing unit. The interface unit is configured to receive natural language data including patient-specific clinical information. A computing unit is configured to generate structured information comprising the patient-specific clinical information in a structured format by applying a large-scale language model to the natural language data. The computing unit is further configured to calculate medical imaging decision support data based on said structured information. The interface unit is further configured to provide said medical imaging decision support data.

[0047] The data processing system is configured to carry out a method according to any of the aspects of the present invention.

[0048] The data processing system may include, for example, at least one of a cloud computing system, a distributed computing system, a computer network, a computer, a tablet computer, a smartphone, and the like. The data processing system may be configured to include hardware and / or software. The hardware may be, for example, a processor system, a memory system, and combinations thereof. The hardware may be configurable and / or operable by the software. Calculations for performing the functions of the method are performed in the processor.

[0049] The computing unit is implemented as or as part of a data processing system. The data processing system may include, for example, a cloud computing system, a computer network, a computer, a tablet computer, a smartphone, etc. The computing unit may include hardware and / or software. The hardware may include, for example, one or more processors, one or more memories, and combinations thereof. The one or more memories store instructions for performing the steps of the method according to the invention. The hardware may be configurable and / or operable by software. Generally speaking, all units, subunits, or modules exchange data with each other, at least temporarily, for example via a network connection or a respective interface. Accordingly, the individual units may be located remotely from each other.

[0050] The interface unit may include an interface for data exchange with a local server or a central web server via an Internet connection to receive the medical image dataset. The interface unit may be further configured to interact with one or more users of the system, for example by displaying results of processing by the computing unit to the user (e.g. in a graphical user interface) or by allowing the user to adjust parameters of data processing or visualization. In other words, the interface unit includes a user interface.

[0051] Data, in particular natural language data, queries, further query data, reference data and consultation request data, may be received, in particular through the interface unit, for example by receiving a signal carrying the data and / or by reading data from a computer memory and / or by manual user input, for example via a graphical user interface. Data, in particular medical imaging decision support data and / or explanatory data, may be provided, in particular through the interface unit, for example by sending a signal carrying the data and / or by writing data to a computer memory and / or by displaying data on a display.

[0052] Any algorithms, functions, and / or models referred to herein are based on one or more of the following architectures: deep convolutional neural networks, deep belief networks, random forests, deep residual learning, deep reinforcement learning, recurrent neural networks, Siamese networks, generative adversarial networks, or autoencoders.

[0053] The invention further relates to a medical imaging device including a data processing system according to any of the aspects of the invention, the medical imaging device being configured to perform a medical imaging examination of a patient based on a scan protocol for the medical imaging examination, the medical imaging device being, for example, a computed tomography device, a magnetic resonance imaging device, an ultrasound imaging device, a PET imaging device, a SPECT imaging device, an X-ray imaging device, a cone beam CT device and / or a combination thereof.

[0054] The invention further relates to a computer program product comprising instructions which, when executed by a computer, cause the computer to perform a method according to any of the aspects of the invention. The invention further relates to a computer readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform a method according to any of the aspects of the invention. The implementation of the invention by a computer program product and / or a computer readable storage medium has the advantage that existing computers can be easily adapted by software updates to operate as proposed by the invention.

[0055] A computer program product is for example a computer program or includes not only a computer program but other elements. The other elements may be hardware, e.g. a memory device on which the computer program is stored, a hardware key for using the computer program etc., and / or software, e.g. documentation or a software key for using the computer program etc. The computer program product further includes development materials, a run-time system, and / or databases or libraries. The computer program product may be distributed among several computer instances. The computer readable storage medium may be implemented as a non-permanent primary memory (e.g. random access memory) or as a permanent mass storage (e.g. hard disk, USB stick, SD card, solid state disk).

[0056] Utilizing large-scale language models capable of language understanding rather than parsing alone offers significant advantages compared to symbolic ontology-based methods that usually tackle negation, inversion, abbreviations, typos, and redundant information or are parsed "hard-coded" and therefore are very limited or cannot handle this information at all. In contrast, language understanding, information extraction, and content summarization are key capabilities of typical large-scale language models, making it possible to avoid these problems. Specifically, negation and inversion are an integral part of natural language and therefore richly represented, while query information given explicitly with knowledge of the world can be used to resolve omissions and typos. Redundant information is queried and summarized explicitly, such as in a form that itself references the original report in easily understandable bullet points.

[0057] As a result, the use of large-scale language models will revolutionize the way structured information is created from unstructured natural language data, becoming a major step towards the digital transformation of clinics, improving throughput, accelerating treatment, acting as a double check to avoid treatment errors, and ultimately reducing the workload of those involved. With a specific focus on scanning and post-processing workflow automation, as addressed by Tech Co-pilot, the benefits are similarly diverse, with key aspects including time savings with smart scanners, automatic pre-selection of correct workflows and steps, and triggering only relevant and specially tailored algorithms to generate relevant (AI-based) image analysis results, standardization, and improved consultation consistency. Taken together, it offers a strong potential for error reduction by enabling highly reproducible guidance-based workflow recommendations and acting as an automated "second eye" that automatically parses any available text-based source for relevant information.

[0058] In the context of the present invention, the phrase "based on" can in particular be understood to mean "particularly with..." In particular, the phrase that a first feature is calculated (or generated, determined, etc.) based on a second feature does not exclude the possibility that the first feature is calculated (or generated, determined, etc.) based on a third feature.

[0059] It should be noted that the disclosed method and the disclosed system are merely examples of preferred embodiments of the present invention, and the present invention can be modified by a person having ordinary skill in the art without departing from the scope of the present invention as defined by the claims.

[0060] The invention is explained below on the basis of exemplary embodiments and with reference to the accompanying drawings, in which the examples are schematic and highly simplified and are not necessarily drawn to scale. [Brief description of the drawings]

[0061] [Figure 1] 1 shows a flowchart of a computer-implemented method for providing medical imaging decision support data. [Diagram 2] 1 illustrates a data processing system. [Diagram 3] This shows a data flow diagram in an information system. [Figure 4] 1 illustrates a medical imaging workflow environment. [Diagram 5] This shows a structured reporting table. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0062] FIG. 1 illustrates a flowchart of a computer-implemented method for providing medical imaging decision support data, the method comprising: Receiving natural language data U including patient-specific clinical information (S1); generating structured information N comprising patient-specific clinical information in a structured format by applying a large-scale language model to the natural language data U (S2); Computing medical imaging decision support data based on the structured information N (S3); Providing medical imaging decision support data (S4).

[0063] Figure 2 shows a data processing system P arranged to provide medical imaging decision support data. The data processing system P comprises an interface unit PI and a computing unit PC. The interface unit PI is adapted to receive (S1) natural language data U comprising patient-specific clinical information. The computing unit PC is configured to generate (S2) structured information N comprising patient-specific clinical information in a structured format by applying a large-scale language model to the natural language data U. The computing unit PC is further configured to calculate medical imaging decision support data based on the structured information N (S3). The interface unit PI is further adapted to provide (S4) medical imaging decision support data.

[0064] FIG. 3 shows a data flow diagram through an information system M, in which a large-scale language model is implemented. The information system M can be, for example, a computer information system M. Thus, the information system M can smoothly handle almost all of the idiosyncrasies of natural language as used in daily clinical routine. The information system M includes a component 1 for converting unstructured information to structured information, a component 2 for medical imaging workflow automation and extension, a component 3 for on-the-fly learning, and a component 4 for explainable processing.

[0065] Each of these components utilizes LLM (Large Language Model) based processing for result inference. The large language model of the information system M includes a function L1 for generating (S2) structured information N based on the natural language data U, and a function L2 for computing medical imaging decision support data based on the structured information N. The large language model of the information system M further includes a function 4R for explanation and / or inference. The component 4 includes a user interface 4Q for interactive querying.

[0066] Component 2 comprises a module 2A for proposing, selecting and / or triggering medical imaging diagnostic workflow steps, options and / or parameters related to the acquisition, in particular related to a scan protocol, and / or related to the reconstruction, in particular related to a reconstruction algorithm. Component 2 further comprises a module 2B for proposing, selecting and / or triggering medical imaging diagnostic workflow steps, options and / or parameters related to image processing, in particular related to an image processing algorithm. For example, module 2B may comprise an interface for injecting structured information as prior information to be input to an image processing algorithm.

[0067] Reference data indicative of rules 3G and / or examples 3E related to the generation S2 of the structured information N and / or related to the calculation S3 of the medical imaging decision support data is received. In this case, the large-scale language model is adjusted based on the reference data. The component 3 is configured to adjust the large-scale language model based on the reference data. The reference data indicative of rules 3G and / or examples 3E, in particular in the form of generic medical guidelines, site-specific rules and / or preferences. The component 3 may further be configured to adjust the large-scale language model based on patient history data 5. These allow the information system M to learn and reason on the fly from context information and prior examples, as described above.

[0068] 4 shows a medical imaging workflow environment 8. An information system M can be used as a Tech Co-pilot 80 for a technician 83 in the medical imaging workflow environment 8. When there is a query by a physician 81, which includes an imaging request to resolve a clinical question (indication), a radiologist 82 determines the appropriate base protocol. An electronic medical record (EMR) and / or a radiology information system (RIS) can be used as a communication platform 91 for the physician 81 and the radiologist 82.

[0069] The appropriate base protocol is recorded, for example, in a DICOM modality worklist 92 and transferred to the medical imaging device 93 in the form of a request procedure 74 using only technical terms, for example "CT Thorax w / o contrast". Also, background information on the protocol selection 71, for example "Long nodule follow up CT", can be transferred directly from the doctor 81 and / or radiologist 82 to the technician 83. The reason for the consultation 73, for example "Pulmonary nodules follow-up", is provided to the Tech Co-pilot 80. Upon receiving additional information, such as a pre-diagnosis report 72 via a communication interface, such as FHIR, the Tech Co-pilot 80 infers and suggests relevant protocol refinements 75 for the technician 83, significantly reducing the workload for adjusting the scan, reconstruction and / or post-processing protocols. The medical imaging device 93 includes a data processing system P.

[0070] Table T of the structured report is shown in Figure 5. For example, ChatGPT's transformer-based large-scale language model can generate a structured report based on the following query: The query includes unstructured natural language data and a report template.

[0071] Extract the findings from the following radiological report and summarize the findings in the following template: Name: Age: Gender: Medical history: <bullet points> Findings: <a table with three columns - abnormality, location of corresponding icd 10 code> Summary: <bullet points> CLINICAL HISTORY: John Doe, a 68-year-old white male with recently diagnosed adenocarcinoma by sputum cytology. An abnormal chest radiograph shows right middle lobe infiltrate and collapse. Patient needs staging CT of chest with contrast. Right sided supraclavicular and lower anterior cervical adenopathy noted on physical exam. TECHNIQUE: Multiple transaxial images utilized in 10 mm sections were obtained through the chest. Intravenous contrast was administered. FINDINGS: There is a large 3 x 4 cm lymph node seen in the right supraclavicular region. There is a large right paratracheal lymph node best appreciated on image #16 which measures 3 x 2 cm. A subcarinal lymph node is enlarged also. It measures 6 x 2 cm. Multiple pulmonary nodules are seen along the posterior border of the visceral as well as parietal pleura. There is a pleural mass seen within the anterior sulcus of the right hemithorax as well as the right crus of the diaphragm. There is also a soft tissue density best appreciated on image #36 adjacent to the inferior aspect of the right lobe of the liver which most likely also represents metastatic deposit. The liver parenchyma is normal without evidence of any dominant masses. The right kidney demonstrates a solitary cyst in the mid pole of the right kidney. IMPRESSION: 1. 右侧发现超过二十个肺结节,包括壁层和各脏层胸膜内的肺结节,并伴有相邻实变,极有可能为肺肿瘤。 2. 上述广泛的纵隔淋巴结肿大。 3. 此时左肺未见病变。 4. 锁骨上淋巴结肿大。

[0072] The answers to the above queries are shown below.

[0073] 姓名:John Doe 年龄:68 性别:男 病史: - 近期通过痰细胞学检查诊断为腺癌 检查结果: 图5所示的表T。 总结: - 右侧存在超过二十个肺结节,包括壁层和脏层胸膜内的结节,并伴有相邻实变,可能提示肺肿瘤。 - Extensive mediastinal adenopathy with enlarged lymph nodes in the right supraclavicular region, right paratracheal region, and subcarinal region. - No lesion observed within the left lung at this time. - Supraclavicular adenopathy.

Claims

1. A method for providing medical imaging decision support data, which is performed by a computer, Receiving natural language data (U) containing patient-specific clinical information (S1), S2) A large-scale language model is applied to the natural language data (U) to generate structured information (N) that includes the patient-specific clinical information in a structured format. Based on the structured information (N), calculate the medical imaging decision support data (S3). A method comprising providing the aforementioned medical imaging decision support data (S4).

2. The method according to claim 1, wherein the large-scale language model includes a transformer network.

3. A query is received, and the query includes the natural language data (U) and additional query data. The method according to claim 1, wherein the structured information (N) is generated by applying the large-scale language model to the natural language data (U) and the additional query data.

4. The aforementioned additional query data includes template data, The method according to claim 3, wherein the template data shows the structured format.

5. Reference data is received, and the reference data shows rules (3G) and / or examples (3E) relating to generating the structured information (N) (S2) and / or calculating the medical imaging decision support data (S3), The method according to claim 1, wherein the large-scale language model is adjusted based on the reference data.

6. The large-scale language model calculates explanatory data, which provides explanations and / or reasons for generating the structured information (N) (S2) and / or calculating the medical imaging decision support data (S3). The method according to claim 1, wherein the explanatory data is provided.

7. A consultation request data is received, and the consultation request data indicates the type of medical imaging consultation. The method according to claim 1, wherein the medical imaging decision support data indicates the appropriateness of the type of medical imaging examination, taking into account the patient-specific clinical information.

8. The method according to claim 1, wherein medical imaging consultation workflow steps, options, and / or parameters are proposed, selected, and / or triggered based on the medical imaging decision support data.

9. The method according to claim 1, wherein the medical imaging decision support data shows the values ​​and / or changes in the values ​​of scan parameters of a scan protocol for a medical imaging examination of a patient by a medical imaging device.

10. The method according to claim 1, wherein the medical imaging decision support data indicates the values ​​and / or changes in the values ​​of the reconstruction parameters of a reconstruction algorithm for reconstructing a medical image based on medical imaging data.

11. The method according to claim 1, wherein the medical imaging decision support data indicates the values ​​and / or changes in the values ​​of image processing parameters of an image processing algorithm for processing medical images.

12. A data processing system (P) that provides medical imaging decision support data, It includes an interface unit (PI) and a computing unit (PC), The interface unit (PI) is configured to receive natural language data (U) containing patient-specific clinical information (S1), The computing unit (PC) is configured to generate structured information (N) containing patient-specific clinical information in a structured format by applying a large-scale language model to the natural language data (U) (S2). The computing unit (PC) is further configured to calculate the medical imaging decision support data (S3) based on the structured information (N), The interface unit (PI) is further configured to provide the medical imaging decision support data (S4), and is a data processing system (P).

13. A medical imaging apparatus (93) comprising the data processing system (P) described in claim 12.

14. A computer program that includes instructions, wherein when the instructions are executed by a computer, the computer causes the computer to perform the method described in any one of claims 1 to 11.

15. A computer-readable storage medium containing instructions, wherein when the instructions are executed by a computer, the computer causes the computer to perform the method described in any one of claims 1 to 11.

Images