Methods and systems for providing image acquisition information of a medical image

Abstract

Computer-implemented methods and systems are provided for providing image acquisition information of a medical image. The methods and systems implement a plurality of steps. One step is directed toward receiving the medical image. Another step is directed toward transforming the medical image (or data of the medical image) into the frequency domain to obtain a k-space image. Another step is directed toward determining the image acquisition information by applying a trained function to the k-space image (or data of the k-space image). Another step is directed toward providing the image acquisition information.

Description

[0001] Aspects of the invention relate to methods and systems for providing image acquisition information of medical images. In particular, aspects of the invention relate to systems and methods for deriving image acquisition information from pixel or voxel data of medical images. Furthermore, aspects of the invention relate to the use of the image acquisition information for processing the medical image, in particular for deriving a medical diagnosis.

[0002] Workflow automation remains a challenge in the processing of medical images to arrive at a potential patient diagnosis. Image processing begins with tasks that might seem trivial at first glance, such as selecting appropriate display settings or placing the image data on a reviewing physician's display screen. It continues with more subtle decisions, such as selecting the correct image processing tools, especially computer-aided detection (CAD) tools, selecting auxiliary data, routing data to the competent physician, selecting appropriate report templates to generate the final medical report, and so on.

[0003] Automating these and other steps is difficult because they depend critically on the underlying medical image. In other words, different images may require very different treatment in a reading and reporting workflow. The characteristics of the medical image can be summarized in the image acquisition information, which indicates, for example, the imaging modality and settings used or the body part displayed.

[0004] One option might be to read this information from the documentation available for the medical images. A problem with this approach is that there is often too little information documented for the image acquisition parameters to allow for an informed decision regarding automated processing steps. Furthermore, there is often enormous diversity regarding the image studies to be processed automatically. This can arise from different available acquisition and / or reconstruction technologies, different modalities, or even different acquisition preferences.

[0005] This is even more important because the type of image data and the image acquisition parameters, such as the CT imaging protocol or the MR sequence, are crucial for many tasks that can potentially be automated. This includes potential post-processing steps (e.g., convolution kernels), image enhancement steps (applying the correct window, e.g., bone), tool selection (e.g., lung CAD for image data related to the lung), hanging order, or pre-selection. The latter is particularly important in cases where longitudinal patient observation is required to determine the evolution of certain diseases, such as multiple sclerosis, or tumor follow-up.

[0006] Furthermore, in clinical reality, documentation of imaging parameters and imaged organs is not only rare, but often nonexistent or completely inaccurate. Due to time constraints in everyday clinical practice, there are often dummy entries in the protocols that cannot be used.

[0007] As a result, any simple automation based (only) on what is explicitly documented in a patient case often leads to inappropriate results. This not only consumes system resources but can also result in additional work on the user side—which is completely contrary to the original intention of automating the reading and report generation workflow in radiology.

[0008] To overcome these difficulties, it has been proposed to automatically extract relevant image acquisition information by applying a machine learning technique to medical images. For example, the reference by van der Voort et al., "DeepDicomSort: An Automatic Sorting Algorithm for Brain Magnetic Resonance Imaging Data," in Neuroinformatics, January 2021, 19(1):159-184, doi: 10.1007 / s12021-020-09475-7, proposes the use of a convolutional neural network (CNN) that automatically recognizes eight different brain magnetic resonance imaging (MRI) scan types based on visual appearance.

[0009] Although such methods generally work well for sorting tasks, the inventors found that their accuracy is insufficient for correctly predicting subtle differences, which are important for workflow automation. Furthermore, such methods do not generalize very well to unseen anatomy.

[0010] Accordingly, one object of embodiments of the invention is to improve the processing of medical images for obtaining a medical diagnosis. In particular, one object of embodiments of the invention is to improve the provision of image acquisition information of a medical image, particularly with regard to subsequent use in processing the medical image aimed at obtaining a medical diagnosis based on the medical image.

[0011] This object is achieved by a method for providing image acquisition information of a medical image, a system for providing image acquisition information of a medical image, corresponding computer program products, and corresponding computer-readable storage media according to the main claims. Alternative and / or preferred embodiments are the subject of the dependent claims.

[0012] In the following, the technical solution according to the present invention is described with reference to the claimed devices and with reference to the claimed methods. Features, advantages, or alternative embodiments described herein may equally be assigned to other claimed subject matter, and vice versa. In other words, claims addressing the inventive method may be enhanced by features described or claimed with reference to the devices. In this case, for example, functional features of the method are embodied by objective units or elements of the device.

[0013] The technical solution is described both with respect to methods and systems for providing image acquisition information of a medical image, and also with respect to methods and systems for providing a trained function designed to extract the image acquisition information from the medical image. Features and alternative forms of embodiments of data structures and / or functions for methods and systems for providing trained functions can be transferred to analogous data structures and / or functions for methods and systems for providing the image acquisition information. Analogous data structures can be identified, in particular, by using the prefix "training."Furthermore, the prediction functions used in methods and systems for providing information may in particular have been adapted and / or trained and / or provided by methods and systems for adapting trained functions.

[0014] According to one aspect, a computer-implemented method for providing image acquisition information of a medical image is provided. The method comprises a plurality of steps. One step is directed to receiving the medical image. Another step is directed to transforming the medical image (or data of the medical image) into the frequency domain to obtain a k-space image. Another step is directed to determining the image acquisition information by applying a trained function to the k-space image (or data of the k-space image). Another step is directed to providing the image acquisition information.

[0015] The medical image may be a two-dimensional image. The medical image dataset may also be a three-dimensional image. Furthermore, the medical image may be a four-dimensional image with three spatial dimensions and one temporal dimension. Furthermore, the medical image dataset may include a plurality of individual medical images.

[0016] The medical image comprises image data, for example, in the form of a two- or three-dimensional array of pixels or voxels. Such arrays of pixels or voxels may represent color, intensity, absorption, or other parameters as a function of two- or three-dimensional position and may be obtained, for example, by appropriate processing of measurement signals obtained with a medical imaging modality or image scanning device.

[0017] The medical image may depict a patient's body part. Accordingly, it may contain two- or three-dimensional image data of the patient's body part. The medical image may be representative of an image volume or a cross-section through the image volume. The patient's body part may be contained within the image volume.

[0018] A medical imaging modality is a system used to generate or produce medical image data. For example, a medical imaging modality can be a computed tomography (CT) system, a magnetic resonance imaging (MR) system, an angiography (or C-arm X-ray) system, a positron emission tomography (PET) system, or the like. Specifically, computed tomography is a widely used imaging technique and utilizes "hard" X-rays generated and detected by a spatially rotating instrument. The resulting attenuation data (also called raw data) is processed by analytical computer software, producing detailed images of the internal structure of the patient's body parts.The resulting sets of images are called CT scans and can represent numerous series of sequential images to present internal anatomical structures in cross-sections perpendicular to the axis of the human body. Magnetic resonance imaging (MRI), to provide another example, is an advanced medical imaging technique that utilizes the effect of a magnetic field on the movement of protons. In MRI machines, the detectors are antennas, and the signals are analyzed by a computer, producing detailed images of the internal structures in any section of the human body.

[0019] A medical image may comprise a plurality of image slices. The slices each depict a cross-sectional view of the image volume. The slices may comprise a two-dimensional array of pixels or voxels as image data. The arrangement of slices in the medical image dataset may be determined by the imaging modality or by any post-processing scheme employed. Furthermore, slices may be artificially defined in the imaging volume spanned by the medical image dataset. Optionally, this may occur as a function of the image data contained in the medical image dataset to optimally preprocess the medical image dataset for the subsequent diagnostic workflow.

[0020] Furthermore, a medical image may comprise a plurality of image series, each representing a single scan of the patient. Each series may refer to a captured 2D or 3D region.

[0021] The medical image may be stored in a standard image format, such as the Digital Imaging and Communications in Medicine (DICOM) format, and in a memory or computer storage system such as a picture archiving and communication system (PACS), a radiology information system (RIS), and the like. Whenever DICOM is mentioned herein, it should be understood that this refers to the Digital Imaging and Communications in Medicine (DICOM) standard, e.g., according to the DICOM PS3.1 2020c standard (or any later or earlier version of this standard).

[0022] According to some examples, the medical image is received by a picture archiving and communication system (PACS). The PACS may store the medical images in a post-processed version after acquisition. The post-processed version may refer to an easily viewable version in an image viewer, rather than raw data. In particular, the medical image may not include data in the frequency domain. In other words, the image data may include (only) "real" data.

[0023] The frequency domain can be viewed as the mathematical space of spatial frequencies. The frequency domain and real (Euclidean) space (in the spatial domain) are related by the Fourier transform. Specifically, data can be transformed from real space to the frequency domain by applying the 2D or 3D Fourier transform, and from the frequency domain to real space using the inverse or reciprocal Fourier transform.

[0024] Other words for frequency domain can be reciprocal space, frequency space, k-space, or Fourier space.

[0025] According to some examples, the medical image may be regarded as the real image, and the k-space image may be regarded as the frequency domain image (of the medical image) or as the medical image transformed into the frequency domain.

[0026] The k-space image can have the same dimensions, that is, the same number of rows and / or columns, as the medical image. The entries in the rows and / or columns can reflect the occurrence of spatial frequencies in the medical image. The entries can have real and imaginary components. It should be noted that k-space can have arbitrary sizes. However, it is not advisable to make k-space very large, for example, beyond the highest frequencies that scanners can measure, as this would not increase the information content.

[0027] According to some examples, the step of transforming the medical image (or data of the medical image) into the frequency domain to obtain a k-space image may comprise applying a Fourier transform, in particular a discrete Fourier transform, to the medical image. This may comprise decomposing a sequence of values, i.e., the pixels / voxels of the medical image, into components of different frequencies.

[0028] The trained function is configured (or has been trained) to extract image acquisition information related to a medical image from a corresponding k-space image of the medical image. Applying the trained function may comprise providing the trained function and / or inputting the respective data (i.e., data of the k-space image and / or data of the medical image) to the trained function.

[0029] In general, a trained function simulates cognitive functions that people associate with other human ways of thinking. Specifically, by training based on training data, the machine learning function can adapt to new circumstances, detecting and extrapolating patterns, such as different scanners and manufacturers. Other terms for machine learning function can be machine learning function, trained machine learning model, trained mapping specification, mapping specification with trained parameters, function with trained parameters, artificial intelligence-based algorithm, or machine learning algorithm.

[0030] In general, the parameters of a machine learning function can be adjusted through training. In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of the machine learning function can be adjusted iteratively through multiple training steps.

[0031] In particular, a trained function may comprise a neural network, a support vector machine, a decision tree, and / or a Bayesian network, and / or the trained function may be based on k-means clustering, Q-learning, genetic algorithms, and / or association rules. In particular, a neural network may be a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, a neural network may be an adversarial network, a deep adversarial network, and / or a generative adversarial network.

[0032] The trained function may generally be configured to determine image acquisition information based on k-space images obtained by transforming the medical image into the frequency domain. For example, the trained function may be configured to extract one or more features from the k-space and image and map / classify these features into a feature space associated with different image acquisition parameters to determine which acquisition information the k-space image indicates. Thus, the trained function may comprise a feature extractor and a classifier.In particular, the feature extractor and the classifier may be implemented as a neural network, in particular a convolutional neural network, where some network layers are trained to extract features and other network layers are trained to provide a classification according to the most likely image acquisition information.

[0033] According to some examples, the image acquisition information relates to an image acquisition procedure using a medical imaging modality, wherein the medical image was acquired using this image acquisition procedure. According to some examples, the image acquisition information includes a modality, an anatomy, and / or a procedure based on which the medical image was acquired. According to some examples, the image acquisition information includes one or more image acquisition parameters with or based on which the medical image was acquired, such as a type of medical imaging modality used, settings of the medical imaging modality, or a type of medical image data.

[0034] The inventors recognized that by transforming medical image data into the frequency domain, the image acquisition information becomes more accessible for automated feature extraction and classification of a trained function. This allows the image acquisition information to be obtained more reliably, particularly in cases where this information is not documented in the available patient medical record and / or the medical image metadata. In turn, this enables more efficient automation of the process aimed at providing a medical diagnosis based on the medical image.

[0035] According to some examples, the step of transforming the medical image (or data of the medical image) into the frequency domain to obtain a k-space image includes applying a fast Fourier transform (FFT) to the medical image.

[0036] An FFT is an algorithm that computes the discrete Fourier transform of original data or its inverse. An FFT computes such transforms using matrix defactorization. This allows for a computationally efficient procedure.

[0037] In principle, any suitable trained features can be used to obtain the image acquisition information from the k-space image.

[0038] However, according to some examples, the trained function comprises at least one of the following: a convolutional neural network, a transformer network, and / or a focal network.

[0039] A convolutional neural network is a neural network that uses a convolution operation instead of general matrix multiplication in at least one of its layers (called a "convolutional layer"). Specifically, a convolutional layer performs a dot product of one or more convolutional kernels on the input data / image of the convolutional layer, where the entries of the one or more convolutional kernels are the parameters or weights that are adjusted through training. In particular, the Frobenius dot product and the ReLU activation function can be used. A convolutional neural network may include additional layers, such as pooling layers, fully connected layers, and normalization layers.

[0040] For an example of the general utility of convolutional neural networks for deriving image acquisition parameters, reference is made to van der Voort et al., "DeepDicomSort: An Automatic Sorting Algorithm for Brain Magnetic Resonance Imaging Data," in Neuroinformatics, January 2021, 19(1):159-184, doi: 10.1007 / s12021-020-09475-7, the contents of which are hereby incorporated by reference in their entirety. While van der Voort et al. applies it to real-world image data, the inventors have recognized that the architecture can, in principle, also be applied to k-space images.

[0041] By using convolutional neural networks, input k-space images can be processed very efficiently, as a convolution operation based on different kernels can extract different image features. By adjusting the weights of the convolution kernel, relevant frequency patterns can be easily found during training. Furthermore, based on the weight distribution in the convolution kernels, fewer parameters need to be trained, preventing overfitting during the training phase and allowing for faster training or more layers in the network, thus improving network performance.

[0042] A transformer network is a neural network architecture that generally includes an encoder, a decoder, or both an encoder and a decoder.

[0043] In some cases, the encoders and / or decoders consist of multiple corresponding encoding and decoding layers. Each encoding and decoding layer contains an attention mechanism. The attention mechanism, sometimes referred to as self-attention, relates data elements (such as words or pixels) within a sequence of data elements to other data elements in the sequence.

[0044] In particular, the encoder can be configured to transform the input k-space image into a numerical representation. The numerical representation can comprise one vector per input token (e.g., per image patch). The encoder can be configured to implement an attention mechanism such that each patch is influenced by the other patches in the input. In particular, the encoder can be configured so that the representations resolve the desired output, i.e., the image acquisition information of the trained function.

[0045] In particular, the decoder can be designed to transform an input into a sequence of output tokens. In particular, the decoder can be designed to implement a masked self-attention mechanism, so that each vector of a token is only influenced by the other tokens on one side of a sequence. Furthermore, the decoder can be autoregressive, in the sense that the intermediate results (such as a previously predicted sequence of tokens) are fed back.

[0046] According to some examples, the output of the encoder is input to the decoder.

[0047] Furthermore, the transformer network may comprise a classification module or a classification unit configured to map the output of the encoder or the decoder to a set of learned outputs in the form of the image acquisition information.

[0048] In particular, the transformer network can be implemented as a vision transformer. The vision transformer can be configured to decompose input k-space images into frequency patches and tokenize them (extracting representation vectors) before applying the tokens to a standard transformer architecture. The vision transformer can include an attention mechanism configured to repeatedly transform representation vectors of image patches to progressively incorporate semantic relationships between image patches within a k-space image.

[0049] According to some examples, the vision transformer can be obtained by training a masked autoencoder. A masked autoencoder comprises two vision transformers concatenated. The first takes k-space image patches with position encoding in frequency domain and outputs vectors representing each patch. The second takes vectors with frequency-position encodings and again outputs k-space image patches. During training, both vision transformers are used. A k-space image is sliced ​​into patches. The second vision transformer takes the encoded vectors and outputs a reconstruction of the full k-space image. During use, the first vision transformer can be used as an encoder, and / or the second vision transformer can be used as a generative AI function.

[0050] For an overview of transformer networks, see Vaswani et al., “Attention Is All You Need,” in arXiv: 1706.03762, June 12, 2017, the contents of which are incorporated herein by reference.

[0051] One advantage of transformer networks is that, thanks to the attention mechanism, they can efficiently handle long-range dependencies in the input data. Furthermore, the encoders used in transformer networks can process data in parallel, which saves computational resources during inference. Furthermore, thanks to autoregression, the decoders of transformer networks can iteratively generate a sequence of output tokens with high confidence.

[0052] FocalNet is short for focal modulation network. A FocalNet is a neural network that uses a focusing mechanism to enable the model's interaction with the input, in this case, the k-space image. Specifically, FocalNets use lightweight element-wise multiplication as a focusing operator to view or interact with the input using the proposed modulator. The modulator is computed using a two-step focal aggregation procedure: focal contextualization to extract contexts from local to global regions at different granularity levels, and gated aggregation to condense all context features at different granularity levels into the modulator. For an overview of FocalNets, see Yang et al., "Focal Modulation Networks," in arXiv:2203.11926, the contents of which are incorporated herein by reference in their entirety.

[0053] The inventors have found that FocalNet can perform better than attention-based neural networks when processing k-space images.

[0054] According to some examples, the medical image is a 3D medical image, and the method further comprises obtaining an image slice of the medical image, wherein the k-space image is generated by transforming the image slice into the frequency domain.

[0055] Obtaining the image slice involves selecting the image slice from a plurality of slices contained in the medical image or defining the image slice in the 3D medical image.

[0056] According to some examples, obtaining the image slice may include determining a fitness measure (or quality score) of a plurality of candidate slices of the medical image, wherein the fitness measure indicates a suitability of the slice for the following steps. According to some examples, the fitness measure may be based on a position of the slice in the medical image (where edge slices may have lower fitness than more central slices) and / or an image content of the slice (where image slices with imaging artifacts may have lower fitness).

[0057] According to some examples, the fitness measure may be determined by a further trained function configured to obtain the image slice. The further trained function may be based on a convolutional neural network. Furthermore, the further trained function may be part of the trained function. The further trained function may be trained by processing a plurality of slices from the medical image with the trained function and comparing the result to a ground truth for the image acquisition information. The result of the comparison may be fed back to the further trained function to optimize the slice obtaining process.

[0058] By obtaining the image slice, information from the medical image can be preselected to best represent the medical image. This can increase the quality of the processing results and the efficiency of the procedure.

[0059] According to one aspect, the image acquisition information comprises an image acquisition parameter and / or information about a body part depicted in the medical image.

[0060] An image acquisition parameter may include the type and / or make of imaging modality used and / or control parameter settings of the imaging modality used during acquisition of the medical image. To provide an example, image acquisition parameters may include the following information: Type Chest CT Scan, Bolus Agent: xyz, Modality: Siemens Healthineers CT Scanner, Model Number: 12345, Kilovoltage Peak: xxx, Milliampereseconds: yyy. The body part information may include an indication of the imaged body part and / or the body compartments contained therein and / or an indication of findings contained in the body part(s). To provide an example, body part information may include: Chest area showing the lungs, rib cage, spine of the patient with a pulmonary nodule in the upper left lobe.

[0061] The image acquisition parameters and / or body part information provide valuable information that determines the subsequent processing steps in the diagnostic image processing workflow. This makes it easier to trigger these steps automatically without user input.

[0062] According to one aspect, the medical image was acquired with a magnetic resonance acquisition procedure using a medical magnetic resonance imaging modality, and the image acquisition parameter relates to the image weighting and / or the magnetic resonance sequence used in the acquisition procedure. In other words, determining the image acquisition parameter comprises classifying the medical image according to the image weighting and / or the magnetic resonance sequence used.

[0063] Image weighting can indicate which relaxation effect the magnetic resonance imaging was focused on. Each tissue returns to its steady state after excitation through independent relaxation processes of T1 (spin-lattice; that is, magnetization in the same direction as the static magnetic field) and T2 (spin-spin; perpendicular to the static magnetic field). To generate a T1-weighted image, the magnetization can recover before the MR signal is measured by changing the repetition time. This image weighting is useful for assessing the cerebral cortex, identifying adipose tissue, characterizing focal liver lesions, and generally preserving morphological information, as well as for post-contrast imaging. To generate a T2-weighted image, the magnetization can decay before the MR signal is measured by changing the echo time.This image weighting is useful for detecting edema and inflammation, revealing lesions and abnormalities.

[0064] More subtle variations exist within the T1 / T2 weightings. T2* weighting is based on a distribution of resonance frequencies around the ideal. Over time, this distribution can lead to a scattering of the distribution of magnetic spin vectors. This results in dephasing. For molecules that are not moving, the deviation from the ideal relaxation is consistent over time, and the signal can be recovered by performing a spin-echo experiment. T2*-weighted sequences are used to detect deoxygenated hemoglobin, methemoglobin, or hemosiderin in lesions and tissues. Diseases with such patterns include intracranial hemorrhage, arteriovenous malformation, cavernoma, hemorrhage within a tumor, punctate hemorrhage in diffuse axonal injury, superficial siderosis, thrombosed aneurysm, phleboliths in vascular lesions, and some forms of calcification.

[0065] The magnetic resonance sequence can refer to the sequence of pulse sequences and pulsed field gradients to which a sample is exposed. By varying the pulse sequence parameters, different contrasts between tissues can be generated based on their relaxation properties. In other words, different image weightings can be generated. Furthermore, different weightings are possible for different sequences. For example, BLADE (see below) can be combined with T1, T2, or STIR weighting.

[0066] The inventors recognized that the step of transforming real medical images from magnetic resonance sources provides valuable insight into the image acquisition parameters. This is because, in magnetic resonance imaging, complex values ​​in k-space are sampled during measurement in a pre-determined scheme controlled by a pulse sequence—i.e., a precisely timed sequence of radiofrequency and gradient pulses. During acquisition, k-space often refers to the temporary image space, usually a matrix, in which data from digitized magnetic resonance signals are stored during data acquisition. At the end of the scan, the data is mathematically processed to generate a final medical image. Thus, the matrix contains raw data prior to reconstruction. During the medical image reading phase, the raw data is often no longer available, as typically only the reconstructed image is stored (e.g., in a PACS).Accordingly, the proposed process step of transforming the medical image (back) to k-space can reveal or focus on information more directly related to the image acquisition process. This can lead to better predictions of image acquisition information and enable easier automation of the reading and reporting workflow.

[0067] In particular, the magnetic resonance sequence and / or image weighting can provide important clues about how the medical image needs to be processed for image reading and what type of CAD tools should be applied to the medical image. This is because a T2*-weighted image or corresponding sequence may indicate a very different underlying clinical question and suggest very different subsequent processing steps than a T1-weighted image.

[0068] It should be noted that the transformation into k-space is also advantageous with other image acquisition techniques, such as computed tomography, since the method can reveal an additional layer of information in these cases as well.

[0069] According to some examples, the image weighting includes at least or distinguishes between T1, T2, T2*, PD (Proton Density), SSFP (Steady-State Free Precession), SWI (Susceptibility-weighted), STIR (Short tau inversion recovery), IR (Inversion recovery), DIR (Double inversion recovery), DWI (Diffusion), PWI (Perfusion), SWI (Susceptibility-Weighted Imaging), BOLD (Blood-oxygen-level dependent) and / or ToF (Time of Flight) weighting.

[0070] According to some examples, the imaging sequence includes at least or distinguishes between a spin echo sequence, a gradient echo sequence, an inversion recovery sequence, an MR angiography sequence, a saturation recovery sequence, an echo planar sequence, a spiral pulse sequence, an in-and-out-of-phase imaging sequence, and subsequences and combinations of the foregoing.

[0071] According to some examples, the imaging sequence distinguishes between at least two different spin-echo sequences. In other words, determining the image acquisition information comprises classifying the medical image according to at least two different spin-echo sequences.

[0072] According to other examples, the imaging sequence distinguishes between BLADE and HASTE.

[0073] BLADE is a proprietary sequence from Siemens Healthineers AG that reduces sensitivity to motion in magnetic resonance scanning. It is a technique that incorporates a radial k-space trajectory, reducing motion artifacts and helping to visualize even the smallest lesions.

[0074] HASTE is a spin-echo sequence trademarked by Siemens Healthineers AG. It is a single-shot technique, meaning that data from each k-space row is acquired after a single 90° excitation pulse.

[0075] The distinction between spin-echo techniques and / or between, for example, BLADE and HASTE is important for subsequent processing steps, and especially for selecting the correct hanging protocol. At the same time, the differences between real image data from different spin-echo sequences are subtle, making this process difficult. The same applies to the distinction between BLADE and HASTE. In this regard, the inventors recognized that such differences can be more easily resolved in k-space.

[0076] According to one aspect, the trained function is additionally applied to the medical image in the determination step.

[0077] In other words, not only the k-space image data but also the real image data are input into the trained function. Conversely, the trained function can also be trained on the real image data, so that the prediction of the image acquisition information can also be based on this data.

[0078] This has the advantage that features that are more easily derived from the real image, such as the body parts and compartments shown, can be more easily inferred from the data. Furthermore, it enables the use of existing classifiers designed to be applied to real image data to establish the trained function. This can include segmentation algorithms designed to detect and segment a patient's organs, or detection and / or classification algorithms designed to detect and classify image patterns such as lesions or other anomalies.

[0079] According to one aspect, the method further comprises generating a composite image based on the medical image and the k-space image, wherein the step of determining applies the trained function to the composite image.

[0080] According to some examples, the k-space image has the same size and / or dimensions as the medical image. In particular, it may have as many individual values ​​as the medical image has pixels / voxels. Furthermore, there may be a k-space value for each pixel / voxel of the medical image. According to some examples, the k-space values ​​may be overlaid over or added to the pixel or voxel values ​​of the medical image. According to other examples, the k-space values ​​may be added to the medical image in the form of an additional layer or dimension, such that each pixel or voxel of the medical image has an additional dimension provided by the corresponding k-space value. According to some examples, the k-space image data is added to the medical image as one or more additional image channels.

[0081] Creating the concatenated image can provide the advantage of intrinsic registration, which can enable the identification of relationships between image space and k-space. In turn, this can lead to more meaningful predictions and better results in subsequent workflow automation.

[0082] According to one aspect, a computer-implemented method for displaying a representation of a medical image is provided. The method comprises a plurality of steps. One step is directed to receiving the medical image from a database. Another step is directed to determining image acquisition information of the medical image according to any of the aspects and examples described herein. Another step is directed to generating a representation of the medical image for display in a user interface based on the image acquisition information. Another step is directed to displaying the representation in the user interface.

[0083] The representation can be generated by processing the medical image, whereby the processing depends on the image acquisition parameters.

[0084] The representation may comprise one or more two-dimensional representation images rendered from the medical image. The representation images may comprise a plurality of image pixels. In particular, the representation images may be two-dimensional renderings of the medical image or different views of the medical image. Two-dimensional renderings may generally be based on known rendering procedures, including ray casting, ray tracing, texture rendering, or the like. The views and rendering may depend on the image acquisition information. For example, the image acquisition information may suggest a specific rendering or preprocessing and / or a specific view.

[0085] Furthermore, the representation may comprise a graphical user interface containing the representation images at a predefined location. In particular, the graphical user interface may be specifically designed to derive a specific medical diagnosis based on the medical image. Using the graphical user interface, the user can inspect the medical image, take measurements, and record a medical diagnosis (e.g., in the form of a medical report).

[0086] By automatically generating the representation based on the image acquisition parameters, views likely required based on the image acquisition information are automatically generated and presented to the user. This relieves the user of the routine but tedious task of setting up the representation themselves for further diagnosis of the medical image.

[0087] According to one aspect, the step of generating comprises determining a display setting based on the image acquisition information and applying the selected display setting to generate the representation, wherein the display setting is selected from: a contrast setting, a brightness, an intensity windowing, an image enhancement, a lookup table, a view plane, a segmentation mask, a zoom level or panning and / or a volumetric rendering parameter.

[0088] By automatically determining display settings, the user is automatically provided with appropriate parameters and does not have to specify them himself.

[0089] According to one aspect, the representation comprises a volumetric rendering of the medical image, which is generated in particular with a path tracing or ray casting based rendering process, and the display setting comprises a volumetric rendering parameter for generating the volumetric rendering.

[0090] In ray casting, simulated rays emanating from the eye of an imaginary observer are transmitted through the body or object under investigation (see Levoy: "Display of Surfaces from Volume Data", IEEE Computer Graphics and Applications, Volume 8, No. 3, May 1988, pages 29-37). RGBA values ​​for sample points from the voxels are determined along the rays and combined to form pixels for a two-dimensional image using alpha compositing or alpha blending. Here, the letters R, G, and B in the term RGBA represent the color components red, green, and blue that make up the color contribution of the corresponding sample point. A represents the alpha value, which is a measure of the transparency at the sample point. The respective transparency is used when superimposing RGB values ​​at sample points to form the pixel.Lighting effects are usually accounted for using a lighting model within the scope of a technique called "shading." Another technique for volume rendering is the so-called path tracing technique (cf. Kajiya: "The rendering equation," ACM SIGGRAPH Computer Graphics, Volume 20, No. 4, August 1986, pages 143-150). Here, a large number of simulated rays are fired into the volume data per visualization pixel. The simulated rays then interact with the volume, i.e., are reflected, refracted, or absorbed, with at least one random ray being generated each time (except in the case of absorption). Each simulated ray thus finds its path through the volume data. The more virtual rays used per visualization pixel, the better the image. In particular, the processes and techniques described in EP 3 178 068 B1 can be used here.The contents of EP 3 178 068 B1 are incorporated herein by reference in their entirety.

[0091] Accordingly, the display settings can specify parameters for the path and / or raycasting process, such as zoom levels, viewing angles, transfer functions, texture values, number of rays, transparency levels, scene lighting, and so on.

[0092] On the one hand, such methods enable the creation of particularly realistic visualizations. This provides the human recipient with an instructive picture of the imaging examination and its results. On the other hand, since volumetric image rendering is triggered automatically, the user does not need to be involved, thus saving the user from having to familiarize themselves with the intricacies of a volumetric rendering pipeline (which can be complex).

[0093] According to one aspect, the method further comprises selecting, based on the image acquisition information, a hanging protocol including a set of rules for displaying one or more representations of a medical image in a user interface, wherein the step of displaying displays the representation based on the hanging protocol.

[0094] Current reading and reporting systems use common techniques known as "hanging protocols" to format the display or layout of medical images or sections of medical images. Hanging protocols allow a user to set display environments specifically according to modality, anatomy, and procedure. Hanging protocols present a user, such as a radiologist, with one or more perspectives or views (e.g., in the form of the aforementioned representations) of the medical image. Representations can be grouped and located in a graphical user interface. Additionally, hanging protocols can include rules or instructions for obtaining additional information, such as comparative medical images acquired before or after the medical image, or for applying specific image analysis tools.

[0095] According to some examples, the hanging protocol may be selected from a plurality of predefined hanging protocols. The predefined hanging protocols may be configured according to different use cases, and the selecting step may include identifying a use case based on the image acquisition information and selecting the hanging protocol according to the identified use case.

[0096] By selecting the appropriate hanging protocol, the user is automatically provided with a user interface that is specifically tailored to the image data and diagnostic task. This not only reduces the user's workload but also automates image processing to provide a medical diagnosis.

[0097] In one aspect, the method further comprises retrieving, from the database and based on the image acquisition information, a comparative medical image, processing the comparative medical image to generate a comparative representation for display on the user interface, and displaying the comparative representation on the user interface.

[0098] According to some examples, the comparative medical image may include the same or similar image acquisition information as the medical image. According to some examples, the comparative medical image may be processed in the same way as the medical image. For example, the comparative medical images may be based on the same magnetic resonance sequence and / or image weighting as the medical image. This allows the comparative medical image to be more easily compared to the medical image, and / or the same measurements may be taken.

[0099] The comparative medical image may refer to the same patient as the medical image. The comparative medical image may refer to a different patient than the medical image. In particular, the comparative medical image may have a degree of similarity to the medical image. The comparative medical image may be obtained from a database of comparative medical images. Furthermore, the comparative medical image may be obtained from an electronic medical textbook. The comparative medical image may be associated with a verified medical diagnosis.

[0100] By offering a user a comparative medical image, the user can be provided with additional information to derive a medical diagnosis.

[0101] According to some examples, the medical image of the patient was acquired at a first time point, and the comparative medical image was acquired of the patient at a second time point that is different from the first time point. In other words, a previous or subsequent medical image of the patient may be automatically retrieved that includes comparable image acquisition information. In particular, this may mean that the medical image shows a body part of the patient at the first time point, and the comparative medical image shows the body part at a second time point. This allows the user to more easily determine an evolution of the patient's health status and thus make a better diagnosis.In this respect, since the comparative medical image is automatically retrieved based on the image acquisition information, the user is spared the time-consuming task of, for example, searching the database for appropriate priors.

[0102] According to one aspect, the method further comprises selecting, based on the image acquisition information, an image processing tool configured to provide an image processing result, applying the selected image processing tool to generate the image processing result, and displaying the image processing result in the user interface.

[0103] According to some examples, the image processing tools can also be applied to any comparative medical images.

[0104] The image processing tool can be selected from a variety of available image processing tools. The image processing tools can be specific to certain use cases / image acquisition information. For example, the variety of image processing tools can include tools specific to a particular modality, anatomy, and / or procedure. In particular, there could be specific image processing tools for certain magnetic resonance sequences.

[0105] Furthermore, the image processing tool may be selected from one or more segmentation tools, one or more detection / classification tools, one or more change detection tools, etc. Accordingly, the image detection result may include: a detection result of a medical finding in the medical image, a classification of a medical finding in the medical image, a segmentation of the medical image, and / or a change detected in the medical image.

[0106] By automatically identifying and applying the image processing tool, results can be generated automatically. The user is provided with guidance for obtaining a medical diagnosis without having to search the library of available tools for appropriate ones.

[0107] According to some examples, the method further comprises selecting, based on the image acquisition information, a report template for generating a medical report corresponding to the medical image and providing the report template via the user interface.

[0108] A report template can be a preconfigured data structure, a building block, or a module on the basis of which a structured medical report can be generated. A medical report can be generated based on at least one report template.

[0109] Selecting the report template may involve selecting from a variety of report templates. Each report template may be specific to a particular piece of imaging information. For example, a particular report template may be associated with a brain magnetic resonance scan, while another report template may be associated with a chest CT scan.

[0110] Each report template can specify one or more data fields that must be addressed or filled in to complete the medical report. Furthermore, a report template can include one or more pull-down menus with items that a user can select. Thus, a report template can also be viewed as an input form or screen that structures the information to be provided for a given diagnostic task.

[0111] According to some examples, the method may further include pre-populating the report template based on the image acquisition information and / or any other image processing results, e.g., as obtained by applying image processing tools.

[0112] By retrieving appropriate report templates, appropriate template data structures are automatically provided to the user. In return, the user is relieved of the burden of having to search for the correct template data structure in potentially massive databases.

[0113] According to one aspect, the method further comprises determining, based on the image acquisition information, a designated recipient of the medical image in a medical information system comprising a plurality of possible recipients, wherein the possible recipients comprise a database, an image processing server, a user's worklist, a user's reading and reporting workstation, and / or a data exchange server for exchanging data with outside the healthcare information system. Optionally, the method may include forwarding / transmitting the medical image to the designated recipient.

[0114] By identifying designated recipients, the medical image can be automatically routed to the correct location in the healthcare information system. For example, if the image acquisition information indicates that an image of the brain was acquired using a specific MR pulse sequence that addresses a specific clinical question, the image can be automatically routed to a clinical expert for that type of study.

[0115] According to one aspect, a computer-implemented method for providing a trained function for deriving image acquisition information from a medical image is provided. The method comprises a plurality of steps. A first step is directed to providing a training dataset comprising a medical image and verified image acquisition information. Another step is directed to transforming the medical image (or data of the medical image) into the frequency domain to obtain a k-space image. Another step is directed to applying the trained function to the k-space image to obtain intermediate image acquisition information. Another step is directed to comparing the intermediate image acquisition information with the verified image acquisition information.A further step is aimed at adapting the trained function based on the comparison step. Another step is aimed at providing the adapted trained function.

[0116] According to a further aspect, a computer-implemented method is provided for providing a trained function for deriving image acquisition information from a medical image acquired using a magnetic resonance acquisition procedure. The method comprises a plurality of steps. A first step is directed to providing a training dataset comprising a raw magnetic resonance image in the frequency domain and verified image acquisition information, wherein the medical image is to be reconstructed from the raw image. A further step is directed to applying the trained function to the raw image to obtain intermediate image acquisition information. A further step is directed to comparing the intermediate image acquisition information with the verified image acquisition information.A further step is aimed at adapting the trained function based on the comparison step. Another step is aimed at providing the adapted trained function.

[0117] The latter aspect has the advantage that the training data can be obtained directly from the magnetic resonance machine without requiring image reconstruction or transformation steps.

[0118] According to one aspect, a system for providing image acquisition information of a medical image is provided. The system comprises an interface unit and a computing unit. The computing unit is configured to receive the medical image via the interface unit, transform the medical image into the frequency domain to obtain a k-space image, determine the image acquisition information by applying a trained function to the k-space image, and provide the image acquisition information via the interface unit.

[0119] According to a further aspect, a system for providing control signals for displaying a representation of a medical image is provided. The system comprises an interface unit and a computing unit. The computing unit is configured to receive the medical image from a database via the interface unit, transform the medical image into the frequency domain to obtain a k-space image, determine the image acquisition information by applying a trained function to the k-space image, generate control signals for controlling a user interface to display a representation of the medical image, wherein the representation is generated according to (based on) the image acquisition information, and provide the control signals to the user interface via the interface unit.

[0120] The computing unit(s) may be implemented as a data processing system or as part of a data processing system. Such a data processing system may comprise, for example, a cloud computing system, a computer network, a computer, a tablet computer, a smartphone, and / or the like. The computing unit may comprise hardware and / or software. The hardware may comprise, for example, one or more processors, one or more memories, and combinations thereof. The one or more memories may store instructions for carrying out the method steps according to the invention. The hardware may be configurable by the software and / or operable by the software. In general, all units, subunits, or modules may at least temporarily communicate with one another, e.g., via a network connection or corresponding interfaces. Consequently, individual units may be located remote from one another.

[0121] The interface unit may include an interface for data exchange with a local server or a central web server via an Internet connection for receiving the medical images.

[0122] The user interface may be configured to provide an interface for one or more users of the system, e.g., by displaying the result of the processing of the computing unit to the user (e.g., on a graphical user interface) or by allowing the user to provide inputs to arrive at a medical diagnosis.

[0123] In other aspects, the invention further relates to an integrated data management system (or healthcare information system) comprising the above system and an image archiving system configured to capture, store, and / or forward medical images. The interface unit may be configured to receive the medical image dataset from the image archiving system. According to some examples, the image archiving system may be implemented as cloud storage or as local or distributed storage, e.g., as a PACS (Picture Archiving and Communication System).

[0124] According to other aspects, the systems are configured to implement the method according to the invention in its various aspects for providing a medical candidate finding. The advantages described in connection with the method aspects can also be realized by the correspondingly configured system components.

[0125] According to another aspect, the present invention is directed to a computer program product comprising program elements that cause a computing unit of systems described herein to perform the steps according to one or more of the above method aspects and examples as described herein when the program elements are loaded into a memory of the computing unit.

[0126] In another aspect, the present invention is directed to a computer-readable medium storing program elements readable and executable by a computing device of systems described herein to perform the steps according to one or more method aspects and examples as described herein when the program elements are executed by the computing device.

[0127] The realization of the invention by means of a computer program product and / or a computer-readable medium has the advantage that already existing provisioning systems can be easily adapted by software updates in order to operate as proposed by the invention.

[0128] The computer program product may, for example, be a computer program or comprise another element in addition to the computer program as such. This other element may be hardware, e.g., a storage device in which the computer program is stored, a hardware key for using the computer program, and the like, and / or software, e.g., documentation or a software key for using the computer program. The computer program product may further comprise development material, a runtime system, and / or databases or libraries. The computer program product may be distributed among multiple computer instances.

[0129] Both the characteristics, features, and advantages of the invention described above, as well as the manner in which they are achieved, will become clearer and more understandable in light of the following description of embodiments, which are described in detail with reference to the figures. This following description does not limit the invention to the embodiments included. The same components, parts, or steps may be designated by the same reference numerals in different figures. In general, the figures are not drawn to scale. In the following: Fig. 1 schematically illustrates a system for providing image acquisition information and / or a representation of a medical image according to an embodiment, Fig. 2 schematically illustrates a method for providing image acquisition information of a medical image according to an embodiment, Fig. 3 schematically illustrates data flows in a method for providing image acquisition information of a medical image according to an embodiment, Fig. 4 schematically illustrates a method for providing a representation of a medical image according to an embodiment, Fig. 5 schematically illustrates data flows in a method for providing a representation of a medical image according to an embodiment, Fig. 6 schematically illustrates a graphical user interface for displaying a representation of a medical image according to an embodiment, and Fig. 7 schematically illustrates a trained function for determining image acquisition information of a medical image according to an embodiment.

[0130] Fig. 1 illustrates a health information system 1 for providing image acquisition information IAI and / or a representation RE of a medical image MI. In this regard, the health information system 1 is configured to perform the methods according to one or more embodiments, such as further described with reference to Fig. 2 to 6.

[0131] A user U of the health information system 1 may, according to some examples, generally refer to a healthcare professional, such as a physician, clinician, technician, radiologist, etc.

[0132] The health information system 1 may include a user interface 10 and a processing system 20. Furthermore, the system 1 may include or be connected to a database DB, which is generally configured for storing and / or forwarding medical images (MI) and supplementary (non-image) information. The components of the health information system 1 may also be referred to as "receivers" because they can receive data such as medical images (MI) and information derived therefrom.

[0133] The database DB may include one or more storage devices for medical images MI, which may be implemented in the form of one or more cloud storage, local or distributed storage modules, such as a PACS (Picture Archiving and Communication System).

[0134] According to some examples, the health information system may include one or more medical imaging modalities (not shown) for acquiring medical images MI, such as a computed tomography system, a magnetic resonance system, an angiography (or C-arm x-ray) system, a positron emission tomography system, a mammography system, an x-ray system, or the like.

[0135] The medical images MI may be three-dimensional image datasets acquired, for example, using an x-ray system, a computed tomography system, a magnetic resonance imaging system, or other systems. The image information may be encoded in a three-dimensional array of m by n by p voxels. The medical images MI may comprise a plurality of image slices stacked in a stacking direction to span the image volume covered by the medical image MI.

[0136] Furthermore, the medical images MI may comprise two-dimensional medical image data, wherein the image information is encoded in an array of m by n pixels. According to some examples, these two-dimensional medical images may have been extracted from a three-dimensional medical image MI.

[0137] An ensemble of voxels or pixels can subsequently be designated as the image data of the respective medical image (MI). In general, any type of imaging modality and scanner can be used to acquire such image data. In general, medical images (MI) can depict a body part or an anatomical region or an anatomical object of a patient, which may include various anatomies or organs. Considering the chest area as a body part, the medical images (MI) could, for example, depict the lung lobes, the rib cage, the heart, lymph nodes, and so on.

[0138] Medical images (MI) can be formatted according to the DICOM format. DICOM (Digital Imaging and Communications in Medicine) is an open standard for the communication and management of medical imaging information and associated data in healthcare informatics. DICOM can be used to store and transmit medical images and associated information, enabling medical imaging integration. A DICOM data object consists of a set of attributes, including elements such as patient name, ID, etc., as well as special attributes containing the image pixel data and metadata extracted from the image data. The metadata can be stored in what is known as the DICOM header.

[0139] The user interface 10 may comprise a display unit and an input unit. The user interface 10 may be embodied by a mobile device, such as a smartphone or a tablet computer. Furthermore, the user interface 10 may be embodied as a workstation in the form of a desktop PC or laptop. The input unit may be integrated into the display unit, e.g., in the form of a touchscreen. Alternatively or additionally, the input unit may comprise a keyboard, a mouse, or a digital pen, and any combination thereof. The display unit may be designed to display representations RE of the medical image MI, medical report templates RT, and image processing results FI in a graphical user interface GUI, wherein all elements to be displayed may be arranged according to a hanging protocol HP.The user interface 10 may further comprise an interface computing unit configured to execute at least one software component for operating the display unit and the input unit to provide a graphical user interface to enable the user to select a case of a target patient to be examined and to enter various inputs. In addition, the interface computing unit may be configured to communicate with the database DB or the processing system 20 to receive the medical images MI and any supplementary information. The user U may activate the software component via the user interface 10 and acquire the software components, e.g., by downloading them from an Internet application store. According to one example, the software component may also be a client-server computer program in the form of a web application executed in a web browser.The interface computing unit may be a general-purpose processor, a central processing unit, a control processor, a graphics processing unit, a digital signal processor, a three-dimensional rendering processor, an image processor, an application-specific integrated circuit, a field-programmable gate array, a digital circuit, an analog circuit, combinations thereof, or other now-known devices for processing image data. User interface 10 may also be embodied as a client.

[0140] The processing system 20 may include a computing unit CU and an interface unit IU. Furthermore, the processing system 20 may include or be connected to a plurality of dedicated repositories or databases, including a reporting database RDB, a tools database TDB, and a hanging log database HPDB. According to some examples, the databases RDB, TDB, and HPDB may be part of the healthcare information system 1.

[0141] The reporting database RDB is a storage device, such as a cloud or local storage, that serves as an archive for preconfigured report templates RT.

[0142] A report template RT can be viewed as a building block for a medical report. The report template RT can be designed for editing by the user via the user interface 10. The report template RT can include one or more data fields in which diagnostic information specific to the patient and / or the underlying medical image MI can be specified. The data fields can be empty fields or placeholders for various types of data, such as text, measurements, or images.

[0143] An RT report template may be specific to a particular diagnostic use case (which may be specified by the image acquisition information described herein).

[0144] The hanging log database HPDB is a storage device, such as cloud or local storage, that serves as an archive for preconfigured HP hanging logs.

[0145] A hanging protocol (HP) can comprise a set of rules in the form of computer-executable instructions for the optimal arrangement of medical information, particularly medical images, in a graphical user interface (GUI) according to a dedicated use case. A hanging protocol can specify what type of representations of a medical image (MI) and other information should be generated and where these elements should be displayed in the graphical user interface (GUI).

[0146] The tool database TDB is a storage device, such as a cloud or local storage, that serves as a repository for preconfigured image processing tools IPT.

[0147] The image processing tools IPT are generally designed for application to medical image data MI. In other words, these are tools designed to process image data to provide a corresponding processing result FI. The image processing result may relate to a medical finding FI. According to some examples, the processing tools IPT may be specialized for a specific use case, such as a type of medical image data (e.g., MR image data or CT image data) and / or a specific type of processing result. For example, one of the image processing tools IPT may be designed to detect lesions in an MR scan of a patient acquired with a HASTE sequence, while another image processing tool IPT may be designed to segment bones in a bone window of a CT scan.In general, the IDB tool database can include all image processing algorithms available for processing all types of image data that may occur at a particular healthcare facility or diagnostic workstation.

[0148] The computing unit CU can be a processor. The processor can be a general-purpose processor, a central processing unit, a control processor, a graphics processing unit, a digital signal processor, a three-dimensional rendering processor, an image processor, an application-specific integrated circuit, a field-programmable gate array, a digital circuit, an analog circuit, combinations thereof, or any other currently known device for processing image data. The processor can be a single device or multiple devices operating serially, in parallel, or separately. The processor can be a main processor of a computer, such as a laptop or desktop computer, or it can be a processor for handling certain tasks within a larger system, such as the medical information system or the server.The processor is configured by instructions, design, hardware, and / or software to perform the steps discussed herein. Further, the processing system 20 may include memory such as RAM for temporarily loading the medical images MI and any intermediate processing results. According to some examples, such memory may also be included in the user interface 10.

[0149] The processing system 20 may comprise subunits DET-U, PROC-U, DISP-U configured to process the medical image MI and to provide image acquisition information IAI, and / or further process the medical image MI based on the image acquisition information IAI.

[0150] The DET-U subunit is designed to determine the image acquisition information (IAI) for the medical images (MI). The image acquisition information (IAI) comprises the framework under which the medical image (MI) was acquired. This includes the type of modality used and the imaging parameters applied. The DET-U subunit is specifically designed to extract the image acquisition information (IAI) directly from the image data of the medical image (MI), i.e., the pixels and voxels. Non-image data is also considered as an auxiliary source. For this purpose, the DET-U subunit is designed to transform the medical image (MI) into the frequency domain (e.g., based on a Fourier transform) to generate a k-space image (KI) and to derive the image acquisition information (IAI) from the k-space image (KI).In particular, the subunit DET-U is designed to execute an appropriately configured trained function TF, which has been trained to derive the used image modality and imaging parameters, etc., from k-space images KI.

[0151] The subunit PROC-U is designed to leverage the image acquisition information IAI for further processing of the medical image MI to derive a medical diagnosis from the medical image MI by a user U. This may include providing an image processing result by selecting an image processing tool IPT from the tool database TDB according to the image acquisition information IAI and applying it to the medical image MI. Furthermore, the subunit PROC-U may be designed to generate one or more representations RE from the medical image MI for display to the user U that match the image acquisition information IAI. Furthermore, the subunit PROC-U may use the image acquisition information IAI to select suitable report templates RT and hanging protocols HP from the report database RDB or the hanging protocol database HPDB.

[0152] The DISP-U subunit is a display module or display unit. In particular, the DISP-U subunit can be configured to use the hanging protocol HP selected by the processing unit PROC-U and to arrange any representations RE, image processing results FI, report templates RT, or any other data elements in a graphical user interface (GUI) according to the hanging protocol HP.

[0153] The designation of the various subunits DET-U, PROC-U, DISP-U is to be interpreted as exemplary and not restrictive. Accordingly, the subunits DET-U, PROC-U, DISP-U can be integrated to form a single unit (e.g., in the form of "the computing unit"), or they can be embodied by computer code segments designed to execute the corresponding method steps running on a processor or the like of the processing system 20. The same applies to the interface computing unit. Each subunit DET-U, PROC-U, DISP-U and the interface computing unit can be individually connected to other subunits and / or other components of the system 1, whereby data exchange is required to perform the method steps.

[0154] The processing system 20 and the interface computing unit(s) can together form the computing unit of the system 1. It is important to note that the layout of this computing unit, i.e., the physical distribution of the interface computing unit and the subunits DET-U, PROC-U, DISP-U, is fundamentally arbitrary. In particular, the processing system 20 can also be integrated into the user interface 10. As already mentioned, the processing system 20 can alternatively be embodied as a server system, e.g., a cloud server, or a local server located, e.g., at the site of a hospital or radiology department. According to such an implementation, the user interface 10 could be designated as a user-facing "frontend" or user-facing "client," while the processing system 20 could then be thought of as a "backend" or server.Communication between user interface 10 and processing system 20 can be carried out, for example, using the https protocol. The system's computing power can be distributed between the server and the client (i.e., user interface 10). In a system with "thin clients," the majority of the computing power is located on the server. In a system with "thick clients," more of the computing power and possibly data is located on the client.

[0155] Individual components of system 1 can be connected to one another, at least temporarily, for data transmission and / or data exchange. User interface 10 communicates with processing system 20 via interface unit IU, for example, to exchange medical images MI, elements of a graphical user interface (GUI), or any user input. Furthermore, processing system 20 can communicate with database 40 and / or the dedicated database (TDB, HPDB, RDB) via interface unit IU. The interface unit IU can be implemented as a hardware or software interface, e.g., as a PCI bus, USB, or FireWire. Data transmission can be implemented using a network connection. The network can be implemented as a local area network (LAN), e.g., an intranet, or a wide area network (WAN). The network connection is preferably wireless, e.g., as a wireless LAN (WLAN or WiFi).Furthermore, the network may include a combination of different network examples.

[0156] Fig. 2 illustrates a method for providing image acquisition information IAI according to one embodiment. The corresponding data streams are shown in Fig. 3. The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but may also vary between different embodiments. Furthermore, individual steps or a sequence of steps may be repeated.

[0157] At step S10, a medical image MI of a patient is obtained. This may involve selecting the medical image MI from a plurality of cases stored, e.g., in the database DB. The selection may be performed manually by the user U, e.g., by selecting appropriate image data in a graphical user interface GUI executing in the user interface 10. Alternatively, the medical image MI may be provided to the computing unit CU by the user U, who uploads the medical image MI to the computing unit CU. According to one example, the medical image MI was acquired using a magnetic resonance imaging modality using a specific image weighting and magnetic resonance sequence.

[0158] In step S20, the medical image MI is transformed into the frequency domain to obtain a k-space image KI of the medical image MI. This may involve subjecting the medical image MI to a Fourier transform. For this purpose, a fast Fourier transform (FFT) algorithm may be applied to the medical image MI. The FFT algorithm may be hosted at the computing unit CU.

[0159] Step S20 may include transforming the entire medical image MI into the frequency domain. According to other examples, step S20 may include selecting a representative portion of the medical image MI, in particular an image slice, and transforming only the representative portion.

[0160] In step S30, image acquisition information IAI is determined based on the k-space image KI and optionally based on the medical image MI. This involves inputting the data of the k-space image KI and optionally the data of the medical image MI into the trained function TF hosted at the computing unit CU.

[0161] In the optional sub-step S31, the data of the k-space image KI can be aligned with the image data of the medical image MI. As shown in Fig. 4, this may involve generating a concatenated image CCI. In the concatenated image CCI, the data of the k-space image KI may be overlaid over the image data of the medical image MI. Alternatively, the data of the k-space image may be added as another image channel to the medical image MI, e.g., alongside the pixel- or voxel-wise intensity values. Due to the orientation of sub-step S31, each pixel / voxel of the medical image MI may be assigned a k-space value. The concatenated image CCI may then be input to the trained function TF. In this way, the trained function TF is provided with both the k-space and the real image data and, further, a relationship between the two.

[0162] In step S40, the image acquisition information IAI is provided. This may include displaying the image acquisition information IAI in the user interface 10, e.g., in a suitable graphical user interface GUI. Furthermore, step S40 may include providing the image acquisition information IAI for subsequent image processing steps, as described in connection with the Fig. 4 to 6 shown.

[0163] Fig. 4 illustrates a method for displaying a representation RE of a medical image MI according to an embodiment. Corresponding data streams are shown in Fig. 5 illustrates. Fig. Figure 6 shows a corresponding graphical user interface (GUI) for displaying the representation RE and other information in a reading and report generation workflow. The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but may also vary between different embodiments of the present invention. Furthermore, individual steps or a sequence of steps may be repeated.

[0164] In step I10, the medical image MI is received. Step I10 essentially corresponds to step S10.

[0165] At step 120, the image acquisition information IAI is determined. As in Fig. 4, this may include performing steps S20 to S40 as described in connection with the Fig. 2 and Fig. 3. Optionally, step 120 may further include classifying the image acquisition information according to a plurality of predefined diagnostic use cases. According to some examples, the diagnostic use cases may relate to a diagnostic task that a user needs to perform.

[0166] At step I30, the image acquisition information IAI is used to generate one or more appropriate representations RE of the medical image MI for display to a user U in the user interface 10. This may include determining the type of representations RE that are eligible for the image acquisition information IAI and selecting and processing appropriate image data from the medical image MI.

[0167] In particular, this may involve determining appropriate display settings for the representations RE (optional substep 131). For example, the display settings may include contrast, brightness, an intensity window (e.g., for lungs or bones), viewing angle, image enhancement, cinematic rendering parameters, and the like.

[0168] The display settings can be preconfigured and assigned to specific diagnostic use cases. Since the image acquisition information (IAI) can specify the diagnostic use case, it becomes possible to determine the display settings based on the image acquisition information (IAI).

[0169] At step I40, the one or more representations RE generated in step I30 are displayed to the user U in the user interface 10. This may include generating appropriate control signals to operate the user interface 10 to display the representation(s) RE.

[0170] Optionally, the representation(s) RE can be generated according to a hanging protocol HP. The hanging protocol HP can define what type of representation(s) RE should be displayed and where they should be displayed in a graphical user interface GUI. Furthermore, the hanging protocol HP can specify subsequent processing steps, such as retrieving comparative images CI or applying image processing tools IPT. At step 141, a preconfigured hanging protocol HP can be retrieved from the hanging protocol database HPDB that matches the image acquisition information IAI. For this purpose, a lookup operation can be performed in the hanging protocol database HPDB for a hanging protocol HP that corresponds to the image acquisition information IAI.In particular, an association linking the hanging logs HP in the hanging log database HPDB with the image acquisition information IAI or corresponding use cases can be used to find suitable hanging logs HP.

[0171] In optional step I50, a comparative medical image CI may be retrieved and provided together with the medical image MI. A comparative medical image CI may generally refer to a medical image useful in arriving at a medical diagnosis based on the medical image MI. Accordingly, the comparative medical image CI may refer to a previous study of the patient. Alternatively, the comparative medical image CI may be a similar image of another patient who may already have been diagnosed. Furthermore, the comparative medical image CI may be an excerpt from an electronic compendium, such as an electronic medical textbook.The type of comparative medical image CI can be defined in the hanging protocol HP or linked to the diagnostic use case, each of which is identified based on the image acquisition information IAI.

[0172] In particular, the comparative medical image CI can be retrieved from the database DB at sub-step 151. In doing so, medical images that match the image acquisition information IAI can be retrieved. For example, if the image acquisition information IAI indicates that a specific body part was imaged based on a HASTE sequence, the database DB can be searched for medical images that relate to the same body part and at least comparable sequences. To make such a comparison, the medical images of a patient in the database that are in question can be subjected to the same processing as the medical image MI to derive image acquisition information IAI.

[0173] In sub-step I52, the comparative medical image CI can be subjected to appropriate image processing to create a comparative representation CRE that can be easily compared with the representation RE of the medical image MI. In particular, the same image processing can be applied to the comparative medical image CI as was used for the medical image MI. In particular, the same display settings can be used.

[0174] In substep I53, the comparative representation CRE is displayed together with the representation RE. More specifically, the comparative representation CRE can be displayed in the graphical user interface (GUI) according to the hanging protocol HP selected in step I41.

[0175] In optional step I60, an image processing result FI can be generated based on the medical image MI and provided to the user U via the graphical user interface GUI. The image processing result can be generated according to the image acquisition information IAI and / or the hanging protocol HP. The image processing result FI can relate to a medical finding, measurement, or segmentation extracted from the medical image MI. The image processing result FI can be generated using a corresponding image processing algorithm or tool IPT, which can be executed by the computing unit.

[0176] In particular, in sub-step I61, an image processing tool IPT can be selected from the tool database TDB according to the image acquisition information IAI (or according to the diagnostic use case and / or the hanging protocol HP, each identified based on the image acquisition information IAI). The lookup of the image processing tool IPT can be based on an association that links the image processing tools IPT in the tool database TDB to the image acquisition information IAI (or hanging protocols HP or diagnostic use cases).

[0177] In substep I62, the selected image processing tool IPT can be applied to the medical image MI. Optionally, the image processing tool IPT can also be applied to any comparative medical image CI in substep I62 to obtain a comparative processing result FI.

[0178] In substep I63, the image processing result FI is displayed to user U in the graphical user interface (GUI). The display can be performed according to any rules in the selected hanging protocol (HP) that specify the arrangement of the image processing results FI in the graphical user interface (GUI).

[0179] In the optional step I70, a report template RT, based on which a medical report can be completed by user U, can be selected and provided. The report template RT can be provided to user U in the graphical user interface (GUI). The location of the report template RT can be determined by the hanging protocol (HP).

[0180] Specifically, at step I71, a report template RT is retrieved from the report database RDB that matches the image acquisition information IAI. For this purpose, a lookup operation can be performed in the report database RDB for a report template RT that matches the image acquisition information IAI. More specifically, an association linking the report templates RT to hanging protocols HP or diagnostic use cases (both of which can be identified based on the image acquisition information IAI) can be used to find correct report templates RT.

[0181] In Fig. 7 shows an embodiment of the trained function TF. In the Fig. In the example shown in Figure 7, the trained function TF is a convolutional neural network, specifically a deep convolutional neural network. It should be noted that this is merely an illustrative example, as the trained function TF can also be implemented by any other suitable machine learning models, such as Transformer architectures or FocalNets, as described elsewhere herein.

[0182] The trained function TF according to Fig. 7 includes convolutional layers, pooling layers, and fully connected layers. In the input layer L.1, there is a node for each pixel of the k-space image KI, where each pixel has a channel (the respective intensity value). After the input layer, there are four convolutional layers L.2, L.4, L.6, L.8, with each of the four convolutional layers followed by a pooling layer L.3, L.5, L.7, L.9. For each of the convolutional layers, a 5x5 kernel is used (indicated by "K: 5x5"), with a padding of 2 (indicated by "P: 2") and either one or two filter / convolution kernels (indicated by "F:1" or "F:2").

[0183] The first three pooling layers L.3, L.5, L.7, L.9 implement an averaging operation over patches of size 4x4, and the last pooling layer L.9 implements a maximum operation over patches of size 2x2. The additional layer L.10 of Fig.7 flattens the input k-space images KI. However, this layer is not relevant for the actual calculation.

[0184] The final layers of the network are three fully connected layers L.11, L.12, L.13, where the first fully connected layer has 128 input and 40 output nodes, the second fully connected layer L.12 has 40 input and 10 output nodes, and the third fully connected layer L.13 has 10 input and n output nodes, where the n output nodes form the output layer of the entire machine learning model.

[0185] The value of the first node of the output layer can correspond to an element of the image acquisition information IAI (e.g., MR or CT imaging procedure) of the medical image MI with respect to the input k-space image. The second node can refer to another element of the image acquisition information (e.g., spin-echo sequence or gradient-echo sequence), and so on. There can be as many output nodes as there are elements in the image acquisition information IAI that the trained function TF must distinguish.

[0186] For training the trained function TF, a database of 500 medical images MI with confirmed image acquisition information IAI was used. The database was divided into training data (320 datasets), validation data (80 datasets), and test data (100 datasets). The medical images MI were then transformed into k-space images KI. For training the trained function TF, the backpropagation algorithm was used based on a cost function L(x, y1, y2,...y n ) = |M(x)1 - y1| 2 + |M(x)2 - y2| 2 + ... + |M(x) n - y n | 2 used, where x denotes an input k-space image KI, y1 denotes whether a first element of the image acquisition information IAI is specified, y2 denotes whether a second element of the image acquisition information IAI is specified, and y ndenotes whether an n-th element of the image acquisition information IAI is specified. Furthermore, M(x) denotes the result of applying the trained function TF to the input k-space image KI, and M(x)1, M(x)2,...,M(x) n correspond to the value of the first, second,... n-th output node if the trained function TF is applied to the input k-space image KI.

[0187] Based on the validation set of 80 datasets and their corresponding annotations, the trained function TF with the best performance was selected from several machine learning models (with different hyperparameters, e.g., number of layers, size and number of kernels, padding, etc.). Specificity and sensitivity were determined based on the test set, which includes 100 datasets, and the image acquisition information (IAI).

[0188] Wherever appropriate, individual embodiments or their individual aspects and features may be combined or interchanged without limiting or expanding the scope of the present invention. Advantages described with reference to one embodiment of the present invention are also advantageous for other embodiments of the present invention, wherever applicable. Regardless of the grammatical usage of the term, the term includes persons of male, female, or other gender identities. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Zitierte Patentliteratur

[0000] EP 3 178 068 B1

[0090] Zitierte Nicht-Patentliteratur

[0000] van der Voort et al., „DeepDicomSort: An Automatic Sorting Algorithm for Brain Magnetic Resonance Imaging Data“, in Neuroinformatics, Januar 2021, 19(1):159-184, doi: 10.1007 / s12021-020-09475-7

[0008] Voort et al., „DeepDicomSort: An Automatic Sorting Algorithm for Brain Magnetic Resonance Imaging Data“, in Neuroinformatics, Januar 2021, 19(1):159-184, doi: 10.1007 / s12021-020-09475-7

[0040] Vaswani u. a., „Attention Is All You Need“, in arXiv: 1706.03762, 12. Juni 2017

[0050] Yang et al. „Focal Modulation Networks“, in arXiv: 2203.11926

[0052] Levoy: „Display of Surfaces from Volume Data“, IEEE Computer Graphics and Applications, Ausgabe 8, Nr. 3, Mai 1988, Seiten 29-37

[0090] Kajiya: “The rendering equation”, ACM SIGGRAPH Computer Graphics, Volume 20, No. 4, August 1986, pages 143-150

[0090]

Claims

[1] A computer-implemented method for providing image acquisition information (IAI) of a medical image (MI), the method comprising: - Obtaining (S10) the medical image (MI); - Transforming (S20) the medical image (MI) into the frequency domain to obtain a k-space image (KI), - determining (S30) the image acquisition information (IAI) by applying a trained function (TF) to the k-space image (KI), and - Providing (S40) the image acquisition information (IAI). [2] The method according to claim 1, wherein the image acquisition information (IAI) comprises an image acquisition parameter and / or information about a body part depicted in the medical image (MI). [3] Method according to claim 2, wherein - the medical image (MI) was acquired with a magnetic resonance acquisition procedure using a medical magnetic resonance imaging modality, and - the image acquisition parameter refers to the image weighting and / or the magnetic resonance sequence used in the acquisition procedure. [4] Method according to one of the preceding claims, wherein - in the determining step (S30) the trained function (TF) is additionally applied to the medical image (MI). [5] The method of claim 4, further comprising - generating (S31) a composite image (CI) based on the medical image (MI) and the k-space image, wherein - in the determining step (S30) the trained function (TF) is applied to the composite image (CI). [6] A computer-implemented method for displaying a representation (RE) of a medical image (MI), comprising: - Receiving (I10) the medical image (MI) from a database (DB), - determining (120) image acquisition information (IAI) of the medical image (MI) according to the method according to one of claims 1 to 5, - generating (130) a representation (RE) of the medical image (MI) for display in a user interface (GUI) based on the image acquisition information (IAI), - Display (I40) of the representation (RE) in the user interface (GUI). [7] Method according to claim 6, wherein - the step of generating (I30) the representation (RE) comprises determining (I31) a display setting for generating the representation (RE) based on the image acquisition information (IAI) and applying (I32) the selected display setting when generating the representation (RE), - wherein the display setting is selected from the following: a contrast setting, a brightness, an intensity windowing, an image enhancement, a lookup table, a viewing plane, a segmentation mask, a zoom level or panning and / or a volumetric rendering parameter. [8] A method according to any one of claims 6 or 7, further comprising: - selecting (I41), based on the image acquisition information (IAI), a hanging protocol (HP) containing a set of rules for displaying one or more representations (RE) of a medical image in a user interface (GUI), wherein - in the display step (I40) the representation (RE) is displayed based on the hanging protocol (HP). [9] A method according to any one of claims 6 to 8, further comprising: - Retrieving (151), from the database (DB) and based on the image acquisition information (IAI), a comparative medical image (CMI), - processing (I52) the comparative medical image (CMI) to generate a comparative representation (CRE) for display in the user interface (GUI), and - Display (153) the comparative representation (CRE) in the user interface (GUI). [10] The method of claim 9, wherein - the medical image (MI) was acquired from the patient at a first time point, and the comparative medical image (CMI) was acquired from the patient at a second time point different from the first time point. [11] A method according to any one of claims 6 to 10, further comprising: - selecting (I61), based on the image acquisition information (IAI), an image processing tool (IPT) designed to provide an image processing result (FI), - applying (162) the selected image processing tool (IPT) to generate the image processing result (FI), and - Displaying (163) the image processing result (FI) in the user interface (GUI). [12] A method according to any one of claims 6 to 11, further comprising: - selecting (171), based on the image acquisition information (IAI), a report template (RT) for generating a medical report corresponding to the medical image (MI), and - Providing (170) the report template (RT) via the user interface (GUI). [13] System (1) for providing image acquisition information (IAI) of a medical image (MI), comprising an interface unit (IU) and a computing unit (CU), wherein the computing unit (CU) is designed to - Receiving (S10, I10) the medical image via the interface unit (IU), - Transforming (S20) the medical image (MI) into the frequency domain to obtain a k-space image (KI), - determining (S30) the image acquisition information (IAI) by applying a trained function (TF) to the k-space image (KI), and - Providing (S40) the image acquisition information (IAI) via the interface unit (IU). [14] System (1) for providing control signals for displaying a representation (RE) of a medical image (MI), comprising an interface unit (IU) and a computing unit (CU), wherein the computing unit (CU) is designed to - receiving (S10, I10) the medical image (MI) from a database (DB) via the interface unit (IU), - Transforming (S20) the medical image (MI) into the frequency domain to obtain a k-space image (KI), - determining (S30, I20) the image acquisition information (IAI) by applying a trained function (TF) to the k-space image (KI), - generating (I30) control signals for controlling a user interface (GUI) to display a representation (RE) of the medical image (MI), wherein the representation (RE) is configured according to the image acquisition information (IAI), - Providing (150) the control signals via the interface unit (IU) to the user interface (GUI). [15] Computer program product comprising program elements which cause a computing unit (CU) to carry out steps of the method according to one of claims 1 to 12 when the program elements are loaded into a memory of the computing unit (CU). [16] Computer-readable medium in which program elements are stored which are readable and executable by a computing unit (CU) in order to carry out steps of the method according to one of claims 1 to 12 when the program elements are executed by the computing unit (CU).

Images