METHOD FOR ADJUSTING AN IMAGE Impression

Description

[0001] The invention relates to a method for adjusting the visual impression of an image, in particular an image obtained in the context of medical imaging, comprising the steps: Providing an image or input data from which the image is determined; specifying a target image impression class for at least one image segment of the image, either directly or by specifying at least one segment type, wherein a relationship between the image data of the respective image segment and an assigned image impression class is specified by a classification algorithm; modifying the image or the input data by a modification algorithm in order to align the image impression class assigned to the resulting modified image data of the respective image segment with the target image impression class, wherein at least one modification parameter of the modification algorithm is specified or is specified depending on the classification algorithm, wherein different target image impression classes are specified for different image segments and / or different segment types.wherein the image or input data are determined from source data by a processing algorithm that depends on at least one processing parameter, wherein the image impression class of the respective image segment and / or segment type depends on the processing parameter, and the processing algorithm comprises a reconstruction algorithm for reconstructing three-dimensional volume data from two-dimensional source data, in particular from X-ray images, wherein the reconstruction algorithm depends on the processing parameter or on at least one of the processing parameters, and / or wherein the image or input data are two-dimensional image data generated from the volume data by an imaging algorithm, wherein the imaging algorithm depends on the processing parameter or on at least one of the processing parameters.

[0002] In addition, the invention relates to a processing device, a computer program and an electronically readable data carrier.

[0003] Images acquired during medical imaging are typically evaluated by physicians as part of a diagnosis. These images are also used to draw the attention of others, such as colleagues or patients, to certain features within the images. Typically, this does not involve visualizing the directly captured images, but rather preprocessing the acquired data. This is particularly true when three-dimensional volume data is first reconstructed from acquired data, such as individual X-ray images from a computed tomography scan, and then two-dimensional images, such as cross-sectional images or simulated X-rays, are generated from this reconstruction. It is known that the visual appearance of the acquired image, or of individual features or segments within the image, can depend on the parameters chosen during image acquisition.For example, when reconstructing a volumetric dataset, different reconstruction kernels can be selected, which may differ in terms of the achievable spatial resolution and the noise level of the result. Depending on the intended use—for example, whether an image is to be used to visualize certain information for a patient or whether the image is to be examined for specific abnormalities, and especially depending on which abnormalities are to be detected in which image area—different preprocessing parameters and thus different resulting image appearances can be advantageous.Furthermore, different users may desire different visual impressions for the same purpose, which necessitates processing the originally recorded data with different parameters in order to provide ideal images to each user.

[0004] Since users are not necessarily aware of which processing parameters are ideal for them or a given task, it may be necessary to perform multiple processing operations on the captured data until a desired result is achieved. For example, since using iterative reconstruction of image data results in a relatively high processing overhead to provide a new image, changing the desired image appearance can lead to significant delays and thus disrupt a user's workflow.

[0005] From the article YUBIN DENG ET AL.: "Aesthetic-Driven Image Enhancement by Adversarial Learning", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, July 17, 2017 (2017-07-17), an approach to aesthetic-driven image enhancement is known that utilizes a Generative Adversarial Network (GAN). A classification algorithm is trained on image data, each of which is assigned an image quality label indicating whether the image quality is good or poor. This label is used as a discriminator in the Generative Adversarial Network to train a generator that aims to produce high-quality images from input images.

[0006] In the article MANSOOR AWAIS ET AL: "Adversarial approach to diagnostic quality volumetric image enhancement", 2018 IEEE 15TH INTERNATIONAL SYMPOSIUM ON BIOMEDICAL IMAGING (ISBI 2018), IEEE, April 4, 2018 (2018-04-04), pages 353-356, a Generative Adversarial Network is used to provide volumetric images of sufficient quality for diagnostic applications in magnetic resonance imaging (MRI) and computed tomography (CT). Low-resolution image data is used as input. A term of the cost function is determined using a discriminator, which is trained to distinguish between estimated images and images of sufficient quality for diagnostic purposes.

[0007] One approach to imprinting a specific artistic style on an image is described in the article ZHENG XU ET AL: "Beyond Textures: Learning from Multi-domain Artistic Images for Arbitrary Style Transfer", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, May 25, 2018 (2018-05-25). This approach uses a discriminator from a Generative Adversarial Network (GAN) that classifies the artistic style of input data and also determines whether the input data is real or fake.

[0008] The publication ANOOP CHERIAN ET AL: "Sem-GAN: Semantically-Consistent Image-to-Image Translation", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, July 12, 2018 (2018-07-12), XP081247481, addresses the problem of image-to-image translation, which consists of mapping an image in a source domain to an image in a target domain. To ensure that the translated images are plausible, this mapping must be invertible. This requirement yields promising results when the domains are unimodal. However, multimodal domains, such as in an image segmentation task, are problematic. Therefore, a semantically consistent GAN, called Sem-GAN, is presented in which the semantics are defined by the class identities of image segments in the source domain, as generated by a semantic segmentation algorithm.The proposed GAN forces the translated images to adopt the appearance of the target domain, while largely preserving their identities from the source domain. It is shown that semantic segmentation models trained on synthetic images translated with Sem-GAN yield significantly better segmentation results than other approaches.

[0009] The invention is therefore based on the objective of improving the workflow when a user evaluates image data.

[0010] The problem is solved by a procedure of the type mentioned at the beginning.

[0011] It is therefore proposed that the visual appearance of specific segments or segment types within an image should not be altered by reprocessing the original image data with different parameters, but instead by modifying the image itself using a modification algorithm. This involves using a modification algorithm that is parameterized, or becomes parameterized, based on a classification algorithm. As will be explained in detail later, such parameterization can be achieved, for example, through a machine learning method. In principle, however, manual parameterization, taking into account a predefined classification algorithm, would also be possible.

[0012] The proposed approach allows, once a suitable modification algorithm is available, the visual appearance of specific segments or segment types of the image to be adjusted as needed. Such adjustments can be made manually according to a user's requirements, or they can be automated based on specific parameters, as will be explained in more detail later. The ability to quickly adjust the visual appearance enables users to switch between different visual appearances as needed, for example, to find a visual appearance that best supports their task without interrupting their workflow. The inventive method allows different visual appearances to be set for different segments or segment types of the image.For example, it is possible to sharpen features in some image segments while reducing noise in others. This would be very complex with conventional methods, as the source data would have to be processed in different ways, and the resulting image would have to be created by combining several of these processing results to achieve locally different image appearances.

[0013] The classification algorithm is not necessarily used in the process itself. If the modification parameters of the modification algorithm are determined in a preceding step, i.e., not as part of the process itself, it is sufficient if the classification algorithm is available in this preceding step. As will be explained in more detail later, the classification algorithm can be parameterized, in particular, using a machine learning method.

[0014] The modification algorithm preferably comprises a plurality of modification parameters, which are determined within the framework of the inventive method or in a preceding step depending on the classification algorithm. This can be achieved through machine learning. As will be explained in more detail later, the classification algorithm and the modification algorithm can be trained together. For example, these algorithms can jointly form a Generative Adversarial Network (GAN) during training, with the modification algorithm acting as the generator and the classification algorithm as the discriminator.

[0015] The image impression class can relate to, for example, the sharpness of an image segment or the type of representation, such as whether contiguous areas are only shown as outlines, brightness and / or contrast, whether certain features are shown at all, or similar aspects. The target image impression classes for individual image segments or segment types can be specified directly. Alternatively, a rendering task can be defined that specifies target image impression classes for multiple image segments or segment types. For example, it can be specified that a particular organ should be displayed sharply, while other image segments should be displayed with low noise.

[0016] A segment type can be, for example, a specific anatomical label, whereby the modification algorithm itself or a preceding image segmentation process can identify image segments that are assigned to a corresponding label. Segment types can describe individual organs, bones, or similar features; however, it is also possible for a segment type to describe an entire group of features. For example, the segment type "bones" can be assigned to all image segments that show bones or similar features.

[0017] To modify an image impression class for a specific image segment, in the simplest case, the modification algorithm can be applied exclusively to the data of that image segment. This can be achieved, for example, by first segmenting the image manually or automatically using a segmentation algorithm trained, for instance, through machine learning. Preferably, however, the modification algorithm recognizes all relevant image areas—that is, image segments of a specific segment type—itself and modifies them. In this case, it is not necessary for the modification algorithm to explicitly output a segmentation or image data assigned to specific segments.Rather, it is sufficient if the result is that the image impression class for the corresponding image segment, specified directly or via a segment type, is aligned with the target image impression class specified for that image segment or segment type.

[0018] Adjusting the image impression class of an image segment can result in the segment's image impression class being identical to the target image impression class. However, it is also possible for the image impression classes to be ordered in a specific way. This can be a one-dimensional order, for example, based on the achieved sharpness of the image or image segment, or a multi-dimensional order, such as based on sharpness and contrast. Adjustment in this context means that the segment's image impression class, after applying the modification algorithm, is closer to the target image impression class than it was for the original image segment.

[0019] The image data is preferably two-dimensional. In particular, it can be two-dimensional image data generated from two-dimensional volume data, for example, cross-sectional images or artificial projections.

[0020] The input data preferably consists of several two-dimensional images that are combined to generate the image. For example, the input data could be a mask image acquired during angiography and an image with contrast medium, and the image can be derived by subtracting these input images from each other. To modify the image, the resulting image can be modified. However, it is also possible to modify the input data, such as the contrast medium image or the mask image, separately using the modification algorithm to indirectly alter the image.

[0021] The modification parameter can be determined or generated using a machine learning method. In the simplest case, the classification algorithm can be predefined. For example, the classification algorithm could be a previously trained convolutional neural network. Using the classification algorithm, unsupervised learning of the modification parameters can occur; that is, it is not necessary to specify the desired results, such as pre-modified images. To train the modification algorithm, it may be sufficient to provide a training dataset that includes only the images to be modified and instructions on how they should be modified.

[0022] The modification parameters of the modification algorithm, such as the weighting factors of individual neurons when a neural network is trained as a modification algorithm, can initially be initialized arbitrarily, for example, randomly. The modification algorithm is provided with input data such as a specific image and a desired modification, i.e., a target image impression class for specific segments or segment types. Using the established classification algorithm, it can be checked after the modification of the respective image whether and how successful it was. This information can then be used to modify the modification parameters of the modification algorithm. The modification algorithm can, in particular, be a generative network or a deconvolutional network.Such neural networks are particularly suitable for generating or modifying certain image data.

[0023] The training of the modification algorithm, i.e., the determination of the modification parameters, can be a process step of the method according to the invention. However, it is also possible to perform this training as a separate process. In this respect, the invention also relates to a method for training a modification algorithm that serves to modify an image or segments of the image to change the visual impression.

[0024] Pre-segmented images can be used to train the modification algorithm. Segmentation can be performed manually or by a segmentation algorithm, particularly one trained through prior machine learning. Segment types can be assigned to individual segments or groups of segments. In principle, segmentation information can also be provided to the modification algorithm as input data. This can be advantageous, for example, if the modification algorithm is to be trained to modify pre-segmented images.

[0025] However, it can also be advantageous not to evaluate the segmentation information within the modification algorithm, but rather to use it exclusively to determine image impression classes for the individual predefined segments of the modified image data using the classification algorithm. These classes are then compared with the target image impression class for the corresponding segment or segment type, and the modification parameters are adjusted accordingly. In this case, the modification algorithm can receive, in addition to the image, pairs of segment types and the target image impression classes specified for each segment type as input data. The modification algorithm is thus trained to independently recognize which image areas need to be modified and how, in order to achieve an alignment of the image impression class of image segments of a specific segment type with a target image impression class.This can, for example, prevent artifacts that might result if different image segments are modified in different ways to achieve different image impression classes for those segments.

[0026] At least one classification parameter of the classification algorithm and the modification parameters can be jointly determined or determined using a machine learning method. This involves, in particular, supervised training of the classification algorithm. For example, a training dataset can comprise several images, with the images, or rather the individual segments or segment types of the images, being assigned respective image impression classes, for example, through prior manual classification. The classification algorithm can be, for example, a neural network, such as a convolutional neural network. Approaches to supervised learning of classification algorithms are generally known and will therefore not be explained in detail here.For example, error feedback can be used to minimize the deviation between an image impression class determined by the classification algorithm and the image impression class specified by the training data set.

[0027] Subsequent joint training of the classification algorithm and the modification algorithm enables further continuous improvement of the classification algorithm and thus ultimately also of the modification algorithm. The classification algorithm and the modification algorithm can be trained alternately or simultaneously.

[0028] Within the framework of machine learning, a learning algorithm can simultaneously or alternately attempt to select the modification parameter such that, when applying the classification algorithm to an image segment of a result image, determined by applying the modification algorithm to an input image, and to select the classification parameter such that, when applying the classification algorithm to the at least one image segment of the result image and to an image segment of the input image assigned to the image segment of the result image, the same image impression class is determined.

[0029] In other words, the modification algorithm is trained simultaneously so that the segment exhibits the specified target image impression class when the classification algorithm is subsequently applied, while the classification algorithm is trained to recognize such manipulation. One possible approach is to train a Generative Adversarial Network (GNA) in which the modification algorithm is used as the generative network and the classification algorithm as the discriminator. The described learning approach achieves, in particular, that apart from the prior training of the classification algorithm, the training of the modification algorithm can be unsupervised. This means that no target results need to be specified for the modification algorithm, since learning occurs using the classification algorithm instead of a comparison with a target result.

[0030] Depending on the number of image impression classifications to be distinguished, it may also be advantageous to use several parallel or combined neural networks to determine the classification parameters and modification parameters.

[0031] During training, target image impression classes can also be specified for multiple segments or segment types. In this case, the above explanation applies to each pair of image segment from the result image and its corresponding image segment from the input image.

[0032] In the method according to the invention, the image or input data is determined from source data by a processing algorithm that depends on at least one processing parameter, wherein the image impression class of the respective image segment and / or segment type depends on the processing parameter. The processing of the source data is part of the method according to the invention, so that the image or input data is provided for the method according to the invention. The classification algorithm can be predefined, or the at least one classification parameter of the classification algorithm can be determined, such that the image impression classes determined by the classification algorithm depend on the respective processing parameter used. This can be achieved, for example, by machine learning.For example, supervised learning can be implemented by generating the image or input data from preferably several predefined source datasets, each with different processing parameters. Here, sets of processing parameters can be assigned to specific image impression classes, for example, based on prior knowledge of how these processing parameters work. Thus, the training dataset can include an image impression classification associated with each processing parameter for each image or input dataset.If the classification algorithm is implemented as a neural network, for example, the learning can then be carried out by comparing the determined image impression classification with the image impression classification stored in the training data set, and classification parameters of the classification algorithm can be adjusted, for example, by error feedback.

[0033] The processing parameter can, for example, influence image sharpness or image noise in the image or in specific segments of the image. This can occur, for instance, if the processing algorithm adjusts a convolution kernel or similar depending on the processing parameter. The image impression classes determined by the classification algorithm can, in this case, indicate how sharp or soft a particular segment of the image is. The modification algorithm can then be used to subsequently smooth or sharpen the image accordingly.

[0034] In the method according to the invention, the processing algorithm comprises a reconstruction algorithm for reconstructing three-dimensional volume data from two-dimensional source data, in particular from X-ray images, wherein the reconstruction algorithm depends on the processing parameter or on at least one of the processing parameters and / or wherein the image or the input data are two-dimensional image data generated from the volume data by an imaging algorithm, wherein the imaging algorithm depends on the processing parameter and / or at least one of the processing parameters. For example, the three-dimensional volume data can be generated from two-dimensional X-ray images in the context of computed tomography, for example by filtered backprojection, iterative reconstruction, or similar methods.The image data or input data can, for example, be layer images or synthetic X-ray images generated from volume data.

[0035] The reconstruction parameter can, for example, relate to the selection of the reconstruction kernel used or its parameterization. For instance, a reconstruction kernel can be chosen so that the volume data specifies Hounsfield units, or a reconstruction kernel can be chosen that highlights edges.

[0036] In the method according to the invention, different target image impression classes are specified for different image segments and / or different segment types. For example, it may be desirable to have a particularly sharp image or to emphasize edges in a specific relevant area, such as the area of ​​a particular organ, while noise reduction is preferred in the surrounding background area, even if this may lead to a reduction in spatial resolution. It may also be desirable to omit certain segment types that are less relevant to a current visualization, or to display them with lower brightness or contrast, so that relevant features can be recognized more quickly and easily.Here, different representations and thus corresponding sets of target image impression classes for the various image segments or segment types can be selected automatically or manually as needed. For example, by selecting the appropriate target image impression classes, it can be achieved that, based on the same image or the same input data, either bones or the vascular system are emphasized more, depending on the requirements.

[0037] The target image impression class and / or the at least one image segment and / or the at least one segment type for which the target image impression class is specified can be defined based on user input and / or user-identifying information and / or the image and / or the input data and / or the source data and / or additional information relating to the image processing and / or a patient depicted in the image. This can serve, for example, to automatically modify the image as soon as a user opens it, so that an image impression suitable for a specific user or task is created. However, it is also possible that the aforementioned parameters are only used to assist a user in selecting a suitable modification.For example, depending on these parameters, only certain modifications can be suggested, or the order of the suggested modifications can be adjusted. These approaches can also be combined. For instance, an automatic modification of the image can be performed first, and the user can then switch to other modifications as needed, with a suitable selection offered.

[0038] Predefining parameters based on user information can be advantageous for adapting a displayed modified image to a user's preferences. These preferences can be learned continuously. For example, initially, a user unknown to the system may need to manually select how an image should be modified, specifically which segments or segment types should exhibit which desired image impression classes. After one or more such inputs, the system can learn the user's preferences, for example, through machine learning or statistical analysis of the inputs, taking into account the specific usage context in which the input was made. Subsequently, the system can automatically select a modification or limit the suggested modifications based on these preferences.Even after learning user preferences, learning can continue, for example by implementing a feedback loop to detect when the user switches from the assumed optimal modification to another modification in certain situations, in order to recognize changed user preferences.

[0039] User information can be obtained, for example, by having a user identify themselves to the system, such as by entering a user ID, using a key card, or similar means. However, automated user recognition is also possible. This can involve analyzing input patterns, such as the sequence of keystrokes, mouse gestures, the selection of specific program options, and similar behaviors. For instance, a machine learning algorithm can be used to identify users based on their actions within the system.

[0040] Dependence of the target image impression class and / or the image segment and / or the segment type on the image, the input data, and / or the source data can be advantageous, for example, when automatic feature recognition is performed within this data. For instance, it can be detected that features in a certain image segment are present, indicating a lesion, tumor, or other anomaly. By selecting an appropriate target impression class for this image segment or by performing other corresponding manipulations of the image, it can be ensured that such potentially highly relevant features are clearly and easily recognizable to a user.

[0041] Imaging parameters can include, for example, exposure times, radiation intensities or tube currents, the temporal progression of X-ray acquisition, information indicating undersampling, or similar factors. Imaging parameters can significantly influence the representation of certain features in images, making it advantageous to adjust how an image is modified based on these parameters—that is, to determine the desired impression class for the image, or for specific image segments or segment types.

[0042] Information concerning the patient can include, for example, information from a patient file. This information can reveal, for instance, the purpose of the image acquisition or the specific representational or diagnostic task the image is intended to accomplish. For example, whether the vascular system or the bones depicted in the image are more relevant to a current diagnostic task can be highly important for selecting an appropriate image.

[0043] The aforementioned list of parameters, on which the selection of the target image impression class or the modified image segments or segment types depends, is not exhaustive. For example, prior user input, particularly while viewing the same image, could also be taken into account to draw further conclusions about the user's preferences, or consideration could be given to whether a visualization is taking place, for example, in a consultation room (i.e., potentially for a patient), on a measuring device, or on an office computer.

[0044] The specification of the target image impression class and / or the at least one image segment and / or segment type for which the target image impression class is specified, and / or the specification of target image impressions and / or image segments and / or segment types selectable or recommended by the user, for which the target image impressions are selectable, recommended, or specified, can be performed by a specification algorithm that is parameterized by machine learning. Training can be carried out by using potentially relevant parameters for a selection, such as the parameters discussed above, for a specific usage situation and a user selection of the target image class(es) or the segments or segment types to be modified within that usage situation, as training data.The target algorithm can be trained through supervised learning, where manual selection defines the target result of the algorithm and the captured relevant parameters serve as inputs. Following a training phase for the target algorithm, further user monitoring can be performed to detect changes in user behavior, for example.

[0045] The modification algorithm can be configured to automatically specify image segments and / or segment types, along with their respective target image impression classes, depending on a predefined rendering task for the image. This can be particularly useful if a user is to manually select the image modification and different target image impression classes are to be used for different image segments or segment types, at least for some of the possible rendering tasks. In this case, it is advantageous to define a type of macro that allows a user to perform complex modifications to the image by selecting a specific rendering task.

[0046] Within the context of a specific rendering task, but also independently, by specifying appropriate target image impression classes for the corresponding segments or segment types, it can be achieved, for example, that certain existing features or the segments they comprise are displayed with low brightness or low contrast, or not at all, or only schematically. The omission of certain features can be achieved, for example, by replacing parts of the segments with a solid color or a specific pattern. This can be advantageous, for instance, when a particular situation needs to be visualized for a patient and irrelevant features need to be hidden.

[0047] A schematic representation can be achieved, for example, by displaying only the boundary lines of captured features in certain image segments. This can be accomplished, for instance, by creating a gradient in these segments and then comparing the limits. Optionally, morphological operations can also be used to create closed boundary lines. Of course, even more advanced abstractions are possible, such as representing vessels solely by their midline, or similar approaches. If the image is generated from input data comprising multiple images, for example, for digital subtraction angiography, it is also possible to display certain segments or segment types transparently.For example, depending on the selected display task or target image impression class, the background of an angiographic image, i.e., bones and / or other organs, can be completely hidden or displayed with low contrast or low brightness for user orientation.

[0048] A possible example of a rendering task is removing bones from the image. The bones can be pre-segmented, or they can be segmented by the modification algorithm itself. In this case, the rendering task "remove bones" could, for example, specify that the content of the corresponding segments for the bone segment type is not displayed, is displayed with lower brightness or contrast, or is shown only schematically.

[0049] The image or the modified image data can be displayed to the user(s), whereby at least one image area is marked that includes a feature whose representation depends on the image impression classification of the image or at least a segment of the image, or on the display task. Certain features depicted in the image may be difficult to discern in the image itself or in certain modified versions of the image. This may be desirable in some circumstances, as it can improve the recognition of other features. At the same time, such features may still be relevant for a current diagnostic task or similar purpose.It can therefore be advantageous, in particular, to mark image areas where features are difficult or impossible to recognize in a current representation, in order to indicate to a user that this feature can be represented or made more clearly visible by selecting different target image impression classes or a different representation task.

[0050] In particular, it may be possible for the user to select marked image areas via user input. For example, the image may be displayed on a touchscreen, and corresponding image areas can be touched by the user, selected with a mouse, or similar. When a marked image area is selected, a different display task or a different target image impression class can be specified automatically or after prompting the user, either for the entire image or at least for a segment, in order to achieve a modified, and especially clearer, representation of the feature through a further modification of the image.

[0051] The image can be generated by superimposing, and in particular subtracting, source images described by the input data, with the modification algorithm processing the source images as input data. Specifically, the image can represent a digital subtraction angiography, in which a mask image acquired without contrast agent is subtracted from a contrast-enhanced image to achieve a clearer visualization of the patient's vascular system. CT angiography can be used for this purpose, where CT scans can be performed with and without contrast agent, so that the contrast-enhanced image and the mask image can be cross-sectional images representing corresponding volume data. In principle, it would be possible to apply the modification algorithm to the image resulting from the superimposition.However, processing the source images as input data allows for the separate modification of the source images and the adjustment of the overlay method. For example, for a specific visualization task or desired image effect, it may be necessary to display organs that are typically almost completely hidden in digital subtraction angiography, either as outlines or with full contrast. It may also be possible to subsequently overlay the subtraction image, which shows only the vascular system, with the mask image, but with different modifications to the mask image and the subtraction image. For instance, the subtraction image could be sharpened while the mask image is softened, or something similar. This allows for the provision of a user-specific, flexibly adjustable display.

[0052] Adapting a superimposed representation, where the source images are used as input data for the modification algorithm, can, for example, be used to display information from different parallel layers of a volume dataset together in a single image. Depending on the display task or the desired image impression class for the entire image, the individual image segment, or the individual segment type, the modification algorithm can influence the superimposition, for example, so that in certain segments or segment types only information from one of the layers is displayed, or so that a superimposition occurs in such a way that information from one of the layers lies as a kind of transparent layer over the information of the other layer.

[0053] As already explained, the determination of the at least one classification parameter of the classification algorithm and / or the determination of the at least one modification parameter of the modification algorithm can be carried out independently of the method according to the invention as a preliminary step. The classification parameter and / or the modification parameter can be determined by a respective machine learning method or, in particular, jointly by a machine learning method.

[0054] Therefore, the invention also relates to a method for determining a classification parameter of a classification algorithm, which serves to assign an image impression class to image data of an image and / or a respective image segment, by means of a machine learning method. This has already been explained in detail above.

[0055] Furthermore, the invention relates to a method for determining at least one modification parameter of a modification algorithm, which serves to modify an image or input data from which the image is derived, such that an image impression class, the resulting modified image data of the image or of a respective image segment of the image, is assigned by a classification algorithm, and is aligned with a predetermined target image impression class, wherein the modification parameter is determined by a machine learning method. In particular, the modification parameter and a classification parameter that parameterizes the classification algorithm can be determined jointly by a machine learning method. The determination of the modification parameter, or the joint determination of the modification parameter and the classification parameter, has already been explained in detail above.

[0056] Thus, the invention also relates to an electronically readable data carrier on which classification parameters and / or modification parameters and / or an implementation of a classification algorithm parameterized by these classification parameters and / or a modification algorithm parameterized by these modification parameters, determined or determinable by the method described above, are stored.

[0057] In addition to the method according to the invention, the invention relates to a processing device for processing images, which is configured to carry out the method according to the invention. The processing device can, in particular, be integrated into an X-ray device, for example, a computed tomography scanner, or be part of an X-ray system, which in particular includes an X-ray device for acquiring X-ray images, especially a computed tomography scanner. However, it is also possible for the processing device to be a correspondingly programmed workstation or server. Such a server can, for example, be located locally in the same building as an image acquisition device, in particular an X-ray device, but at a distance from it.The processing device can also be configured as a cloud system, which can be implemented by a multitude of servers, particularly those located at different sites. The processing device preferably includes a memory for storing data generated during the process according to the invention. The memory, or a further memory, can also store a program that implements the process according to the invention. The processing device also preferably includes a processor that can perform the steps of the process according to the invention.

[0058] In addition, the invention relates to a computer program that can be loaded directly into a memory of a processing device, with program means to carry out the steps of the method according to the invention when the program is executed in the processing device.

[0059] The invention also relates to an electronically readable data carrier with electronically readable control information stored thereon, which comprises at least one computer program according to the invention and is designed in such a way that, when the data carrier is used in a processing device, it carries out the method according to the invention.

[0060] Further advantages and details of the invention will become apparent from the following exemplary embodiments and the accompanying drawings. These schematically illustrate: Figures 1 and 2 show the process of an embodiment of the method according to the invention, Figures 3 and 4 show the training of a [subject / subject] in the [method / subject]. Fig.1 and 2The modification algorithm usable in the method shown, Fig. 5 a further embodiment of the method according to the invention, Fig. 6 the display of modified image data in an embodiment of the method according to the invention, and Fig. 7 an embodiment of an X-ray system comprising a processing device according to the invention.

[0061] Fig. 1 Figure 1 shows a flowchart of a procedure for adjusting the perceived appearance of an image, in particular an image acquired during medical imaging. The data and processing modules used in the procedure are schematically shown in Figure 2. Fig. 2 depicted.

[0062] In step S1, source data 1 is first determined, which could be, for example, individual X-ray images taken of a patient during a computed tomography scan. In step S2, this data is processed by a processing algorithm 15, which depends on processing parameters 3 and 6, to provide image 7. The perceived image quality of image 7, or of individual image segments 16 and 17 of image 7, typically depends on these processing parameters 3 and 6. Such an image quality can be quantified by applying a predefined classification algorithm 13 to individual image segments 16 and 17 of image 7 to determine a corresponding image quality class 14. As will be explained in more detail later, such a classification algorithm can, for example, be defined or parameterized using a machine learning method.

[0063] In the illustrated embodiment, the processing algorithm 15 comprises a reconstruction algorithm 2 that reconstructs three-dimensional volume data 4 from the two-dimensional source data 1. This reconstruction algorithm is parameterized by the processing parameters 3, which can specify, for example, which reconstruction kernel should be used. For instance, a reconstruction can be performed such that the volume data 4 specifies Hounsfield units for individual points in the volume. However, it is also possible to highlight edges in the volume data. Furthermore, depending on the specific reconstruction used, a higher spatial resolution or a better signal-to-noise ratio can be achieved, for example. The processing parameters 3, on which the reconstruction algorithm 2 depends, can thus directly influence the image impression and therefore the image impression class of the individual image segments 16, 17.

[0064] Within the framework of the processing algorithm 15, the volume data 4 are subsequently mapped by an imaging algorithm 5, which is parameterized by further processing parameters 6, to generate the image 7. The imaging algorithm can, for example, generate tomographic images or synthetic X-ray images from the volume data 4. The processing parameters 6 can, for example, describe which layer thickness perpendicular to an image plane is taken into account when generating tomographic images, or to what extent blurring or sharpening takes place during the imaging process. These parameters also influence the image impression and thus the image impression class of the image segments 16, 17.

[0065] Different image impressions may be advantageous for various diagnostic purposes. Furthermore, different users may prefer different image impressions. To avoid having to repeat the entire processing algorithm 15 each time an image impression is changed—which can be very computationally intensive in some cases, for example, when reconstruction algorithm 2 is used as an iterative reconstruction algorithm—a modification algorithm 8 can be used in step S3 to generate modified image data 12. The modification is carried out such that the image impression class 14, which is assigned to the modified image data 18, 19 of the respective image segments 16, 17 via the classification algorithm 13, corresponds to a predefined target image impression class 9.

[0066] The target image impression class 9 is determined from target data 11 by a predefined algorithm 10, which, as will be explained later, can be parameterized, in particular by machine learning. The target data 11 can relate to the user in order to adapt the image impression to user preferences. In addition, it can relate to the specific usage situation, e.g., a display or diagnostic task. Various possible target data will be discussed later with reference to Fig. 5 be discussed.

[0067] The modification of image 7 can occur even before it is displayed to a user in step S4. For example, based on available default data or a user ID, it can be determined that a particular image appearance is likely to be advantageous. After the modified image data 12 is displayed in step S4, user input can be captured in step S5. Depending on this input, a further modification of image 7, adapted to the user input, is performed in step S3. This procedure can be repeated as often as necessary to provide the user with optimal information. The user input captured in step S5 can be considered in the next application of the modification algorithm, particularly as part of the default data 11.

[0068] The described procedure can be modified so that instead of directly modifying image 7, it modifies the input data from which image 7 is derived, or the image can be modified by modifying the process of deriving the image from input data. The input data can, in particular, consist of several two-dimensional images that are superimposed or subtracted to determine the image. For example, it could be a mask image and a contrast-enhanced image that are subtracted from each other using digital subtraction angiography. Here, the individual images of the input data can be processed as follows: Fig. 2 This is explained by first reconstructing the respective two-dimensional source data 1 into the respective volume data 4, from which input images are generated. This is useful, for example, to provide a mask image and a contrast agent image. However, the input images can also be different images generated from the same volume data 4, for example, different layer images that can be superimposed. By using several two-dimensional input images to determine the image and modifying these images or their superposition, the resulting image impression can be further customized, for example, by specifying in which of the segments 16, 17 of the image image data from which input image is shown, or how the image data of the input images are weighted when superimposed.

[0069] The Figuren 3 and 4We demonstrate a method for determining classification parameters of classification algorithm 13 and modification parameters of modification algorithm 8, which can be carried out as part of the previously described procedure, but also independently to prepare the described procedure. Here, we show Fig. 3 a flowchart of the procedure and Fig. 4 the data structures and algorithms used here. Steps S6 to S9 serve to provide a training dataset 20, which is used to train the classification algorithm 13 and can also provide images for training the modification algorithm 18. For this purpose, reference data is first acquired in step S6. For example, several computed tomography scans for different patients can be acquired. For the individual reference datasets, the following is determined in step S7: Fig. 2 The discussed processing algorithm 15 is used to first generate volume data sets and then to create two-dimensional images from these. Here, the processing parameters 3 and 6 are varied to generate multiple input images 21 for each of the reference data sets acquired in step S6. These input images differ from one another in terms of the overall image impression and the image impression of individual image segments, respectively, due to the different processing parameters 3 and 6 used during the processing.

[0070] In step S8, the input images 21 are segmented. If the classification algorithm is to be trained solely for classifying the overall image impression, or if the modification algorithm is to be trained solely for modifying the overall image impression, then step S8 can be omitted. The segmentation can be performed manually by one or more experts. However, it is particularly preferable to perform the segmentation in step S8 using an algorithm for the automatic segmentation of the input images 21. Such an algorithm can be trained beforehand, for example, using machine learning. This involves, for instance, performing a manual pre-segmentation on a sufficiently large number of images and then training the algorithm within a supervised learning environment so that its segmentation closely approximates the predefined segmentation.

[0071] In step S9, each resulting image, or, if the resulting images were previously segmented, each segment of the resulting image, is assigned a corresponding image impression class. It is also possible for this assignment to be made not for individual segments, but for segment types, for example, for all segments showing bones or part of a vascular system. The assignment can only be done manually by one or more experts based solely on the input images themselves.However, since it can typically be assumed that, at least for all segments of a given segment type, a strong correlation exists between the image impression in the input images or image segments and the processing parameters 3, 6 used to determine the images, it is also possible to assign a specific image impression class 29 to the various combinations of processing parameters or specific value ranges of processing parameters, either generally or for a specific segment type. This avoids the need to manually classify all input images 21, as it is sufficient to define a corresponding assignment between processing parameters 3, 6 and the image impression class 29.

[0072] As a result of step S9, a training dataset 20 is available, comprising a large number of input images 21 and image impression classes 29 assigned to the respective input images 21 or to the individual image segments of the input images 21. Using this training dataset 20, a preliminary training of the classification algorithm 13 is performed in step S10. The classification algorithm 13 is used at several points in the described procedure. To achieve the clearest possible presentation, the classification algorithm 13 has therefore been presented in Fig. 4 It is shown multiple times. However, as indicated by the dashed double arrows and the identical reference symbol, it is the same classification algorithm 13 in each case.

[0073] The classification algorithm 13 can be trained using supervised learning. The training serves to determine several classification parameters. If the classification algorithm 13 is, for example, an artificial neural network, in particular a convolutional neural network, the classification parameters can be the input weights of individual artificial neurons. The initial values ​​of the classification parameters can be arbitrarily defined, i.e., chosen randomly. During training, the appropriately parameterized classification algorithm 13 is applied to at least one of the input images 21 in each case to determine an image impression class 22. The image impression class 22 can be determined for the entire image and / or for the segments determined previously in step S8.Through error feedback 23, the determined image impression class 22 is compared with the image impression class 29 specified in step S9, and the classification parameters of the classification algorithm 13 are adjusted based on this comparison. Corresponding methods for supervised learning of classification algorithms are known in principle and will therefore not be discussed in detail.

[0074] After sufficient training of the classification algorithm 13, for example, after a fixed number of training iterations or after fulfilling a convergence criterion for the classification parameters, the supervised learning of the classification algorithm 13 is initially complete. The following steps serve to perform joint unsupervised learning of the classification algorithm 13 and the modification algorithm 8. While the input images 21 of the training dataset 20 can still be used, other input images can also be used arbitrarily, since the associated predefined image impression classes 29 are no longer required. The learning structure used below has the structure of a Generative Adversarial Network, with the modification algorithm 8 serving as the generator and the classification algorithm 13 as the discriminator.

[0075] In step S11, input images are first provided, which can be, for example, input images 21 from training dataset 20 or other input images. In addition, a target image impression class is specified for each input image or specific segment or segment type in order to train the modification algorithm 8 to achieve this target image impression class for the image segment or all segments of a segment type.

[0076] In step S12, the input image is modified by the modification algorithm 8. This modification depends on the modification parameters of the modification algorithm. The modification algorithm can, for example, be a neural network, in particular a deconvolutional neural network. In this case, the modification parameters can be, for example, the input weights of different artificial neurons. The modification parameters can initially be chosen arbitrarily, i.e., randomly. A result image 24 is output as the processing result.

[0077] In step S13, the classification algorithm is used to identify the image impression classes of the result image 24, or rather, of the individual segments of the result image 24. The classification algorithm 13 is initially parameterized by the classification parameters determined in the previously described training. These parameters can change during further training, as will be explained later. If a classification for individual image segments is to be determined, the result image 24 can be segmented beforehand. However, it is particularly advantageous if the input images provided in step S11 are segmented and the corresponding segmentation is then transferred unchanged to the result image 24. Various segmentation options have already been explained with reference to step S8.

[0078] The image impression classes 25 determined for the modified image data 24 or their image segments are compared by the error feedback 26 with the target image impression classes specified in step S11 for the corresponding image segment and, depending on the comparison result, the modification parameters of the modification algorithm 8 are adjusted.

[0079] To achieve more robust training of the modification algorithm 8, it is advantageous to simultaneously train the classification algorithm 13 further in such a way that it attempts to continue recognizing the same image impression class despite the modification by the modification algorithm 8. This prevents, in particular, the modification algorithm from being trained to alter the image in a way that hardly changes the actual image impression but leads to a misclassification of the image impression. To achieve this parallel training, in step S14 the classification algorithm 13 is applied to the same input images that are also fed to the modification algorithm 8, in order to determine a respective image impression class 27 for the unchanged input images or their image segments.In step S15, the individual image impression classes 27 for the unmodified input images are compared with the corresponding image impression classes 25 for the result images 24 by further error feedback 28. Depending on this, the classification parameters of the classification algorithm 13 are adjusted, with the optimization being carried out in such a way that, where possible, the image impression classes 27 for the input images or their segments correspond as closely as possible to the image impression classes 25 for the result images 24 or their segments.

[0080] Steps S12 to S15 are repeated, for example, for a specific number of iterations or until a convergence criterion for the modification parameters and / or the classification parameters is met. Since this is an unsupervised learning process, the number of iterations is not limited by the number of available training datasets. Therefore, high-quality training of the modification algorithm can be achieved, with the training quality being limited only by the quality of any prior training of the classification algorithm 13.

[0081] As already mentioned Fig. 2 As explained, the specification of the target impression classes 9 for segments 16 and 17 of Figure 7 can be carried out by a specification algorithm 10, which is trained by a machine learning method. This will be explained below with reference to Fig. 5 This will be explained in more detail using an exemplary embodiment. The aim of the training is for the trained processing algorithm 10 to automatically evaluate the target image impression classes 9 that are particularly suitable for a user and a current usage situation for the image segments 16, 17 of the image 7, so that the modification algorithm 8 can provide correspondingly modified image data 12. This procedure was previously described with reference to Fig. 1 and Fig. 2 explained.

[0082] In the illustrated embodiment, the following are used as input data 11: user information 29 that identifies a user, features 30 recognized in the image, additional information 31 regarding the imaging, additional information 32 regarding the patient, and a user-defined display task 33 that specifies, for example, what type of diagnosis image data should be provided for.

[0083] The target algorithm 10 can, for example, be a neural network, whereby parameters describing the input weights of the network's artificial neurons are determined during training. Preferably, supervised learning is used, meaning that, in addition to the target data 11, user inputs 34 are recorded at least during a training phase. These inputs directly specify which target image impression classes 9 should be present in the modified image data 12 for which image segments 16, 17 or segment types. The target image impression classes determined by the target algorithm 10 can then be compared with the target image impression classes specified by the user input 34 as part of error feedback 35. The parameterization of the target algorithm 10 can be adjusted accordingly to minimize deviations between these pieces of information.

[0084] After an initial learning phase, the additional input 34 can be omitted. However, it should still be possible for the user to enter input 34, for example, if the automatically predefined target image impression classes 9 are deemed unsuitable for adjusting the image impression as needed. In this case, if user input 34 is entered, it can be used to further train the default algorithm 10.

[0085] Fig. 6 Figure 7 schematically shows an example of how an image 7 is modified by the modification algorithm 8. The original image 7 shows a multitude of features 36 to 43, which are recognizable with approximately the same clarity. If a user now wants to emphasize feature 37, for example, for a specific diagnostic purpose or for visualization for a patient, target image impression classes can be specified for the segments that comprise the individual features 36 to 43, so that feature 37 is highlighted, for example, by being displayed with particularly strong contrast and sharpness. Features 36 and 38 can, for example, be blurred so that they still provide the viewer with information about the location of feature 37, but do not distract from it. The relatively small features 39 to 43 may not be recognizable or barely recognizable in the modified image data 12.To nevertheless indicate to a user that these features are present and can be made visible or highlighted by selecting a different image modification, the image areas 44 to 48, in which the features 39 to 43 that are not or hardly recognizable in the modified image data 12 are located, can be marked, for example by surrounding the corresponding areas with a colored dashed line.

[0086] Fig. 7 Figure 51 shows an X-ray system 49 comprising an X-ray device 50, namely a computed tomography scanner, and a processing device 51. The processing device is configured to process the source data provided by the X-ray device 50 in order to display images, in particular cross-sectional images, on a display device 54 for a user. As required, the processing device 51 can modify the images using a modification algorithm, as described above, to specify target image impression classes for individual image segments or segment types. This modification can occur even during the initial display of the image. To achieve this, for example, a similar approach to that described in relation to [reference to] can be used. Fig. 5It was discussed that the training of the default algorithm 10 is carried out in such a way that specifying a display task 33 is not necessary. If a user is not satisfied with the resulting modified image, or if, for example, they want to display a different image modification for a different diagnostic purpose, they can specify a corresponding display task or a desired target impression class for the entire image or for segments of the image using a control 55.

[0087] The described method can be implemented by a computer program that can be loaded into a memory 52 of the processing unit 51. The method steps can be executed by a processor 53. The computer program can be provided on an electronically readable data carrier, for example, a CD-ROM, DVD, hard drive, USB stick, or similar.

[0088] Although the invention has been illustrated and described in detail by the preferred embodiment, the invention is not limited by the disclosed examples and other variations can be derived by the person skilled in the art without leaving the scope of protection of the invention.

Claims

1. Computer-implemented method for adapting an image impression of an image (7), in particular of an image (7) acquired in the context of medical imaging, including the steps: - providing an image (7) or input data from which the image (7) is acquired, - specifying a respective ideal image impression class (9) for at least one image segment (16, 17) of the image (7) that has been specified directly or by specifying at least one segment type, wherein a connection between the image data in the respective image segment (16, 17) and an assigned image impression class (14) is specified by a classification algorithm (13), - modification of the image (7) or of the input data by a modification algorithm (8), in order to adapt the image impression class (14) assigned to the resulting modified image data (12, 18, 19) pertaining to the respective image segment (16, 17) to match the respective ideal image impression class (9), wherein at least one modification parameter of the modification algorithm (8) is or becomes specified as a function of the classification algorithm (13), - wherein different ideal image impression classes (9) are specified for different image segments (16, 17) and / or different segment types, characterised in that - the image (7) or the input data is or are acquired from original data (1) by a processing algorithm (15), which is dependent on at least one processing parameter (3, 6), wherein the image impression class (14) of the respective image segment (16, 17) and / or of the segment type is dependent on the processing parameter (3, 6), - and the processing algorithm (15) includes a reconstruction algorithm (2) for reconstructing three-dimensional volume data (4) from two-dimensional original data (1), in particular from X-ray images, wherein the reconstruction algorithm (2) depends on the processing parameter (3) or on at least one of the processing parameters (3, 6) and / or wherein the image (7) or the input data are two-dimensional image data generated by a mapping algorithm (5) from the volume data (4), wherein the mapping algorithm (5) is dependent on the processing parameter (6) or on at least one of the processing parameters (3, 6).

2. Method according to claim 1, characterised in that the modification parameter is or becomes determined by a method of machine learning.

3. Method according to one of the preceding claims, characterised in that at least one classification parameter of the classification algorithm (13) and the modification parameter are or become determined together by a method of machine learning.

4. Method according to claim 3, characterised in that, in the context of machine learning, a learning algorithm tries at the same time or alternately to select the modification parameter such that when the classification algorithm (13) is applied to an image segment, which has been specified directly or by specifying at least one segment type, of a resulting image, which is determined by applying the modification algorithm (13) to an input image (21), a specified ideal image impression class is determined, and tries to select the classification parameter such that when the classification algorithm (13) is applied to the at least one image segment of the resulting image and to an image segment of the input image (21) that is assigned to the image segment of the resulting image, the same image impression class (25, 27) is determined.

5. Method according to of the preceding claims, characterised in that the ideal image impression class (9) and / or the at least one image segment (16, 17) and / or the at least one segment type, for which the ideal image impression class (9) is specified, is specified as a function of a user input (33, 34) by a user and / or of user information (29) identifying the user and / or of the image (7) and / or of the input data and / or of the original data (1) and / or of additional information (31, 32) relating to the imaging of the image (7) and / or to a patient shown in the image (7).

6. Method according to one of the preceding claims, characterised in that the specification of the ideal image impression class (9) and / or of the at least one image segment (16, 17) and / or segment type, for which the ideal image impression class (9) is specified, and / or a specification of ideal image impressions (9) that can be selected by the or by a user or have been recommended to them and / or image segments (16, 17) and / or segment types for which ideal image impressions (9) are selectable or recommended or specified ensues by means of a specifying algorithm (10), which is or becomes parameterised by machine learning.

7. Method according to one of the preceding claims, characterised in that the modification algorithm (8) is configured to automatically specify image segments (16, 17) and / or segment types and ideal image impression classes (9) that have been assigned in each case, as a function of a viewing task (33) that has been specified for the image (7).

8. Method according to one of the preceding claims, characterised in that the image (7) or the modified image data (12, 18, 19) are shown for the or a user, wherein at least one image region (44 - 48) is highlighted, which includes a feature (39 - 43), the representation of which depends on the image impression classification (14) of the image (7) or at least of one segment (16, 17) of the image (7) or on the viewing task (33).

9. Method according to one of the preceding claims, characterised in that the image (7) is acquired by means of a superimposed view, in particular by subtraction, from source images that describe the input data, wherein the modification algorithm (8) processes the source images as input data.

10. Processing device for processing images (7), characterised in that it is configured to carry out the method according to one of the preceding claims.

11. Computer program, which can be loaded directly into a memory (52) of a processing device (51), with programming means to carry out the steps in the method according to one of claims 1 to 9 when the program is run in the processing device (51).

12. Electronically readable data carrier with electronically readable control information stored thereon, which includes at least one computer program according to claim 11 and is embodied such that, when the data carrier is used in a processing device (51), it runs the method according to one of claims 1 to 9.