DETECTING ARTIFACTS IN SYNTHETIC MEDICAL IMAGES

Description

COPYRIGHT NOTICE

[0001] Part of the disclosure in this patent specification contains material that is subject to copyright protection. The copyright holder has no objection to the facsimile reproduction of the patent specification as it appears in a patent application or in the files of the patent office, but otherwise reserves all copyrights and rights of any kind. © 2023 Bayer AG TECHNICAL AREA

[0002] The present disclosure relates to the technical field of generating synthetic medical images. The subject matter of this disclosure includes a method, a computer system, and a computer-readable storage medium comprising a computer program for detecting artifacts in synthetic medical images. INTRODUCTION

[0003] Artificial intelligence is increasingly finding its way into medicine. Machine learning models are not only used to identify signs of disease in medical images of the human or animal body (see, for example, WO2018202541A1, WO2020229152A1), they are also increasingly used to generate synthetic (artificial) medical images.

[0004] WO2021052896A1 and WO2021069338A1, for example, describe methods for generating an artificial medical image that shows an examination area of ​​a subject in a first time interval. The artificial medical image is generated using a trained machine learning model based on medical images that show the examination area in a second time interval. This method can, for example, accelerate radiological examinations; instead of measuring radiological images over a longer period, measurements are taken only within a portion of the time interval, and one or more radiological images for the remaining portion of the time interval are predicted using the trained model.

[0005] WO2019 / 074938A1 and WO2022184297A1 describe, for example, methods for generating an artificial radiological image that shows an area of ​​examination of a subject after the administration of a standard amount of contrast agent, even though only a smaller amount of contrast agent than the standard amount has been administered. The standard amount is the amount recommended by the manufacturer and / or distributor of the contrast agent and / or the amount approved by a regulatory authority and / or the amount listed in the package leaflet for the contrast agent. The methods described in WO2019 / 074938A1 and WO2022184297A can therefore be used to reduce the amount of contrast agent.

[0006] Medical images generated by trained machine learning models may contain errors (see, e.g., K. Schwarz). et al.: On the Frequency Bias of Generative Models, https: / / doi.org / 10.48550 / arXiv.2111.02447).

[0007] Such errors can be problematic because a physician might make a diagnosis and / or initiate therapy based on the artificial medical images. When a physician reviews artificial medical images, they must know whether features in the images can be attributed to real features of the patient or whether they are artifacts resulting from errors in prediction by the trained machine learning model.

[0008] Documents US 2019 / 188852 A1, US 2019 / 333219 A1 and Richard Osuala et al.: "Data synthesis and adversarial networks: A review and meta-analysis in cancer imaging" reveal further state of the art. SUMMARY

[0009] The invention is defined in the attached claims.

[0010] These and other problems are addressed by the subject matter of the present disclosure.

[0011] A first subject of the present disclosure is a computer-implemented method for generating at least one confidence value for a synthetic image, comprising the steps: Receiving at least one image of an area of ​​investigation of an object under investigation; generating a multitude of sub-images based on the at least one received image, wherein each sub-image represents a sub-area of ​​the area of ​​investigation of the object under investigation, with sub-areas represented by different sub-images partially, but not completely, overlapping; for each generated sub-image: generating a synthetic sub-image at least partially based on the generated sub-image; determining the color values ​​of corresponding image elements of synthetic sub-images, wherein corresponding image elements represent the same sub-area of ​​investigation; determining a measure of dispersion of the color values ​​of corresponding image elements; determining a confidence value based on the measure of dispersion; outputting the confidence value.

[0012] Another subject of the present disclosure is a computer system comprising a processor and a memory that stores an application program configured to perform an operation when executed by the processor, wherein the operation comprises: receiving at least one image of a study area of ​​a study object, generating a plurality of sub-images based on the at least one received image, wherein each sub-image represents a sub-area of ​​the study area of ​​the study object, with sub-areas represented by different sub-images partially but not completely overlapping.For each generated sub-image: generating a synthetic sub-image at least partially based on the generated sub-image, determining the color values ​​of corresponding image elements of synthetic sub-images, where corresponding image elements represent the same sub-area of ​​the investigation area, determining a measure of dispersion of the color values ​​of corresponding image elements, determining a confidence value based on the measure of dispersion, and outputting the confidence value.

[0013] Another subject of the present disclosure is a computer-readable storage medium comprising a computer program that can be loaded into the main memory of a computer system and causes the computer system to perform the following steps: Receiving at least one image of an area of ​​investigation of an object under investigation; generating a multitude of sub-images based on the at least one received image, wherein each sub-image represents a sub-area of ​​the area of ​​investigation of the object under investigation, with sub-areas represented by different sub-images partially, but not completely, overlapping; for each generated sub-image: generating a synthetic sub-image at least partially based on the generated sub-image; determining the color values ​​of corresponding image elements of synthetic sub-images, wherein corresponding image elements represent the same sub-area of ​​investigation; determining a measure of dispersion of the color values ​​of corresponding image elements; determining a confidence value based on the measure of dispersion; outputting the confidence value. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Fig. 1 This shows, by way of example and schematically, the generation of partial images based on a received image of an area of ​​investigation of an object under investigation. Fig. 2 shows, by way of example and schematically, the creation of synthetic partial images based on partial images using a generative model and the merging of the synthetic partial images into a synthetic image. Fig. 3 An exemplary and schematic illustration of combining synthetic images into a unified synthetic image. Fig. 4 This shows, in an exemplary and schematic way, the combination of synthetic images into a unified synthetic image. Fig. 5a This is shown in an exemplary and schematic way, illustrating how sub-images are generated from each image of a plurality of received images and how synthetic sub-images are generated based on the generated sub-images. Fig. 5b shows, by way of example and schematically, how synthetic partial images can be joined together to form synthetic images and how the synthetic images can be combined to form a unified synthetic image. Fig. 6 This shows, in an exemplary and schematic way, how to determine a confidence level. Fig. 7 shows an embodiment of the method of the present disclosure in the form of a flowchart. Fig. 8 This shows an exemplary and schematic method for training a generative machine learning model. Fig. 9 shows an exemplary and schematic computer system according to the present disclosure. Fig. 10 shows, by way of example and schematically, another embodiment of the computer system of the present disclosure. DETAILED DESCRIPTION

[0015] The invention is explained in more detail below without distinguishing between the subject matter of the present disclosure (method, computer system, computer-readable storage medium). Rather, the following explanations are intended to apply analogously to all subject matter of the invention, regardless of the context in which they are described (method, computer system, computer-readable storage medium).

[0016] If the present description or the claims specify steps in a sequence, this does not necessarily mean that the invention is limited to the specified sequence. Rather, it is conceivable that the steps could also be carried out in a different sequence or even in parallel with one another, unless one step builds upon another, which necessarily requires that the building step be carried out subsequently (this will become clear in the specific case). The sequences mentioned are therefore preferred embodiments of the present disclosure.

[0017] The invention is explained in more detail at several points with reference to the drawings. These drawings depict specific embodiments with specific features and combinations of features, primarily for illustrative purposes; the invention should not be understood as being limited to the features and combinations of features shown in the drawings. Furthermore, statements made in the description of the drawings with regard to features and combinations of features are intended to be generally applicable, that is, transferable to other embodiments and not limited to the embodiments shown.

[0018] The present disclosure describes means for assessing the trustworthiness of a synthetic image of an area of ​​investigation of an object under investigation.

[0019] The term "trustworthiness" means that a person examining the synthetic image can trust that structures and / or morphologies and / or textures depicted in the synthetic image can be traced back to real structures and / or real morphologies and / or real textures of the area under investigation and are not artifacts.

[0020] The term "synthetic" means that the synthetic image is not the direct result of a measurement on a real object under investigation, but was artificially generated (calculated). However, a synthetic image can be based on images of a real object under investigation; that is, one or more images of a real object under investigation can be used to generate the synthetic image. Examples of synthetic images are described in the introduction and in the further description of this disclosure.

[0021] The "object of investigation" is preferably a human being or an animal, preferably a mammal, and most preferably a human being.

[0022] The "area of ​​investigation" is a part of the object being examined, for example, an organ of a human or animal such as the liver, brain, heart, kidney, lung, stomach, intestines, pancreas, thyroid gland, prostate, breast, or a part of these organs, or several organs, or another part of the object being examined. The area of ​​investigation may also include multiple organs and / or parts of multiple organs.

[0023] The term "image" refers to a data structure that represents a spatial distribution of a physical signal. This spatial distribution can have any dimension, e.g., 2D, 3D, 4D, or higher. The spatial distribution can have any shape, e.g., forming a grid and thereby defining pixels or voxels, where the grid can be irregular or regular. The physical signal can be any signal, e.g., proton density, echogenicity, transmittance, absorbance, relaxivity, information about rotating hydrogen nuclei in a magnetic field, color, grayscale, depth, surface area, or volume coverage.

[0024] The term "image" is preferably understood to mean a two-, three-, or higher-dimensional, visually perceptible representation of the area under investigation. The received image is usually a digital image. The term "digital" means that the image can be processed by a machine, generally a computer system. "Processing" refers to the known methods of electronic data processing (EDP).

[0025] A digital image can be processed, edited, and reproduced using computer systems and software, as well as converted into standardized data formats such as JPEG (Joint Photographic Experts Group graphics format), PNG (Portable Network Graphics), or SVG (Scalable Vector Graphics). Digital images can be visualized using suitable display devices, such as computer monitors, projectors, and / or printers.

[0026] In a digital image, image content is typically represented and stored using integers. In most cases, these are two- or three-dimensional images that are binary encoded and may be compressed. Digital images are usually raster graphics, in which the image information is stored in a uniform grid. Raster graphics consist of a grid-like arrangement of so-called picture elements (pixels) in the case of two-dimensional representations or volume elements (voxels) in the case of three-dimensional representations. For four-dimensional representations, the term doxel is often used. (dynamic voxel) used for the image elements. In higher-dimensional representations, or more generally, the term " n-xel" is used, where n indicates the respective dimension. In this disclosure, the term image element is used generally. Thus, an image element can be a pixel in the case of a two-dimensional representation, a voxel in the case of a three-dimensional representation, a doxel in the case of a four-dimensional representation, or a higher-dimensional image element in the case of a higher-dimensional representation.

[0027] Each element of an image is assigned a color value. The color value specifies how (e.g., in which color) the image element should be visually displayed (e.g., on a monitor or a printer).

[0028] The simplest case is a binary image, where each image element is represented as either white or black. Typically, the color value "0" represents "black" and the color value "1" represents "white".

[0029] In a grayscale image, each image element is assigned a grayscale level, ranging from black through a defined number of grayscale gradations to white. These grayscale levels are also referred to as grayscale values. The number of gradations can, for example, range from 0 to 255 (i.e., encompassing 256 grayscale levels / values), where the value "0" typically represents "black" and the highest grayscale value (in this example, the value 255) represents "white".

[0030] In a color image, the color encoding used for each pixel is defined, among other things, by the color space and color depth. In an image whose color is defined by the so-called RGB color space (RGB stands for the primary colors red, green, and blue), each pixel is assigned three color values: one for red, one for green, and one for blue. The color of a pixel is created by superimposing (additively mixing) these three color values. A single color value can be discretized into, for example, 256 distinguishable levels, called tonal values, which typically range from 0 to 255. The tonal value "0" of each color channel is usually the darkest shade. If all three color channels have a tonal value of 0, the corresponding pixel appears black; if all three color channels have a tonal value of 255, the corresponding pixel appears white.

[0031] Regardless of whether it is a binary image, a grayscale image, or a color image, in this disclosure the term "color value" is used to describe the color (including the "colors" "black" and "white," as well as all shades of gray) in which an image element is to be represented. A color value can therefore be a tonal value of a color channel, a shade of gray, or stand for "black" or "white."

[0032] A color value in an image (especially a medical image) typically represents the strength of a physical signal (so). It should be noted that the color value can also be a value for the physical signal itself.

[0033] There are numerous possible digital image formats and color encodings. For the sake of simplicity, this description assumes that the images in question are RGB raster graphics with a specific number of image elements. However, this assumption should in no way be considered limiting. Those skilled in image processing will understand how to apply the principles of this description to image files in other formats and / or where the color values ​​are encoded differently.

[0034] For the purposes of this disclosure, an "image" may also be one or more excerpts from a video sequence.

[0035] In a first step, at least one image of an area of ​​investigation of an object is received.

[0036] The term "receive" encompasses both the retrieval of images and the acceptance of images that are transmitted, for example, to the computer system of this disclosure. The at least one image may be received from a computed tomography scanner, a magnetic resonance imaging scanner, an ultrasound scanner, a camera, and / or another device for generating images. The at least one image may be read from a data storage device and / or transmitted by a separate computer system.

[0037] Preferably, the at least one received image is a two-dimensional or three-dimensional representation of an area of ​​investigation of an object under investigation.

[0038] In one embodiment of the present disclosure, the at least one received image is a medical photograph.

[0039] A "medical photograph" is a visual representation of an area of ​​examination of a human or animal that can be used for diagnostic and / or therapeutic purposes.

[0040] There are a variety of techniques that can be used to produce medical images; examples of such techniques include X-ray, computed tomography (CT), fluoroscopy, magnetic resonance imaging (MRI), ultrasound (sonography), endoscopy, elastography, tactile imaging, thermography, microscopy, positron emission tomography, optical coherence tomography (OCT), fundus photography, and others.

[0041] Examples of medical images include CT scans, X-rays, MRI scans, fluorescein angiography images, OCT scans, histological images, ultrasound images, fundus images and / or others.

[0042] The at least one image received can be a CT scan, MRI scan, ultrasound scan, OCT scan, and / or another representation of an area of ​​investigation of an object under investigation.

[0043] The at least one received image may also include representations of different modalities, e.g. a CT scan and an MRI scan.

[0044] Each received image comprises a multitude of image elements. Each image element within this multitude represents a sub-area of ​​the investigation area of ​​the object under investigation. The term "multitude of image elements" means at least 1000, preferably at least 10000, and even more preferably more than 100000. It is conceivable that a received image comprises one or more image elements that do not represent the investigation area of ​​the object under investigation, but rather another area, such as an adjacent and / or surrounding area.

[0045] In a first step, a multitude of sub-images are generated based on at least one image. The term "multitude of sub-images" means at least two, preferably at least ten, and even more preferably more than twenty sub-images.

[0046] Each sub-image represents a sub-area of ​​the object under investigation. Sub-areas represented by different sub-images partially, but not completely, overlap. In other words, there are at least two sub-images, each representing a sub-area of ​​the investigation, with the sub-areas partially, but not completely, overlapping. In other words, there are sub-areas of the investigation represented by several (at least two) sub-images, where the sub-images representing the same sub-area represent different further sub-areas.

[0047] Preferably, each sub-area of ​​the investigation area is represented by several (at least two) sub-images in different constellations with other sub-areas.

[0048] A sub-area of ​​the investigation area is represented by one or more image elements of a sub-image. Therefore, there are at least two sub-images that share at least one image element but differ in at least one image element.

[0049] Preferably, for each sub-image, there is at least one other sub-image with at least one common image element and at least one different image element. Even more preferably, for each sub-image, there are several other sub-images with at least one common image element and at least one different image element.

[0050] Preferably, the number of identical (common) image elements and / or different image elements is greater than 10, even more preferably greater than 100.

[0051] Preferably, for each image element of the at least one received image, there is a plurality of sub-images that also comprise this image element, wherein each sub-image of the plurality of sub-images differs from each other sub-image of the plurality of sub-images by at least one other image element.

[0052] The partial images can be generated, for example, by dividing the at least one received image into partial images by cutting, whereby the cutting lines (in the case of 2D images) or cutting surfaces (in the case of 3D images) run through the at least one received image at different angles.

[0053] One embodiment for generating partial images is described below using the following examples: Fig. 1 explained in more detail, without limiting the invention to the in Fig. 1 to limit the embodiment shown. Statements made regarding the embodiments depicted in the drawings of this disclosure shall apply analogously to all other embodiments.

[0054] Fig. 1 Figure 1 shows a received image Ii of an area under investigation of an object. The received image Ii comprises a multitude of image elements; three image elements IE1, IE2, and IE3 are represented as points.

[0055] A multitude of copies I1, I2, I3, and I4 are generated from the received image Ii. One of the copies I1, I2, I3, or I4 can be the received image Ii itself. The number of copies generated for each received image typically corresponds to the number of synthetic images that can be generated and then combined into a unified synthetic image.

[0056] Each copy I1, I2, I3, and I4 is subdivided into a multitude of sub-images. In this example, this is achieved by dividing each copy by making cuts along cutting planes. These cutting planes pass through copies I1, I2, I3, and I4 at different angles. For example, in the case of copy I2, the cutting planes run parallel to the xz-plane, resulting in sub-images PI21, PI22, PI23, PI24, PI25, and PI26. In the case of copy I4, the cutting planes run parallel to the yz-plane, resulting in sub-images PI41, PI42, PI43, PI44, PI45, PI46, and PI47. The subdivision of copies I1 and I3 is as follows: Fig. 1 only indicated; section planes are shown; the partial images resulting from the corresponding sections along the section planes are in Fig. 1 not explicitly shown.

[0057] In the present example, the copies are divided into partial images by flat surfaces. This is a preferred embodiment of the present invention. However, it is also possible to divide the copies into partial images by curved surfaces or by other cuts.

[0058] In the present example, the cutting planes are equidistant from each other in the individual copies. This is a preferred embodiment of the present invention. However, it is also possible for the spacing of the cutting planes (or, more generally, the cutting surfaces) to vary.

[0059] In the case of copies I2 and I4, all sub-images are the same size (they have the same number of image elements). In other words, the portion of the area under investigation that each represents is the same size for all sub-images. In the case of copies I1 and I3, only some of the sub-images are the same size. It is possible to make all sub-images the same size by padding smaller sub-images with zeros (or another value).

[0060] The number of sub-images generated from each copy can be the same or different. In this example, the number of sub-images generated from each copy varies. In the case of copy I1, there are six; in the case of copy I2, there are six; in the case of copy I3, there are nine; and in the case of copy I4, there are seven.

[0061] Preferably, the copies are divided into sub-images such that all resulting sub-images have the same size (i.e., number of image elements); this can be achieved in some cases by padding. This has the advantage that sub-images of the same size can always be supplied to the generative model.

[0062] The in Fig. 1 The depicted generation of partial images fulfills the above-mentioned requirements: There are at least two sub-images that represent the same sub-area of ​​the object under investigation, but each also represents a different sub-area: The sub-area represented by image element IE 1 is represented by both sub-image PI 25 and sub-image PI 41. However, each of sub-images PI 25 and PI 41 also represents other sub-areas of the investigation that are not represented by the other sub-image. For example, sub-image PI 25, with image element IE 3, represents a sub-area that is not represented by sub-image PI 41. Sub-image PI 47, with image element IE 3, represents a sub-area that is also represented by sub-image PI 25; however, with image element IE 2, it represents a sub-area that is not represented by sub-image PI 25.The sub-images PI 25 and PI 47 share the image element IE 3, but differ in at least one other image element: sub-image PI 25, for example, includes the image element IE 1, which sub-image PI 47 does not, and sub-image PI 47 includes the image element IE 2, which sub-image PI 25 does not.

[0063] The in Fig. 1 The depicted generation of partial images also fulfills another condition of one of the aforementioned embodiments: For every sub-image, there exists at least one other sub-image with at least one shared image element and at least one differing image element. In the present example, for each sub-image, there are at least three sub-images with at least one shared image element and at least one differing image element.

[0064] In a next step, synthetic sub-images are generated based on the partial images. The generation of the synthetic sub-images is carried out using a model that is referred to in this disclosure as a generative model.

[0065] The generative model can be a trained machine learning model. A "machine learning model" can be understood as a computer-implemented data processing architecture. Such a model can receive input data and provide output data based on this input data and model parameters. Through training, such a model can learn a relationship between the input data and the output data. During training, model parameters can be adjusted to produce a desired output for a given input.

[0066] When training such a model, it is presented with training data from which it can learn. The trained machine learning model is the result of the training process. In addition to input data, the training data includes the correct output data (target data) that the model is to generate based on the input data. During training, patterns are recognized that map the input data to the target data.

[0067] During the training process, the input data for the training data is fed into the model, and the model generates output data. This output data is then compared to the target data. Model parameters are modified to reduce the deviations between the output and target data to a (defined) minimum. To modify the model parameters with a view to reducing these deviations, an optimization method such as gradient descent can be used.

[0068] The deviations can be analyzed using an error function (English: loss function) can be quantified. Such an error function can be used to measure an error (English: loss) to calculate for a given pair of output data and target data. The goal of the training process can be to modify (adjust) the parameters of the machine learning model so that the error is reduced to a (defined) minimum for all pairs of the training dataset.

[0069] If the output and target data are numbers, for example, the error function can be the absolute difference between these numbers. In this case, a large absolute error may mean that one or more model parameters need to be changed significantly.

[0070] For output data in the form of vectors, difference metrics between vectors such as the mean squared error, a cosine distance, a norm of the difference vector such as a Euclidean distance, a Chebyshev distance, an Lp norm of a difference vector, a weighted norm, or another type of difference metric of two vectors can be chosen as the error function.

[0071] For higher-dimensional outputs, such as two-dimensional, three-dimensional, or even higher-dimensional outputs, an element-wise difference metric can be used. Alternatively or additionally, the output data can be transformed before calculating an error value, for example, into a one-dimensional vector.

[0072] Fig. 8 Figure 1 schematically shows an example of training a machine learning model and is described in more detail below.

[0073] The generative model can include one or more algorithms that specify how a synthetic sub-image can be generated based on one or more sub-images. Typically, one or more sub-images are fed into the generative model, and the model generates a synthetic sub-image based on the one or more sub-images, model parameters, and, if necessary, other input data.

[0074] The generative model can be a machine learning model, such as the one described in one of the following publications: WO2019 / 074938A1, WO2022 / 253687A1, WO2022 / 207443A1, WO2022 / 223383A1, WO20227184297A1, WO2022 / 179896A2, WO2021 / 069338A1, EP 22209510.1, EP23159288.2, PCT / EP2023 / 053324, PCT / EP2023 / 050207, CN110852993A, CN110853738A, US2021150671A1, arXiv:2303.15938v1, doi:10.1093 / jrr / rrz030.

[0075] The generative model can, for example, be configured to generate a synthetic radiological image after the application of a second amount of contrast medium, based on at least one received radiological image of the examination area before and / or after the application of a first amount of contrast medium, wherein the second amount is preferably larger than the first amount (as described, for example, in WO2019 / 074938A1 or WO2022184297A1).

[0076] The at least one radiological image received can be, for example, an MRI image, and the synthetic radiological image can be a synthetic MRI image.

[0077] The at least one radiological image obtained may also include a CT scan before and / or after the administration of a first amount of an MRI contrast agent, and the synthetic radiological image may be a synthetic CT scan after the administration of a second amount of an MRI contrast agent, wherein the second amount is preferably larger than the first amount and preferably larger than the standard amount of MRI contrast agent for MRI examinations (as described, for example, in PCT / EP2023 / 053324). The "standard amount" is usually the amount recommended by the manufacturer and / or distributor of the contrast agent and / or the amount approved by a regulatory authority and / or the amount listed in a package leaflet for the contrast agent.

[0078] The generative model can, for example, be configured to generate a synthetic radiological image based on at least one radiological image of an examination area of ​​an object, representing the examination area in a first time period before and / or after the application of a contrast agent, which represents the examination area in a second time period after the application of the contrast agent (as described, for example, in WO2021052896A1).

[0079] In contrast to the methods described in the publications cited above, the present disclosure does not only generate one synthetic image, but generates a plurality of synthetic images which can be combined in a subsequent step to form a single synthetic image – referred to in this description as the united synthetic image.

[0080] The term "plurality of synthetic images" means at least two, preferably at least five, even more preferably at least ten synthetic images.

[0081] The synthetic images of the majority of synthetic images differ from one another in that they are generated, at least partially, based on different sub-images of the at least one received image. Sub-images differ in the constellation of image elements from which they are composed; different sub-images can contain the same image elements, but they exhibit these same image elements in different constellations with other image elements.

[0082] In other words, each synthetic image out of the majority of synthetic images is generated based on sub-images that exhibit a different constellation of image elements. The generative model is thus fed partially different input data; with the help of the generative model, the majority of synthetic images are generated based on this partially different input data.

[0083] From the differences between the synthetic images of the majority of synthetic images, a confidence score can be determined, indicating how reliable a unified synthetic image, obtained by combining the majority of synthetic images, is. Details on how to determine the confidence score are described further below.

[0084] Fig. 2 This section provides an exemplary and schematic illustration of the generation of synthetic partial images based on partial images using a generative model, and the merging of these synthetic partial images into a single synthetic image. The partial images are those described in Fig. 1 shown partial images PI 21 , PI 22 , PI 23 , PI 24 , PI 25 and PI 26 .

[0085] Even if Fig. 2 It may give the impression that the sub-images PI 21 , PI 22 , PI 23 , PI 24 , PI 25 and PI 26 are from Fig. 1 While the sub-images PI 21, PI 22, PI 23, PI 24, PI 25, and PI 26 are fed to the generative model GM together, they are fed to the generative model GM separately (e.g., one after the other). The generative model GM is configured to generate a synthetic partial representation PS 2m based on a sub-image PI 2m, where m is an index that, in this example, ranges from the integers 1 to 6. In the present example, the synthetic sub-image PS 21 is generated based on the sub-image PI 21, the synthetic sub-image PS 22 is generated based on the sub-image PI 22, the synthetic sub-image PS 23 is generated based on the sub-image PI 23, the synthetic sub-image PS 24 is generated based on the sub-image PI 24, the synthetic sub-image PS 25 is generated based on the sub-image PI 25, and the synthetic sub-image PS 26 is generated based on the sub-image PI 26.

[0086] Each synthetic sub-image preferably represents the same part of the investigation area as the sub-image on which it was based. Thus, each synthetic sub-image preferably represents a different sub-area of ​​the investigation area.

[0087] In Fig. 2 It is further shown that the synthetic partial images PS 21, PS 22, PS 23, PS 24, PS 25 and PS 26 can be combined in a further step to form a synthetic image S2. The synthetic image S2 preferably represents the entire area under investigation (just like the at least one received image).

[0088] The procedure, which in Fig. 2 using the example of sub-images PS 21, PS 22, PS 23, PS 24, PS 25 and PS 26 of copy I 2 from Fig. 1 The process shown is carried out analogously for the partial images of the remaining copies I 1 , I 3 and I 4.

[0089] For each copy I1, I2, I3, and I4, a synthetic image S1, S2, S3, and S4 is obtained. These are in Fig. 3 depicted.

[0090] As in Fig. 3 In a schematic representation, the synthetic images S 1 , S 2 , S 3 and S 4 can be combined in a further step to form a unified synthetic image S.

[0091] The synthetic images S1, S2, S3, and S4 represent the same investigation area and preferably comprise the same number of image elements. Each sub-area of ​​the investigation area is thus represented four times by the synthetic images S1, S2, S3, and S4.

[0092] The combination of the synthetic images S 1 , S 2 , S 3 and S 4 into a unified synthetic image is based on the color values ​​of corresponding image elements.

[0093] Image elements that represent the same sub-area of ​​the investigation area are referred to in this disclosure as "mutually corresponding image elements" or simply "corresponding image elements". Corresponding image elements can, for example, be those image elements that have the same coordinates if the respective synthetic image is a raster graphic.

[0094] For each k-tuple of corresponding image elements in the generated synthetic images, the color values ​​are determined. This gives k the number of corresponding image elements. In the Fig. 3 In the example shown, four image elements of the synthetic images S 1 , S 2 , S 3 and S 4 always correspond to each other.

[0095] An average value (e.g., arithmetic mean, geometric mean, quadratic mean, or another average) can be calculated from the color values. The average color value for a tuple of corresponding image elements can be set as the color value of the corresponding image element in the combined synthetic image.

[0096] In the case of multiple color values ​​(e.g., three color values ​​as with RGB color values), an average value can be calculated for each color channel. The respective average value can then be set as the color value of the corresponding color channel of the associated image element in the combined synthetic image.

[0097] It is also conceivable to determine a maximum or minimum color value for corresponding image elements instead of an average value, and to assemble the combined synthetic image from the image elements with the respective maximum or minimum color values. Instead of maxima / minima, other statistical quantities can also be determined and used to generate the combined synthetic image.

[0098] Fig. 4 This shows, in an exemplary and schematic way, the combination of synthetic images into a unified synthetic image.

[0099] Three synthetic images, S1, S2, and S3, are shown. In the Fig. 4 The synthetic images S1, S2, and S3 shown are 2D raster graphics. Each of the synthetic images S1, S2, and S3 represents the same area of ​​investigation of a study object.

[0100] Each of the three synthetic images S1, S2, and S3 has a number of 10 - 10 = 100 image elements. The image elements are arranged in a grid; each row and each column is assigned a number, so that each image element can be uniquely identified by its coordinates (row value, column value).

[0101] The synthetic images S 1 , S 2 and S 3 are binary images, i.e. each image element is assigned either the color value "white" or the color value "black".

[0102] The combined synthetic image S is generated by combining the synthetic images S1, S2, and S3. This combination is based on corresponding image elements. Corresponding image elements each represent the same sub-area of ​​the investigation area of ​​the object under investigation. In this example, the coordinates of corresponding image elements are identical. For instance, the image element with coordinates (1,1) of synthetic image S1 corresponds to the image element with coordinates (1,1) of synthetic image S2 and to the image element with coordinates (1,1) of synthetic image S3. The image elements with coordinates (1,1) of synthetic images S1, S2, and S3 form a tuple of corresponding image elements.

[0103] For each tuple of corresponding image elements, the color values ​​are determined, and based on the determined color values, the color value of the corresponding image element of the combined synthetic image is determined.

[0104] In the present example, the synthetic images are combined to form the unified synthetic image according to the following rule: the color value of each image element of the unified synthetic image S corresponds to the color value of the majority of the color values ​​of the corresponding image elements of the synthetic representations S 1 , S 2 and S 3 .

[0105] The color value for the image element with coordinates (1,1) of synthetic representation S1 is, for example, "white". The color value for the corresponding image element with coordinates (1,1) of synthetic representation S2 is also "white". The color value for the corresponding image element with coordinates (1,1) of synthetic representation S3 is also "white". The majority of the corresponding image elements (namely all image elements) have the color value "white". Accordingly, the color value of the image element with coordinates (1,1) of the combined synthetic image is also set to "white".

[0106] For example, the color value for the image element with coordinates (1,4) of synthetic representation S1 is "white". The color value for the corresponding image element with coordinates (1,4) of synthetic representation S2 is "black". The color value for the corresponding image element with coordinates (1,4) of synthetic representation S3 is "white". The majority of the corresponding image elements have the color value "white". Accordingly, the color value of the image element with coordinates (1,4) of the combined synthetic image is set to "white".

[0107] For example, the color value for the image element with coordinates (7,10) of synthetic representation S1 is "black". The color value for the corresponding image element with coordinates (7,10) of synthetic representation S2 is also "black". The color value for the corresponding image element with coordinates (7,10) of synthetic representation S3 is "white". The majority of the corresponding image elements have the color value "black". Accordingly, the color value of the image element with coordinates (7,10) of the combined synthetic image is set to "black".

[0108] Other possibilities for combining individual synthetic images into a unified synthetic image are conceivable. For example, a machine learning model (e.g., an artificial neural network) can be trained to generate the unified synthetic image from the synthetic images of a plurality of synthetic images according to predefined criteria. If training data is available that includes not only synthetic images as input data but also images that can be used as target data, the machine learning model can be trained in a supervised learning process to combine synthetic images from a plurality of synthetic images into a unified synthetic image. Attention mechanisms (e.g.,...) can be used in this process. attention, (see e.g. arXiv:2203.14263) are used, in which, for example, different weights are assigned to the individual synthetic images of the majority of synthetic images when combined to form the combined synthetic image.

[0109] It is also possible that the method for generating a unified synthetic image is not the same for every tuple of corresponding image elements. Different methods for generating the unified synthetic image may be used for different sub-areas of the object under investigation. For example, a different method for combining the color values ​​of image elements representing a specific tissue, organ, and / or lesion may be used than for image elements representing a different tissue, organ, and / or area. Areas for which different methods for combining corresponding image elements apply can be identified, for example, using segmentation.

[0110] The term "segmentation" refers to the process of dividing an image into multiple segments, also known as image segments, image regions, or image objects. Segmentation is typically used to locate objects and boundaries (lines, curves, etc.) within images. In a segmented image, the located objects can be separated from the background, visually highlighted (e.g., with color), measured, counted, or otherwise quantified. Segmentation involves assigning each image element a identifier (e.g., a number) so that image elements with the same identifier share certain common characteristics, such as representing the same tissue (e.g., bone tissue, fat tissue, healthy tissue, diseased tissue (e.g., tumor tissue), muscle tissue, and / or the like) and / or the same organ.For corresponding image elements with a specific identifier, a specific calculation rule can then be used to combine their color values ​​to generate a unified synthetic representation; for corresponding image elements with a different (specific) identifier, a different (specific) calculation rule z can be used to combine their color values.

[0111] It is also possible to generate more than one combined synthetic image, e.g., two, three, four, or more than four. For example, a first combined synthetic image can be generated with the respective maxima of the color values, and a second combined synthetic image with the respective minima of the color values. A third combined synthetic image can also be generated with the average color values.

[0112] The combined synthetic image can be output (e.g., displayed on a monitor and / or printed) and / or stored in a data storage device and / or transmitted to a separate computer system, e.g., via a network connection. Based on the output combined synthetic image, a doctor can, for example, make a diagnosis and / or initiate therapy.

[0113] It is also possible that no unified synthetic image is generated and / or output. The analysis of corresponding image elements of synthetic images, as described below, may reveal that a unified synthetic image has low trustworthiness. For example, a determined trust value that correlates positively with trustworthiness may be lower than a predefined threshold. The determined trust value, and thus the trustworthiness, may be so low that no diagnosis should be made and / or therapeutic measures initiated based on the unified synthetic image. In such a case, a unified synthetic image may be worthless or even misleading and therefore dangerous. The generation and / or output of such a unified synthetic image with low trustworthiness can then be avoided.

[0114] In the Fig. 1, Fig. 2 and Fig. 3 In the example shown, exactly one unified synthetic image S is generated based on a single received image I i. However, it is possible to generate exactly one unified synthetic image S based on two or more received images.

[0115] In such a case, partial images are generated from each received image as described in this revelation. Partial images generated from different received images are then fed together to the generative model, and the generative model generates a synthetic partial image based on these input images. The partial images fed to the generative model preferably represent the same sub-area of ​​the investigation domain. The synthetic partial images can then be combined to form a plurality of synthetic images, and the plurality of synthetic images can be combined to form a unified synthetic image. This is described in Fig. 5a and Fig. 5b This is schematically illustrated using an example with two received images.

[0116] In Fig. 5a The schematic representation shows how sub-images are generated from each image of a plurality of received images, and how synthetic sub-images are created based on these generated sub-images. Fig. 5b The diagram schematically illustrates how the synthetic partial images are joined together to form synthetic images and how the synthetic images are combined to form a unified synthetic image.

[0117] Starting point of the in Fig 5a The method shown involves two received images, a first image I 1 and a second image I 2. Each image preferably represents the same area of ​​investigation of the same object under investigation.

[0118] Multiple copies are created of each received image. In the Fig. 5a In the example shown, three copies are generated of each received image; copies I11, I12, and I13 are generated from the first image I1, and copies I21, I22, and I23 are generated from the second image I2. It is possible that one of the copies I11, I12, or I13 is the first image I1 itself; likewise, it is possible that one of the copies I21, I22, or I23 is the second image I2 itself.

[0119] Each copy is subdivided into sub-images; each sub-image represents a sub-area of ​​the object under investigation. Sub-areas represented by different sub-images partially, but not completely, overlap. From copy I 11, the sub-images PI 111, PI 112, PI 113, PI 114, PI 115, and PI 116 are generated; from copy I 12, the sub-images PI 121, PI 122, PI 123, PI 124, PI 125, and PI 126 are generated; from copy I 13, the sub-images PI 131, PI 132, PI 133, PI 134, PI 135, and PI 136 are generated. From copy I 21, the sub-images PI 211, PI 212, PI 213, PI 214, PI 215 and PI 216 are generated; from copy I 22, the sub-images PI 221, PI 222, PI 223, PI 224, PI 225 and PI 226 are generated; from copy I 23, the sub-images PI 231, PI 232, PI 233, PI 234, PI 235 and PI 236 are generated.

[0120] Corresponding partial images, originating from different received images, are added together to a generative model (GM). "Corresponding partial images" are those that represent the same sub-area of ​​the investigation area.

[0121] The generative model GM is in Fig 5a For better understanding, it is shown three times; however, it is always the same generative model.

[0122] In the Fig. 5a In the example shown, sub-image PI 111 corresponds to sub-image PI 211, sub-image PI 112 to sub-image PI 212, sub-image PI 113 to sub-image PI 213, sub-image PI 114 to sub-image PI 214, sub-image PI 115 to sub-image PI 215, sub-image PI 116 to sub-image PI 216, sub-image PI 121 to sub-image PI 221, sub-image PI 122 to sub-image PI 222, sub-image PI 123 to sub-image PI 223, sub-image PI 124 to sub-image PI 224, sub-image PI 125 to sub-image PI 225, and sub-image PI 126. with the sub-image PI 226, the sub-image PI 131 with the sub-image PI 231, the sub-image PI 132 with the sub-image PI 232, the sub-image PI 133 with the sub-image PI 233, the sub-image PI 134 with the sub-image PI 234, the sub-image PI 135 with the sub-image PI 235 and the sub-image PI 136 with the sub-image PI 236.

[0123] The generative model (GM) is fed the corresponding sub-images and generates a synthetic sub-image based on these inputs. Each synthetic sub-image corresponds to the sub-images on which it was based; that is, it represents the same sub-area of ​​the study domain as the sub-images on which it was generated.

[0124] In the Fig. 5a In the example shown, the generative model GM generates the synthetic subimage TS 11 based on the sub-images PI 111 and PI 211; the synthetic sub-image TS 12 based on the sub-images PI 112 and PI 212; the synthetic sub-image TS 13 based on the sub-images PI 113 and PI 213; the synthetic sub-image TS 14 based on the sub-images PI 114 and PI 214; the synthetic sub-image TS 15 based on the sub-images PI 115 and PI 215; the synthetic sub-image TS 16 based on the sub-images PI 116 and PI 216; the synthetic sub-image TS 21 based on the sub-images PI 121 and PI 221; and the synthetic sub-image TS 22 based on the sub-images PI 122 and PI 222. The synthetic subimage TS 23 is based on the sub-images PI 123 and PI 223; the synthetic sub-image TS 24 is based on the sub-images PI 124 and PI 224; the synthetic sub-image TS 25 is based on the sub-images PI 125 and PI 225; and the synthetic sub-image TS 26 is based on the sub-images PI 126 and PI 226.Based on the partial images PI 131 and PI 231, the synthetic partial image TS 31; based on the partial images PI 132 and PI 232, the synthetic partial image TS 32; based on the partial images PI 133 and PI 233, the synthetic partial image TS 33; based on the partial images PI 134 and PI 234, the synthetic partial image TS 34; based on the partial images PI 135 and PI 235, the synthetic partial image TS 35; and based on the partial images PI 136 and PI 236, the synthetic partial image TS 36.

[0125] The synthetic partial images, which were generated based on the same subdivisions of the copies, are then combined into synthetic images in a next step.

[0126] In the Fig. 5b In the example shown, the sub-images TS 11, TS 12, TS 13, TS 14, TS 15 and TS 16 are combined to form the synthetic image S 1; the sub-images TS 21, TS 22, TS 23, TS 24, TS 25 and TS 26 are combined to form the synthetic image S 2 and the sub-images TS 31, TS 32, TS 33, TS 34, TS 35 and TS 36 are combined to form the synthetic image S 3.

[0127] The synthetic images S1, S2, and S3 are combined in a further step to form a unified synthetic image S. This process corresponds to that described in Fig. 3 and Fig. 4 The process described. The combined synthetic image S can be output (e.g., displayed on a monitor and / or printed) and / or stored in a data storage device and / or transferred to a separate computer system, e.g., via a network connection.

[0128] The following section describes in more detail how to determine at least one confidence level.

[0129] The at least one trust value can be a value indicating the degree to which one can trust a synthetic image (e.g., the combined synthetic image). The trust value can correlate positively with the trustworthiness of the synthetic image; that is, if the trust value is low, the trustworthiness is also low, and if the trust value is high, the trustworthiness is also high. However, it is also possible for the trust value to correlate negatively with the trustworthiness; that is, if the trust value is low, the trustworthiness is high, and if the trust value is high, the trustworthiness is low.In the case of a negative correlation, one can speak of an uncertainty value instead of a confidence value: if the uncertainty value is high, then the synthetic image is subject to a high degree of uncertainty; it is possible that the synthetic image contains one or more artifacts; it is possible that structures and / or morphologies and / or textures in the synthetic image have no corresponding reality, i.e., that structures and / or morphologies and / or textures in the synthetic image cannot be traced back to real structures and / or real morphologies and / or real textures in the area under investigation.Conversely, a low uncertainty value indicates that the synthetic image has low uncertainty; features in the synthetic image correspond to reality; the synthetic image can be trusted; a medical diagnosis can be made based on the synthetic image and / or medical therapy can be initiated based on the synthetic image.

[0130] A trust value that correlates positively with trustworthiness can, in principle, also be transformed into a trust value that correlates negatively with trustworthiness (an uncertainty value), for example, by taking the reciprocal. Conversely, a trust value that correlates negatively with trustworthiness (an uncertainty value) can also be transformed into a trust value that correlates positively with trustworthiness.

[0131] At least one confidence level can be determined based on corresponding image elements of synthetic representations.

[0132] For each k The color values ​​are determined from tuples of corresponding image elements in the generated synthetic images. This process gives k the number of corresponding image elements. In the Fig. 4 In the example shown, three image elements of the synthetic images S 1 , S 2 and S 3 correspond to each other.

[0133] The more the color values ​​of corresponding image elements differ in the synthetic images, the greater the impact on which sub-images the generation of a synthetic image is based. However, if the sub-images on which a synthetic image is based make a significant difference, then the synthetic image carries a certain degree of uncertainty; its reliability decreases the greater the differences.

[0134] Therefore, the extent to which color values ​​of corresponding image elements differ can be used as a measure of trustworthiness / uncertainty: the more color values ​​of corresponding image elements differ, the lower the trustworthiness, the higher the uncertainty; the less color values ​​of corresponding image elements differ, the lower the uncertainty, the higher the trustworthiness.

[0135] The trustworthiness / uncertainty can therefore be determined for each tuple of corresponding image elements of synthetic images of the plurality of synthetic images and then represents the trustworthiness / uncertainty (i) of each synthetic image of the plurality of synthetic images, (ii) of the totality of synthetic images of the plurality of synthetic images and (iii) of the combined synthetic image.

[0136] In other words, a confidence value can be determined for each individual image element of the combined synthetic image, indicating how much one can trust the color value of the image element.

[0137] One such confidence level could be the spread of the tuple of corresponding image elements. The spread is defined as the difference between the largest and smallest values ​​of a variable. Therefore, for each tuple of corresponding image elements, a maximum and a minimum color value can be determined, and the difference between the maximum and minimum color values ​​can be calculated. The result is the spread of the color values ​​of the tuple of corresponding image elements, which can be used as a confidence level.

[0138] If there is more than one color value (for example, three color values, as in the case of images whose color values ​​are specified according to the RGB color model), a maximum and a minimum color value can be determined for each color channel, and the difference for each color channel can be calculated. This results in three ranges. The respective range for each color channel can be used as a separate confidence value; it is also possible to combine the ranges of the color channels into a single value; it is possible to use the maximum range as the confidence value; it is possible to calculate an average value (e.g.,to use the arithmetic mean, the geometric mean, the quadratic mean or another mean) of the spreads as a confidence value; it is possible to use the length of the vector that the spreads specify in a three-dimensional space (or a higher-dimensional space when using more than three color channels) as a confidence value; other possibilities are conceivable.

[0139] A confidence level for a tuple of corresponding image elements can also be the variance and / or the standard deviation of the color values ​​of the corresponding image elements. The variance is defined as the mean squared deviation of a variable from its expected value; the standard deviation is defined as the square root of the variance.

[0140] A confidence level can also be another measure of dispersion, such as the sum of squared deviations, the coefficient of variation, the mean absolute deviation, a quantile interval, an interquantile interval, the mean absolute distance from the median, the median of the absolute deviations, and / or the geometric standard deviation. It is also possible for there to be more than one confidence level for a tuple of corresponding image elements.

[0141] It is also possible that the method for calculating a confidence score is not the same for every tuple of corresponding image elements. Different methods for calculating confidence scores may be used for different sub-areas of the object under investigation. For example, a different method may be used to calculate a confidence score for image elements representing a specific tissue, organ, and / or lesion than for image elements representing a different tissue, organ, and / or area. Areas for which different rules apply to calculating confidence scores can be identified, for example, by means of segmentation.

[0142] In segmentation, each image element can be assigned a identifier (e.g., a number) so that image elements with the same identifier share certain common characteristics, such as representing the same tissue (e.g., bone tissue, adipose tissue, healthy tissue, diseased tissue (e.g., tumor tissue), muscle tissue, and / or the same organ). A specific formula can then be used to calculate a confidence level for corresponding image elements with one specific identifier; a different formula can be used for corresponding image elements with a different (specific) identifier.

[0143] The confidence values ​​determined for tuples of corresponding image elements can be output (e.g. displayed on a monitor or printed on a printer), stored in a data storage device and / or transmitted to a separate computer system, e.g. via a network.

[0144] The confidence values ​​determined for tuples of corresponding image elements can also be represented graphically.

[0145] In addition to the combined synthetic image, another representation of the investigation area can be output (e.g., displayed on a monitor) that shows the level of trustworthiness for each image element. Such a representation is also referred to in this description as the trust representation. The trust representation preferably has the same dimensions and size as the combined synthetic image; each image element of the combined synthetic image is preferably assigned a corresponding image element in the trust representation.

[0146] Using such a trust representation, a user (e.g., a physician) can determine the degree of confidence in the color value of each individual image element. The trust representation can be displayed partially or completely overlaid with the combined synthetic image and / or a received image. The overlaid display can be configured so that the user can toggle it on and off. For example, the user can view the combined synthetic image and / or a received image layer by layer, as is common practice with computed tomography, magnetic resonance imaging, and other three-dimensional or higher-dimensional representations. For each layer, the user can display the corresponding layer of the trust representation to check whether image elements in that layer, which depict structures, morphologies, and / or textures in the combined synthetic representation, are trustworthy or uncertain.In this way, the user can recognize the risk that the structures, morphologies and / or textures are real properties of the area under investigation or artifacts.

[0147] Image elements with low trustworthiness (high uncertainty) can be displayed brightly and / or in a signal color (e.g., red, orange, or yellow), while image elements with high trustworthiness (low uncertainty) can be displayed darkly or in a less conspicuous or calming color (e.g., green or blue). It is also possible to display only those image elements in an overlay where the trust value exceeds or falls below a predefined threshold. For example, if the trust value correlates positively with trustworthiness, only those image elements in the trust representation whose trust value is below a predefined threshold can be displayed; in such a case, a user (e.g., a doctor) is only made aware of those image elements they should probably not trust.

[0148] It is also possible to determine confidence levels for sub-areas (sub-images) of the combined synthetic image (e.g., layers within the combined synthetic image) and / or for the entire combined synthetic image. Determining such confidence levels for sub-areas or entire images can be based on the confidence levels of the image elements from which they are composed. For example, to determine the confidence level of a layer, all confidence levels of the image elements located in that layer can be considered. It is also possible to consider neighboring image elements (e.g., image elements in the layer above and / or below the layer under consideration). A confidence level for a sub-area or the entire area can be determined, for example, by averaging (e.g., arithmetic mean, geometric mean, root mean square, or another type of mean).It is also possible to determine the maximum value (e.g., in the case of a trust value that is negatively correlated with trustworthiness) or the minimum value (e.g., in the case of a trust value that is negatively correlated with trustworthiness) of the trust values ​​of the image elements in a sub-area or the entire area and to use this as the trust value of the sub-area or the entire area. Further possibilities for determining a trust value for a sub-area or the entire area based on the trust values ​​of individual image elements are conceivable.

[0149] Such a confidence level for a subset or the entire area can also be output (e.g., displayed on a monitor or printed), stored in a data repository, and / or transmitted to a separate computer system. It can also be represented visually (e.g., using color), as described for the individual confidence levels.

[0150] If a trustworthiness value for a subset or the entire area, which correlates positively with trustworthiness, is lower than a predefined threshold, then it is possible that the corresponding subset or entire area should not be trusted. It is possible that such a subset or the entire area, as described above, will not be displayed at all (e.g., not shown at all), or that it will be displayed with a warning that the user should be cautious when interpreting the displayed data, as the displayed data is uncertain.

[0151] It is also possible that the user of the computer system / computer program of the present disclosure is given the ability, via a user interface, to navigate within the combined synthetic image to sub-areas that exhibit low trustworthiness. For example, the sub-areas with the lowest trustworthiness can be displayed to the user in a list (e.g., in the form of a list with a number). p (in sub-areas that exhibit the lowest trust value positively correlated with trustworthiness, where p is a positive integer). By clicking on a list entry, the user can be shown the corresponding sub-area in the form of a combined synthetic image, a trust map, and / or a received image and / or a section thereof.

[0152] Fig. 6 This illustrates, in a schematic and exemplary manner, how to determine at least one trust value. Determining this at least one trust value is based on corresponding image elements already present in Fig. 4 shown synthetic images S 1 , S 2 and S 3 .

[0153] A confidence value is determined for each tuple of corresponding image elements. In a first step, the color values ​​of all image elements are determined. In this example, the color "black" is assigned the color value "0" and the color "white" the color value "1", as is generally the case.

[0154] The confidence level is calculated as the dispersion for each tuple of corresponding image elements of the synthetic images S 1 , S 2 and S 3.

[0155] The color value for the image element with coordinates (1,1) of the synthetic representation S1 is, for example, "1" (white). The color value for the corresponding image element with coordinates (1,1) of the synthetic representation S2 is also "1" (white). The color value for the corresponding image element with coordinates (1,1) of the synthetic representation S3 is also "1" (white). Therefore, the range for the tuple of corresponding image elements is 1 - 1 = 0.

[0156] The color value for the image element with coordinates (1,4) of the synthetic representation S1 is, for example, "1" (white). The color value for the corresponding image element with coordinates (1,4) of the synthetic representation S2 is "0" (black). The color value for the corresponding image element with coordinates (1,4) of the synthetic representation S3 is "1" (white). The range for the tuple of corresponding image elements is therefore 1 - 0 = 1.

[0157] The color value for the image element with coordinates (7,10) of synthetic representation S1 is, for example, "0" (black). The color value for the corresponding image element with coordinates (7,10) of synthetic representation S2 is also "0" (black). The color value for the corresponding image element with coordinates (7,10) of synthetic representation S3 is "1" (white). The range for the tuple of corresponding image elements is therefore 1 - 0 = 1.

[0158] The confidence levels are listed in the CV table.

[0159] The trust values ​​determined in this way correlate negatively with trustworthiness.

[0160] A trust representation can be determined based on the trust values. In the Fig. 6 In the example shown, the color value of each image element in the trust representation SR is set to the corresponding trust value of the tuple of related image elements. For example, the image element with coordinates (1,1) is assigned the color black in the trust representation, while the image elements with coordinates (1,4) and (7,10) are assigned the color white. Using the trust representation SR, a user (e.g., a doctor) can immediately recognize which image elements are considered safe (black) and which are considered unsafe (white). Users should place less trust in areas where many white image elements appear in the trust representation SR.

[0161] Fig. 7 shows an embodiment of the method of the present disclosure in the form of a flowchart.

[0162] The procedure (100) comprises the following steps: (110)Receiving at least one image of an area of ​​investigation of an object under investigation, (120)Generating a plurality of sub-images based on the at least one received image, wherein each sub-image represents a sub-area of ​​the area of ​​investigation of the object under investigation, wherein sub-areas represented by different sub-images partially but not completely overlap, (130)for each generated sub-image: generating a synthetic sub-image at least partially based on the generated sub-image, (140)Determining color values ​​of corresponding image elements of synthetic sub-images, wherein corresponding image elements represent the same sub-area of ​​investigation, (150)Determining a measure of dispersion of the color values ​​of corresponding image elements, (160)Determining a confidence value based on the measure of dispersion, (170)Outputting the confidence value.

[0163] As described, the generative model described in this description can be a trained machine learning model. Fig. 8 shows an exemplary and schematic method for training such a machine learning model.

[0164] The generative model GM is trained using training data TD. For each reference object of a plurality of reference objects, the training data TD comprises (i) at least one reference image of the reference area of ​​the reference object in at least one first state as input data, and (i) a reference image of the reference object in at least one state different from the first state. The term "plurality of reference objects" preferably means more than 10, and even more preferably more than 100, reference objects.

[0165] The term "reference" is used here to distinguish the training phase from the phase of using the trained model to generate synthetic images.

[0166] A "reference image" is an image used to train the model. The "reference object" is the object from which the reference image is derived. The reference object is usually, like the subject of the study, an animal or a human, preferably a human. The reference area is a part of the reference object. Preferably, the reference area is the same part as the area of ​​investigation of the subject of the study.

[0167] The term "reference" otherwise has no restrictive meaning. Statements made in this description regarding the at least one received image apply analogously to every reference image; statements made in this description regarding the object of investigation apply analogously to every reference object; statements made in this description regarding the area of ​​investigation apply analogously to the reference area.

[0168] In the Fig. 8 The example shown depicts only one set of training data (TD) for a reference object; typically, training data (TD) comprises a multitude of these datasets for a multitude of reference objects. In the example shown in Fig. 8 In the example shown, the training data TD includes a first reference image RI 1, a second reference image RI 2 and a third reference image RI 3.

[0169] The first reference image, RI 1, represents the reference range of the reference object in a first state; the second reference image, RI 2, represents the reference range of the reference object in a second state; the third reference image, RI 3, represents the reference range of the reference object in a third state. The first, second, and third states are usually distinct from one another. For example, a state might represent an amount of contrast agent that is being or has been introduced into the reference range. For example, a state might represent a point in time before and / or after the application of a contrast agent.

[0170] For example, the first reference image RI 1 can represent the reference range without or after administration of a first amount of contrast agent, the second reference image RI 2 the reference range after administration of a second amount of the contrast agent, and the third reference image RI 3 the reference range after administration of a third amount of the contrast agent. The first amount can be smaller than the second amount, and the second amount can be smaller than the third amount (see, e.g., WO2019 / 074938A1, WO2022184297A1).

[0171] For example, the first reference image RI 1 can represent the reference range before or in a first time period after application of a contrast agent, the second reference image RI 2 the reference range in a second time period after application of the contrast agent, and the third reference image RI 3 the reference range in a third time period after application of the contrast agent (see e.g. WO2021052896A1, WO2021069338A1).

[0172] The first reference image RI 1 and the second reference image RI 2 serve in the Fig. 8 The example shown serves as input data; it is fed into the generative model GM. The generative model GM is configured to generate a synthetic image S based on the first reference image RI 1 and the second reference image RI 2, and based on model parameters MP. The synthetic image S should be as close as possible to the third reference image RI 3. That is, the third reference image RI 3 functions in the Fig. 8 shown example as target data (ground truth).

[0173] The synthetic image S generated by the generative model GM is compared with the third reference image RI 3. An error function LF is used to quantify deviations between the synthetic image S and the third reference image RI 3. For each pair of a synthetic image and a third reference image, an error value can be calculated using the error function LF.

[0174] In an optimization procedure, the error value and thus the deviations between the synthetic image S generated by the generative model and the third reference image RI 3 can be reduced by modifying model parameters MP.

[0175] The process is repeated for a large number of reference objects.

[0176] If the error values ​​reach a predefined minimum, or if they cannot be further reduced by modifying model parameters, the training can be terminated. The trained model can then be saved, transferred to a separate computer system, and / or used to generate synthetic images of (new) objects (objects of investigation).

[0177] Fig. 9 shows an exemplary and schematic computer system according to the present disclosure.

[0178] A "computer system" is a system for electronic data processing that processes data using programmable instructions. Such a system typically comprises a "computer," the unit containing a processor for performing logical operations, as well as peripherals.

[0179] In computer technology, "peripherals" refers to all devices connected to a computer that are used to control the computer and / or as input and output devices. Examples include monitors (screens), printers, scanners, mice, keyboards, drives, cameras, microphones, speakers, etc. Internal ports and expansion cards are also considered peripherals in computer technology.

[0180] The in Fig. 9 The computer system shown (10) comprises a receiving unit (11), a control and computing unit (12) and an output unit (13).

[0181] The control and computing unit (12) serves to control the computer system (10), to coordinate the data flows between the units of the computer system (10) and to perform calculations.

[0182] The control and computing unit (12) is configured: to cause the receiving unit (11) to receive at least one image of an area of ​​investigation of an object under investigation, to generate a plurality of sub-images based on the at least one received image, wherein each sub-image represents a sub-area of ​​the area of ​​investigation of the object under investigation, and wherein sub-areas represented by different sub-images partially, but not completely, overlap; to generate a synthetic sub-image for each generated sub-image, at least partially, based on the generated sub-image; to determine the color values ​​of corresponding image elements of synthetic sub-images, wherein corresponding image elements represent the same sub-area of ​​investigation; to determine a measure of dispersion of the color values ​​of corresponding image elements; to determine a confidence value based on the measure of dispersion; and to cause the output unit to output the confidence value.

[0183] Fig. 10 Figure 10 shows an exemplary and schematic embodiment of the computer system. The computer system (10) comprises a processing unit (21) connected to a memory (22). The processing unit (21) and the memory (22) form a control and arithmetic unit, as described in Figure 10. Fig. 9 shown.

[0184] The processing unit (21) (English: processing unit)The processing unit (21) may comprise one or more processors alone or in combination with one or more memories. The processing unit (21) may be ordinary computer hardware capable of processing information such as digital images, computer programs, and / or other digital information. The processing unit (21) typically consists of an arrangement of electronic circuits, some of which may be implemented as an integrated circuit or as several interconnected integrated circuits (an integrated circuit is sometimes referred to as a "chip"). The processing unit (21) may be configured to execute computer programs, which may be stored in a working memory of the processing unit (21) or in the memory (22) of the same or another computer system.

[0185] The memory (22) can be ordinary computer hardware capable of storing information such as digital images (e.g., representations of the study area), data, computer programs, and / or other digital information, either temporarily and / or permanently. The memory (22) can be volatile and / or non-volatile and can be permanently installed or removable. Examples of suitable memory include RAM (Random Access Memory), ROM (Read-Only Memory), a hard disk, flash memory, a removable computer disk, an optical disc, a magnetic tape, or a combination of the above. Optical discs can include read-only compact discs (CD-ROM), read / write compact discs (CD-R / W), DVDs, Blu-ray discs, and similar media.

[0186] In addition to the memory (22), the processing unit (21) can also be connected to one or more interfaces (11, 12, 31, 32, 33) to display, transmit, and / or receive information. The interfaces can include one or more communication interfaces (11, 32, 33) and / or one or more user interfaces (12, 31). The one or more communication interfaces can be configured to send and / or receive information, for example, to and / or from an MRI scanner, a CT scanner, an ultrasound camera, other computer systems, networks, data storage devices, or the like. The one or more communication interfaces can be configured to transmit and / or receive information via physical (wired) and / or wireless communication links.The one or more communication interfaces may include one or more interfaces for connecting to a network, e.g., using technologies such as cellular, Wi-Fi, satellite, cable, DSL, fiber optic, and / or the like. In some examples, the one or more communication interfaces may include one or more near-field communication interfaces configured to connect devices using near-field communication technologies such as NFC, RFID, Bluetooth, Bluetooth LE, ZigBee, infrared (e.g., IrDA), or similar technologies.

[0187] The user interfaces may include a display (31). A display (31) may be configured to show information to a user. Suitable examples include a liquid crystal display (LCD), a light-emitting diode (LED) display, a plasma display (PDP), or the like. The user input interface(s) (11, 12) may be wired or wireless and may be configured to receive information from a user into the computer system (1), for example, for processing, storage, and / or display. Suitable examples of user input interfaces include a microphone, an image or video recording device (e.g., a camera), a keyboard or keypad, a joystick, a touch-sensitive surface (separate from or integrated into a touchscreen), or the like.In some examples, the user interfaces may include automatic identification and data capture (AIDC) technology for machine-readable information. This could include barcodes, radio frequency identification (RFID), magnetic stripes, optical character recognition (OCR), integrated circuit cards (ICC), and similar technologies. The user interfaces may also include one or more interfaces for communication with peripheral devices such as printers and the like.

[0188] One or more computer programs (40) can be stored in memory (22) and executed by the processing unit (21), which is programmed to perform the functions described in this description. The retrieval, loading, and execution of instructions from the computer program (40) can be sequential, with one instruction being retrieved, loaded, and executed at a time. However, the retrieval, loading, and / or execution can also be performed in parallel.

[0189] The computer system of the present disclosure can be implemented as a laptop, notebook, netbook and / or tablet PC; it can also be a component of an MRI scanner, a CT scanner or an ultrasound diagnostic device.

Claims

1. Computer-implemented method comprising: - Receiving at least one image (Ii) of an examination area of an examination object, - Generating a plurality of sub-images (PI21, PI22, PI23, PI24, PI23, PI26) based on the at least one received image (Ii), wherein each sub-image represents a sub-region of the examination area of the examination object, wherein sub-regions represented by different sub-images partially but not completely overlap, - For each generated sub-image (PI21, PI22, PI23, PI23, PI23, PI26): generating a synthetic sub-image (SI21, SI22, SI23, SI24, SI25, SI26) at least proportionally on the basis of the generated sub-image (PI21, PI22, PI23, PI24, PI25, PI26), - Determining color values of corresponding image elements of synthetic sub-images, wherein corresponding image elements represent the same sub-area of the examination area, - Determining a scattering measure of the color values of corresponding image elements, - Determining a confidence value on the basis of the scattering measure, - Outputting the confidence value.

2. The method according to claim 1, wherein a confidence value is determined for each tuple of corresponding image elements of the synthetic sub-images (SI21, SI22, SI23, SI24, SI23, SI26).

3. The method according to any one of claims 1 or 2, wherein the scattering measure is a scatter width, a standard deviation, a variance, a sum of squares of deviation, a coefficient of variation, a mean absolute deviation, a quantile distance, an interquantile distance, a mean absolute distance from a median, a median of absolute deviations and / or a geometric standard deviation of the color values of corresponding image elements, or is derived therefrom.

4. The method according to any one of claims 1 to 3, wherein for each sub-image there is at least one other sub-image having at least one common image element and at least one different image element.

5. The method according to any one of claims 1 to 4, wherein for each image element of the at least one received image (Ii) there is a plurality of sub-images which also comprise this image element, wherein each sub-image of the plurality of sub-images differs from each other sub-image of the plurality of sub-images by at least one other image element.

6. The method according to any one of claims 1 to 5, wherein generating the plurality of sub-images (PI21, PI22, PI23, PI24, PI25, PI26) comprises: - Generating a plurality of copies (I1, I2, I3, I4) of the at least one received image (Ii), - Dividing each copy into sub-images, wherein all sub-images of all copies differ from each other.

7. The method according to any one of claims 1 to 6, wherein generating the plurality of sub-images (PI21, PI22, PI23, PI24, PI25, PI26) comprises: - Generating a plurality of copies (I1, I2, I3, I4) of the at least one received image (Ii), - Dividing each copy into sub-images by cuts, wherein the cuts run differently through the copy for each copy.

8. The method according to any one of claims 1 to 7, further comprising: - Generating a combined synthetic image (S) based on the synthetic sub-images (SI21, SI22, SI23, SI24, SI25, SI26), wherein generating the combined synthetic image (S) comprises combining color values of corresponding image elements of the synthetic sub-images (SI21, SI22, SI23, SI24, SI25, SI26).

9. The method according to claim 8, wherein generating the combined synthetic image (S) comprises: - For each tuple of corresponding image elements of the synthetic sub-images (SI21, SI22, SI23, SI24, SI25, SI26): determining a mean color value by averaging the color values of the corresponding image elements and setting the mean color value as the color value of the corresponding image element of the combined synthetic image (S).

10. The method according to any one of claims 1 to 9, further comprising: - Outputting the combined synthetic image (S) and / or transmitting the combined synthetic image (S) to a separate computer system.

11. The method according to any one of claims 8 to 9, further comprising: - Determining a confidence value for one or more sub-regions of the combined synthetic image (S), and / or for the entire combined synthetic image (S), - Outputting the confidence value.

12. The method according to any one of claims 1 to 11, wherein the examination object is a human or animal, preferably a mammal, most preferably a human.

13. The method according to any one of claims 1 to 12, wherein the at least one received image (Ii) is at least one medical image and each synthetic image (SI21, SI22, SI23, SI24, SI25, SI26) and / or the combined synthetic image (S) is a synthetic medical image.

14. The method according to any one of claims 1 to 13, wherein the at least one received image (Ii) comprises a first radiological image and a second radiological image, wherein the first radiological image represents the examination area of the examination object without contrast agent or after application of a first amount of a contrast agent and the second radiological image represents the examination area of the examination object after application of a second amount of the contrast agent, wherein each synthetic image (SI21, SI22, SI23, SI24, SI25, SI26) and / or the combined synthetic image (S) is a synthetic radiological image, wherein each synthetic image (SI21, SI22, SI23, SI24, SI23, SI26) and / or the combined synthetic image (S) represents the examination area of the examination object after application of a third amount of the contrast agent, wherein the second amount is different from, preferably greater than, the first amount, and the third amount is different from, preferably greater than, the first amount and the second amount.

15. The method according to any one of claims 1 to 13, wherein the at least one received image (Ii) comprises a first radiological image and a second radiological image, wherein the first radiological image represents the examination area of the examination object in a first time period before or after application of a contrast agent and the second radiological image represents the examination area of the examination object in a second time period after application of the contrast agent, wherein each synthetic image (SI21, SI22, SI23, SI24, SI25, SI26) and / or the combined synthetic image (S) is a synthetic radiological image, wherein each synthetic image (SI21, SI22, SI23, SI24, SI23, SI26) and / or the combined synthetic image (S) represents the examination area of the examination object in a third time period after application of the contrast agent, wherein the second time period preferably follows the first time period in time and the third time period preferably follows the second time period in time.

16. Computer system (10) comprising - a receiving unit (11), - a control and computing unit (12) and - an output unit (13), wherein the control and computing unit (12) is configured - to cause the receiving unit (11) to receive at least one image of an examination area of an examination object, - generating a plurality of sub-images based on the at least one received image, wherein each sub-image represents a sub-area of the examination area of the examination object, sub-areas represented by different sub-images partially but not completely overlapping, - generating a synthetic sub-image for each generated sub-image at least proportionally on the basis of the generated sub-image, - determining color values of corresponding image elements of synthetic sub-images, wherein corresponding image elements represent the same sub-region of the examination area, - determining a scattering measure of the color values of corresponding image elements, - determining a confidence value on the basis of the scattering measure, - to cause the output unit (13) to output the confidence value.

17. A computer readable storage medium comprising a computer program (40) which, when loaded into a working memory (22) of a computer system (10), causes the computer system (10) to perform the following steps: - Receiving at least one image (Ii) of an examination area of an examination object, - Generating a plurality of sub-images (PI21, PI22, PI23, PI24, PI23, PI26) based on the at least one received image (Ii), wherein each sub-image represents a sub-area of the examination area of the examination object, sub-areas represented by different sub-images partially but not completely overlapping each other, - For each generated sub-image (PI21, PI22, PI23, PI23, PI23, PI26): generating a synthetic sub-image (SI21, SI22, SI23, SI24, SI25, SI26) at least proportionally on the basis of the generated sub-image (PI21, PI22, PI23, PI24, PI25, PI26), - Determining color values of corresponding image elements of synthetic sub-images, wherein corresponding image elements represent the same sub-area of the examination area, - Determining a scattering measure of the color values of corresponding image elements, - Determining a confidence value on the basis of the scattering measure, - Outputting the confidence value.