Medical analysis method for predicting metastatic lesions in test tissue samples

Abstract

The present disclosure relates to a computer-implemented medical analysis method for predicting metastatic lesions (300) in a test tissue sample, the method comprising: providing a first machine learning model (154) having an input and an output; receiving a description (401) of a tumor (304) and first image data (148) of a test tissue sample of an anatomical region (306), the test tissue sample being free of metastatic lesions (300); providing the first image data (148) and the tumor description (401) to the input of the first machine learning model (154); receiving, in response to the providing, from the output of the first machine learning model (154) a prediction of the occurrence of metastatic lesions (300) originating from the tumor (304) in the test tissue sample; and providing the prediction.

Description

Technical Field

[0001] The present invention relates to a medical analysis method for predicting metastatic lesions in a test tissue sample, a medical analysis system for predicting metastatic lesions in a test tissue sample, an MRI system and a computer program product. Background Art

[0002] Many types of tumors tend to spread throughout the body via metastatic lesions. Clinicians are looking for ways to identify the location of potential metastatic lesions as early as possible when the primary tumor is discovered. Correctly identifying the location of metastatic lesions can provide treatment advantages. For example, prophylactic irradiation can reduce the incidence of metastatic lesions and improve overall survival.

[0003] For imaging the tissue of a subject, various imaging techniques are known in the art. Examples are magnetic resonance imaging (MRI), computed tomography (CT), ultrasound imaging (US) and positron emission tomography imaging.

[0004] Magnetic resonance imaging (MRI) is a state-of-the-art imaging technique that allows cross-sectional observation of targets such as the human body with unprecedented tissue contrast. MRI is based on the principle of nuclear magnetic resonance, a spectroscopic technique used by scientists to obtain microscopic chemical and physical information about molecules. The basis of both nuclear magnetic resonance and MRI is the fact that atomic nuclei with non-zero spin have a magnetic moment. In medical imaging, for example, the nuclei of hydrogen atoms are studied because they are present in the body in high concentrations (e.g., water). If a strong DC magnetic field is applied, the nuclear spins of elementary particles resonate at a resonance frequency. This magnetic resonance (MR) frequency is determined by the magnetic flux level of the DC magnetic field. In an MRI scanner, the magnetic field matches the selected resonance frequency only at a position in space. The presence of these particles can only be detected at this position. By changing this position, the image can be measured.

[0005] The required strong DC magnetic field (B0 field) is usually generated by a superconducting magnet. In order to change the magnetic field so that it matches a given radio frequency at only one position, gradient coils are used to generate magnetic field gradients. The magnetic field gradient can be varied over time to achieve scanning.

[0006] To excite nuclear resonance, the RF coil generates a high frequency magnetic field at the nuclear resonance. The magnetic field must point in a radial direction relative to the axis of the MRI scanner. In order to achieve a radial magnetic field in all directions, a rotating magnetic field is used, which points in any radial direction at one point in time within a cycle. This can be achieved by using, for example, a so-called "birdcage" arrangement. Currents in the opposing plates of the birdcage flow in opposite directions, thereby generating a radial magnetic field. The currents in adjacent plates have a phase shift, causing the magnetic field to rotate.

[0007] The coil is typically a highly resonant antenna designed to generate a clear magnetic field inside the human body.

[0008] WO 2009 / 084995 A1 relates to a detection system for automatically detecting bone cancer metastasis lesions from an isotope bone scan image set of a patient's bones, the system comprising a shape recognizer unit, a hotspot detection unit, a hotspot feature extraction unit, a first artificial neural network unit, a patient feature extraction unit and a second artificial neural network unit.

[0009] US patent application US2017 / 0193175 discloses a computational method for predicting the recurrence of non-small cell lung cancer (NSCLC) based on digitally stained histopathological images. Automated deep learning provides the probability of recurrence in the (same) region of NSCLC. In other words, the probability of cancer recurrence in the same organ from which the biopsy was performed is predicted.

[0010] The scientific publication "Classification of Suspected Liver Metastases Using fMRI Images: A Machine Learning Approach" (EGOV 2016, pp. 93-100) discloses a method for classifying suspected metastatic lesion areas in tissue sample images as metastatic lesions or non-metastatic lesions. The method comprises: computing two different normalized hemodynamic activity maps of the tissue image, adaptively partitioning the maps, extracting features of interest from suspicious areas identified by radiologists, and obtaining a classification by providing the extracted features to the input of a trained classification engine. Summary of the invention

[0011] An embodiment of the present invention provides a computer-implemented medical analysis method for predicting metastatic lesions in a test tissue sample, the method comprising:

[0012] receiving a plurality of different training tissue samples of the training tissue samples for an anatomical region, each training set comprising, for a corresponding one of the training tissue samples, second image data and third image data, the third image data being data acquired after a metastatic lesion has occurred in a subject from which the training tissue sample is derived, the second image data representing the training tissue sample without a metastatic lesion, wherein each training set further comprises a description of a tumor in the subject from which the training tissue sample is derived;

[0013] Running a learning algorithm on the training set to generate a trained first machine learning model having an input and an output;

[0014] receiving a description of the tumor and first image data of a test tissue sample of an anatomical region, the test tissue sample being free of metastatic lesions;

[0015] providing the first image data and the tumor description to the input of the first machine learning model;

[0016] receiving, in response to the providing, from the output of the first machine learning model, a prediction of the occurrence of a metastatic lesion originating from the tumor in the test tissue sample; and

[0017] The prediction is provided.

[0018] The term "image data" refers to data that can be transformed into an image and displayed on a user interface such as a computer screen.

[0019] An "anatomical region" is any predefined region of the anatomy of a subject (eg, a human or an animal). A region may include an organ (eg, the liver or the brain) or a region (eg, the spinal region, the knee, the shoulder, etc.).

[0020] The term "machine learning" refers to a computer algorithm used to extract useful information from a training data set by building a probabilistic model (referred to as a machine learning model) in an automated manner. Machine learning can be performed using one or more learning algorithms such as classification and regression techniques (e.g., support vector methods, trees, neural networks, etc.). A "model" can be, for example, a formula or set of rules that enables prediction of unmeasured values ​​(e.g., which label corresponds to a given token) based on other known values ​​and / or predicts or selects actions for maximizing future rewards. According to one embodiment, the machine learning model is a deep learning model.

[0021] Embodiments of the present invention may have the advantage that the development of metastatic lesions may be predicted before the metastatic lesions themselves are present in the anatomical region. For example, a certain tumor present in a subject may cause the development of metastatic lesions in the future. The above method aims to predict this development, including an optional prediction of the development over time. This may even allow for an accurate ("intra-organ") prediction of metastatic lesions, i.e., the likely location within a specific organ.

[0022] Thus, an embodiment may suggest applying machine learning to intra-organ predictions of the location of metastatic lesions: after the detection of a primary tumor, the target organ most likely to be affected by the metastatic lesions may be imaged, for example, with one or more suitable imaging modalities that provide sufficient radiological information. The resulting image enters a suitably trained machine learning algorithm that generates information about the probability of the intra-organ location of the expected metastatic lesions. That is, when a primary tumor is detected in an organ or tissue region, based on a description of the detected primary tumor and first image data of a test tissue sample of an anatomical region that is separate from the detected primary tumor and (not yet) has no metastatic lesions, a first machine learning model is able to predict the occurrence (when appropriate) of metastatic lesions in the anatomical region. In particular, the first machine learning model can predict the occurrence of metastatic lesions in other different parts of the patient's body (i.e., other organs or tissue regions that are different from the organ or tissue region in which the primary tumor has been detected).

[0023] According to an embodiment of the present invention, the method further includes:

[0024] receiving a plurality of different training tissue samples of the training tissue samples for the anatomical region, each training set comprising, for a corresponding one of the training tissue samples, second image data and third image data, the third image data being data acquired after a metastatic lesion has occurred in a subject from which the training tissue sample is derived, the second image data representing the training tissue sample without a metastatic lesion, wherein each training set further comprises a description of the tumor in the subject from which the training tissue sample is derived; and

[0025] The learning algorithm is run on the training set to generate the first machine learning model. Thus, the second machine learning model can use the third image data to predict the occurrence of metastatic lesions in other different parts of the patient's body (i.e., other organs or tissue regions different from the organ or tissue region where the primary tumor has been detected).

[0026] This may have the benefit that even without prior knowledge of the exact processes leading to the development of metastatic lesions, a machine learning model capable of performing the above-mentioned prediction of metastatic lesions may still be accurately provided. With the aid of the training set, a certain susceptibility to the development of metastatic lesions may be automatically attributed to certain anatomical regions, which may even depend on a certain type of anatomical shape of the anatomical region visible in the second image data. For example, a machine learning model may thus describe that a brain with a certain shape of certain gyri in a specific brain region is more susceptible to the development of metastatic lesions than a brain with a slightly different shape of different gyri in this brain region.

[0027] It must be noted that the third image data is data acquired after metastatic lesions have occurred in the subject, wherein the metastatic lesions are not necessarily located in the anatomical region. Therefore, in the case where the third image data does not have any metastatic lesions, this will be an indication to the learning algorithm that the tissue corresponding to the third image data is less prone to the formation of metastatic lesions. Conversely, in the case where the third image data represents a training tissue sample affected by metastatic lesions, this will be an indication to the learning algorithm that the tissue corresponding to the third image data is more prone to the formation of metastatic lesions. In any case, the second image data is always free of any metastatic lesions.

[0028] For example, the learning algorithm may include a deep learning network that learns to predict the probability of metastatic lesions in a particular anatomical region by providing the second image data and information about the occurrence of metastatic lesions in subsequent acquisitions.

[0029] According to an embodiment, the second image data and the third image data represent a 3D volume of the anatomical structure.For example, the image data may be acquired using a 2D multi-slice acquisition or using a 3D acquisition in order to provide a 3D volume.

[0030] According to an embodiment, the description of the tumor includes any of the following: the spatial location of the tumor relative to the anatomical structure of the object with the tumor, the classification of the tumor, image data of the tumor. Depending on the location of the tumor, the development of metastatic lesions may be different. Tumor classification can be provided, for example, according to the WHO / IARC or TNM tumor classification well known in the art. Therefore, even considering molecular pathology, tumor classification can also be based on a consensus from a histopathological point of view.

[0031] For example, a description of a tumor may include CT or MR or X-ray image data. Based on the size, shape and location of the tumor, a learning algorithm can predict the probability of metastatic lesions in a specific anatomical region different from the region containing the tumor itself.

[0032] According to an embodiment, the prediction of the occurrence of the metastatic lesion includes any of the following: an indication of whether the test tissue sample will be affected by the metastatic lesion originating from the tumor; a probability map across the anatomical region, the probability map having an indication of which discrete locations of the anatomical region will be affected by the metastatic lesion originating from the tumor.

[0033] Thus, depending on the training set available for the entire map, a prediction can be made based on simple yes / no information if a test tissue sample will be affected by metastatic lesions originating from the tumor somewhere, with the map providing detailed information on the probability of which spatial location within the tissue sample will be affected by metastatic lesions originating from the tumor.

[0034] For example, the indication includes any of the following: a probability value describing the probability that the metastasis is expected; a binary value describing whether the metastasis is expected. The probability value can be given in the form of a percentage (e.g., 100%, 50%, 25% ...) or a binary value (e.g., "1" = "yes, metastasis is likely to occur", "0" = "metastasis is unlikely to occur").

[0035] Thus, for each anatomical structure, a probability or binary description of where a metastasis will occur can be predicted. Region proposals may come with a location / spatial extension (and a score / binary decision). Finally, for segmentation-like methods, each voxel or pixel can have a binary decision / probability score.

[0036] According to an embodiment, the first image data, the second image data, the third image data and the image data of the tumor are any of the following: magnetic resonance image data, computed tomography image data, ultrasound image data, positron emission tomography image data, X-ray data. Therefore, the method can freely use any image data that can be obtained from the object for which metastatic lesions must be predicted. The training set may include, for example, a set of second image data and third image data from MRI and a set of second image data and third image data from CT. The reason is that both sets have images showing anatomical regions, but the information content "contrast" of the anatomical regions is different. After combination, the highest information content is provided.

[0037] Thus, the images acquired during the training and prediction phases can be from a single or multiple imaging modalities, e.g., magnetic resonance (MR), computed tomography (CT), ultrasound (US), or positron emission tomography (PET). Furthermore, it can be advantageous to acquire multiple image sets with different contrasts with each modality (e.g., MR images with T1 / T2 / diffusion weighting, images before and after contrast injection, etc.). A suitable compromise has to be found between imaging effort and predictive power. Furthermore, the method is not limited to imaging data only, but can also incorporate information such as age, gender, laboratory values, or patient history.

[0038] Therefore, according to an embodiment, the method further comprises: receiving medical history data of the subject with the tumor, and providing the medical history data to the input of the machine learning model, wherein the training set further comprises the medical history data of the subject with the tumor. For example, a heavy smoker may be more susceptible to the development of metastatic lesions in certain anatomical regions (e.g., the lungs) than a non-smoker.

[0039] Typically, a patient's medical history or record is information obtained by a physician by asking specific questions of the patient or someone else who knows the patient and can give appropriate information, with the goal of obtaining information useful in formulating a diagnosis and providing medical care for the patient. Examples of medical history data are: identity and demographic information (e.g., name, age, height, weight), past medical history (including major illnesses, any previous surgeries / operations), family illnesses, routine and acute medications, allergies, etc.

[0040] According to an embodiment, the third image data is a time-resolved image data set, wherein the indication is time-resolved. This may have the following benefits: namely, a time-resolved prediction of the development of metastatic lesions may be performed. For example, in the case where two locations are predicted to be highly likely to develop metastatic lesions, a corresponding preventive irradiation treatment may be first performed (based on the prediction) for the training tissue where the development of metastatic lesions is expected to occur first, and then after the irradiation treatment is completed, an additional preventive irradiation treatment may be performed (based on the prediction) for the training tissue where the development of metastatic lesions is expected to occur at a later point in time. This may help to focus the pre-treatment on the most important tissue areas, thereby avoiding stress on the subject due to too many parallel pre-treatments, which are usually performed in parallel with the exhausting treatment of the primary tumor (i.e., the source of metastatic lesions).

[0041] According to an embodiment, the method further comprises: analyzing the third image data for the presence or absence of metastatic lesions in the corresponding training tissue sample, and annotating the third image data with respect to the results of the analysis. For example, the analysis is performed using a second machine learning model. The second machine learning model is provided to automatically identify metastatic lesions in the third image data of the training set of training tissue samples of the anatomical region, and the first machine learning model is provided to predict the occurrence of metastatic lesions based on the first image data without metastatic lesions.

[0042] According to an embodiment, the learning algorithm is a deep learning algorithm. Preferably, the deep learning algorithm is a convolutional neural network (CNN). CNN has been developed for image classification and has shown excellent performance in this field. CNN includes a sequence of interconnected layers (i.e., convolutional layers, maximum pooling layers ...) (they are similar to the feature extraction steps in the classical image processing pipeline), followed by one or more fully connected layers. CNN is typically trained to predict class labels or probabilities for natural images or to provide diagnosis for medical images, but the basic concept can be directly transferred to the described metastatic lesion prediction technology. It should be noted that even if CNN is developed primarily for processing image data, non-image data such as descriptions of tumors and / or medical history data of the object can also be integrated, for example, in terms of the connection of additional inputs at the level of fully connected layers.

[0043] As another example of a deep learning algorithm, for the prediction of metastatic lesion locations (including their probabilities), a region proposal network can be considered. Here, compared to conventional CNNs, the output neurons no longer correspond to discrete class labels, but to voxel locations and metastatic lesion probabilities. For this purpose, it can be assumed that the image of the anatomical region is transformed in a normalized reference frame.

[0044] According to an embodiment, the deep learning algorithm may therefore be based on a convolutional neural network architecture, the method further comprising: transforming the second image data and the third image data into a common reference image frame of the anatomical region for all training sets; performing the running of the learning algorithm on the training set comprising the transformed image data to generate the first machine learning model. Of course, the first image data should also be transformed into a common reference image frame so that for all image data used for training and prediction, the algorithm and model know which parts of the image correspond to each other.

[0045] Typically, deep learning-based techniques can be used to generate probability maps. Given an image of a primary tumor, combined with annotated metastatic lesion regions (anatomical regions) in the target organ, the probability at the voxel level can be estimated using a slice-by-slice analysis of the corresponding CNN (centered on the voxel of interest). Alternatively, a more efficient network structure (e.g., a fully convolutional network) from the field of semantic image segmentation can be employed.

[0046] In another aspect, the invention relates to a computer program product comprising machine executable instructions for execution by a processor, wherein execution of the machine executable instructions causes the processor to execute a method as described above.

[0047] In another aspect, the present invention relates to a medical analysis system for predicting metastatic lesions in a test tissue sample, the system comprising a memory containing machine executable instructions and a processor for controlling the medical analysis system, wherein execution of the machine executable instructions causes the processor to: receive a training set of a plurality of different training tissue samples of training tissue samples for an anatomical region, each training set comprising second image data and third image data for a corresponding one of the training tissue samples, the third image data being data acquired after a metastatic lesion occurs in a subject from which the training tissue sample originates, the second image data representing the training tissue sample without metastatic lesions, wherein each training set also comprises a description of a tumor in the subject from which the training tissue sample originates; run a learning algorithm on the training set to generate a trained first machine learning model having an input and an output; receive a description of the tumor and first image data of a test tissue sample of an anatomical region, the test tissue sample without metastatic lesions; provide the first image data and the tumor description to the input of the first machine learning model; receive a prediction of the occurrence of metastatic lesions originating from the tumor in the test tissue sample from the output of the first machine learning model in response to the providing; and provide the prediction.

[0048] In another aspect, the invention relates to an MRI system comprising the described medical analysis system, the MRI system being configured for acquiring the first image data.

[0049] It should be understood that one or more of the aforementioned embodiments of the present invention may be combined as long as the combined embodiments are not mutually exclusive.

[0050] Those skilled in the art will appreciate that the examples discussed in the present disclosure may be implemented as an apparatus, method, or computer program product. Thus, various aspects may take the form of a fully hardware embodiment, a fully software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software aspects and hardware aspects, all generally referred to herein as a "circuit," "module," or "system." In addition, various aspects may take the form of a computer program product implemented in one or more computer-readable media having a computer executable code implemented thereon.

[0051] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. As used herein, "computer-readable storage medium" encompasses any tangible storage medium that can store instructions that can be executed by a processor of a computing device. A computer-readable storage medium may be referred to as a computer-readable non-transient storage medium. A computer-readable storage medium may also be referred to as a tangible computer-readable medium. In some embodiments, a computer-readable storage medium may also store data that can be accessed by a processor of a computing device. Examples of computer-readable storage media include, but are not limited to, floppy disks, magnetic hard disk drives, solid-state hard disks, flash memories, USB thumb drives, random access memories (RAM), read-only memories (ROM), optical disks, magneto-optical disks, and register files of processors. Examples of optical disks include compact disks (CDs) and digital versatile disks (DVDs), such as CD-ROMs, CD-RWs, CD-Rs, DVD-ROMs, DVD-RWs, or DVD-R disks. The term "computer-readable storage medium" also refers to various types of recording media that can be accessed by a computer device via a network or communication link. For example, data may be retrieved on a modem, on the Internet, or on a local area network. Computer executable code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0052] A computer readable signal medium may include, for example, a propagated data signal in which a computer executable code is implemented, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that is capable of conveying, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0053] "Computer memory" or "memory" is an example of a computer-readable storage medium. Computer memory is any memory that can be directly accessed by a processor. "Computer storage" or "storage device" is another example of a computer-readable storage medium. A computer storage device is any non-volatile computer-readable storage medium.

[0054] As used herein, "processor" encompasses an electronic component capable of running a program or machine executable instructions or computer executable code. References to computing devices including "processors" should be interpreted as possibly including more than one processor or processing core. The processor may be, for example, a multi-core processor. A processor may also refer to a collection of processors within a single computer system or distributed between multiple computer systems. The term "computing device" should also be interpreted as possibly referring to a collection or network of multiple computing devices, each of which includes one or more processors. Computer executable code may be executed by multiple processors that may be within the same computing device or may even be distributed on multiple computing devices.

[0055] Computer executable code can include machine executable instructions or programs that cause a processor to perform an aspect of the present invention. Computer executable code for performing operations for various aspects of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Java, Smalltalk, C++, etc.) and conventional program programming languages ​​(e.g., C programming language or similar programming languages), and compiled into machine executable instructions. In some instances, the computer executable code can be in the form of a high-level language or in a precompiled form, and can be used in conjunction with an interpreter that generates machine executable instructions during operation.

[0056] The computer executable code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0057] Various aspects are described with reference to the flowcharts and block diagrams of methods, devices (systems) and computer program products. It should be understood that each frame or part of the frame of the flowchart, diagram and / or block diagram can be implemented by computer program instructions in the form of computer executable code when appropriate. It should also be understood that when mutually non-exclusive, the frames in different flowcharts, diagrams and / or block diagrams can be combined. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a unit for implementing the function / action specified in one or more frames of the flowchart and / or block diagram.

[0058] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other device to function in a specific manner so that the instructions stored in the computer-readable medium produce an article of manufacture that includes instructions for implementing the functions / actions specified in the flowchart and / or one or more block diagram blocks.

[0059] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other devices, thereby producing a computer-implemented process, such that the instructions running on the computer or other programmable apparatus provide a process for implementing the functions / actions specified in the flowchart and / or one or more block diagram blocks.

[0060] "User interface" used herein is an interface that allows a user or operator to interact with a computer or computer system. "User interface" may also be referred to as a "human-machine interface device". A user interface may provide information or data to an operator and / or receive information or data from an operator. A user interface may enable the input from an operator to be received by the computer, and may provide output to a user from the computer. In other words, a user interface may allow an operator to control or manipulate a computer, and an interface may allow a computer to indicate the effect of the control or manipulation of an operator. Displaying data or information on a display or graphical user interface is an example of providing information to an operator. Receiving data by a keyboard, mouse, tracking ball, touch pad, pointing stick, graphic input board, joystick, game controller, webcam, head mounted device, foot pedal, wired gloves, remote controller, and accelerometer is all examples of user interface components that enable receiving information or data from an operator.

[0061] "Hardware interface" as used herein encompasses an interface that enables a processor of a computer system to interact with and / or control an external computing device and / or apparatus. A hardware interface may allow a processor to send a control signal or instruction to an external computing device and / or apparatus. A hardware interface may also enable a processor to exchange data with an external computing device and / or apparatus. Examples of hardware interfaces include, but are not limited to, a universal serial bus, an IEEE 1394 port, a parallel port, an IEEE 1284 port, a serial port, an RS-232 port, an IEEE-488 port, a Bluetooth connection, a wireless LAN connection, a TCP / IP connection, an Ethernet connection, a control voltage interface, a MIDI interface, an analog input interface, and a digital input interface.

[0062] As used herein, a "display" or "display device" encompasses an output device or user interface suitable for displaying images or data. A display may output visual, auditory, and / or tactile data. Examples of displays include, but are not limited to: a computer monitor, a television screen, a touch screen, a tactile electronic display, a Braille screen, a cathode ray tube (CRT), a memory tube, a bi-stable display, an electronic paper, a vector display, a flat panel display, a vacuum fluorescent display (VF), a light emitting diode (LED) display, an electroluminescent display (ELD), a plasma display panel (PDP), a liquid crystal display (LCD), an organic light emitting diode display (OLED), a projector, and a head mounted display.

[0063] Magnetic resonance (MR) data is defined herein as the measurements of radio frequency signals emitted by atomic spins recorded using the antenna of a magnetic resonance device during a magnetic resonance imaging scan. Magnetic resonance data is an example of medical image data. A magnetic resonance (MR) image is defined herein as a two-dimensional or three-dimensional visualization reconstructed from anatomical data contained within the magnetic resonance imaging data. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In the following, preferred embodiments of the present invention will be described by way of example only and with reference to the accompanying drawings, in which:

[0065] Figure 1 illustrates an example of a magnetic resonance imaging system,

[0066] Figure 2 illustrates an example of a medical analysis system,

[0067] Figure 3 is a diagram illustrating the relationship between tumors and metastatic lesions.

[0068] Figure 4 is a block diagram illustrating the development and use of machine learning models,

[0069] Figure 5 is a flowchart for training and using a machine learning model.

[0070] Reference numerals list

[0071] 100 Magnetic Resonance Imaging System

[0072] 104 Magnets

[0073] 106 Magnet bore

[0074] 108 Imaging Area

[0075] 109 Areas of Interest

[0076] 110 Magnetic Field Gradient Coil

[0077] 112 Magnetic Field Gradient Coil Power Supply

[0078] 114 RF Coil

[0079] 116 RF Amplifier

[0080] 118 objects

[0081] 120 Object Support

[0082] 126 Computer Systems

[0083] 128 hardware interfaces

[0084] 130 processors

[0085] 132 User Interface

[0086] 134 Computer memory

[0087] 140 machine executable instructions

[0088] 142 Imaging Scan Protocol

[0089] 144 Magnetic resonance signals

[0090] 146 Magnetic Resonance Data

[0091] 148 first image data

[0092] 150 Image reconstruction instructions

[0093] 152 machine executable instructions

[0094] 154 First Machine Learning Model

[0095] 200 processors

[0096] 202 Memory

[0097] 204 User Interface

[0098] 206 Hardware Interface

[0099] 210 Medical Analysis System

[0100] 300 Metastatic Lesions

[0101] 302 Prediction Image

[0102] 304 Primary Tumor

[0103] 306 anatomical regions

[0104] 308 Disposal

[0105] 400 second image data

[0106] Description of 401 tumor

[0107] 402 Third Image Data

[0108] 404 Machine Learning Algorithms

[0109] 500-506 training phase

[0110] 508-516 Application Phase DETAILED DESCRIPTION

[0111] In the following, elements with similar reference numbers in the drawings are either similar elements or perform equivalent functions. Elements discussed previously need not be discussed in later drawings if the functions are equivalent.

[0112] Various structures, systems and devices are schematically depicted in the drawings for purposes of explanation only so as not to obscure the present invention with details that are well known to those skilled in the art. Nevertheless, the drawings are included to describe and explain illustrative examples of the disclosed subject matter.

[0113] Figure 1 An example of a magnetic resonance imaging system 100 with a magnet 104 is shown. The magnet 104 is a superconducting cylindrical magnet having a bore 106 running through it. Different types of magnets may also be used. For example, a split cylindrical magnet and a so-called open magnet may also be used simultaneously. A split cylindrical magnet is similar to a standard cylindrical magnet, except that the cryostat has been divided into two parts to allow access to the isoplane of the magnet, such a magnet may be used, for example, in conjunction with charged particle beam therapy. An open magnet has two magnet parts, one above the other, with a space large enough between the two magnet parts to accommodate the subject: the area arrangement of the two parts is similar to a Helmholtz coil. Open magnets are popular because the subject is less constrained. Inside the cryostat of the cylindrical magnet, there is a set of superconducting coils. Within the bore 106 of the cylindrical magnet 104, there is an imaging zone 108, in which the magnetic field is strong and uniform enough to perform magnetic resonance imaging. A region of interest 109 is shown within the imaging zone 108. A subject 118 (eg, a patient) is shown supported by a subject support 120 (eg, a movable table) such that at least a portion of the subject 118 is within the imaging zone 108 and the region of interest 109.

[0114] Within the bore 106 of the magnet, there is also a set of magnetic field gradient coils 110, which are used to acquire magnetic resonance data to spatially encode magnetic spins within the imaging zone 108 of the magnet 104. The magnetic field gradient coils 110 are connected to a magnetic field gradient coil power supply 112. The magnetic field gradient coils 110 are intended to be representative. Typically, the magnetic field gradient coils 110 include three separate sets of coils, which are used to perform spatial encoding in three orthogonal spatial directions. The magnetic field gradient power supply supplies current to the magnetic field gradient coils. The current supplied to the magnetic field gradient coils 110 is controlled according to time and can be ramped or pulsed.

[0115] Adjacent to the imaging zone 108 is a radio frequency coil 114 "RF antenna" for manipulating the orientation of magnetic spins within the imaging zone 108 and for also receiving radio transmissions from spins within the imaging zone 108. The RF antenna contains one or more coil elements. The RF coil 114 is connected to a RF amplifier 116. The RF amplifier 116 is providing RF power to the RF coil 114 to manipulate the orientation of magnetic spins within the imaging zone 108.

[0116] The radio frequency amplifier 116 and the magnetic field gradient coil power supply 112 are shown as being connected to a hardware interface 128 of a computer system 126. Thus, the computer system 126 also functions as a receiver for receiving and processing MR signals acquired using the radio frequency coil 114.

[0117] The computer system also includes a processor 130 that communicates with the hardware interface 128, a memory 134, and a user interface 132. The memory 134 can be any combination of memory accessible to the processor 130. This can include, for example, main memory, cache memory, and non-volatile memory such as flash RAM, a hard drive, or other storage devices. In some examples, the memory 134 can be considered a non-transitory computer-readable medium.

[0118] The computer memory 134 is shown as containing machine executable instructions 140. The machine executable instructions include commands or instructions that enable the processor 130 to control the operation and function of the magnetic resonance imaging system 100. The computer memory 134 is shown as also containing imaging scan protocols 142. Each imaging scan protocol 142 may include pulse sequence commands for one or more pulse sequences, the pulse sequence commands being either instructions or data that can be converted into instructions, which enable the processor 130 to control the magnetic resonance imaging system 100 to acquire magnetic resonance data. The pulse sequence commands may thus be part of the imaging scan protocol. The magnetic resonance data may, for example, be used to cause the magnetic resonance imaging system to perform a plurality of pulse repetitions that cause magnetic resonance signals 144 to be acquired. Each pulse corresponds to the supply of RF power to the radio frequency coil 114 by the radio frequency amplifier 116.

[0119] The magnetic resonance signals 144 are shown as being stored in the computer memory 134. The magnetic resonance signals 144 for a particular pulse repetition may be organized into magnetic resonance data 146. The magnetic resonance data 146 may be used to generate a series of first image data 148. The imaging scan protocol 142 may also include instructions 150 for reconstructing the first image data 148 from the MR data 146 acquired using imaging. The imaging scan protocol 142 and the instructions 150 are included in the module 141.

[0120] The computer memory 134 is shown as further comprising machine executable instructions 152 as a module that enable the processor 130 to provide a prediction of metastatic lesions in the image data using the first machine learning model 154. In the following, without any limitation on generality, it is assumed that the image data used to provide a prediction of metastatic lesions is MR image data, such as the first image data 148.

[0121] The machine executable instructions 152 as modules can also be described as an artificial intelligence (AI) component. The AI ​​component can be configured to detect metastatic lesions robustly and quickly. The AI ​​component can be configured to perform machine learning on a training set to generate one or more machine learning models for predicting metastatic lesions in image data. The training process will be described below.

[0122] Even though in the above example the machine executable instructions 152 as modules are shown as part of the computer system 126 for acquiring MR image data, the modules may be part of a computer system independent of the system 126. Figure 2 A corresponding example is depicted in .

[0123] Figure 2 The medical analysis system 210 is depicted as a computer system that includes a processor 200, a memory 202, a hardware interface 206, and a user interface 204. The medical analysis system 210 may be Figure 1 The memory 202 may be any combination of memories accessible to the processor 130. This may include, for example, main memory, cache memory, and non-volatile memory such as flash RAM, hard drives, or other storage devices. Again, in some examples, the memory 202 may be considered a non-transitory computer readable medium.

[0124] Computer memory 202 is shown as containing machine executable instructions 152 as modules. Another example of instructions is a first machine learning model 154. Medical analysis system 210 can receive image data such as first image data 148 (MR image data) from system 126 via hardware interfaces 128 and 206, which can communicate with each other over a network such as the Internet. The machine executable instructions 152 as modules are configured to predict metastatic lesions in a test tissue sample by utilizing the first machine learning model 154.

[0125] This can allow machine learning to be applied to intra-organ predictions of metastatic lesion locations. After the primary tumor is detected, the target organs most likely to be affected by metastatic lesions can be imaged using one or more suitable imaging modalities that provide sufficient radiological information. The resulting images can then be input into a suitably trained machine learning algorithm (first machine learning model 154), which generates information about the probability of metastatic lesion locations.

[0126] Received via user interface 204 is, for example, a description of a tumor of a subject (e.g., a person). First image data 148 is image data of a test tissue sample of an anatomical region of the person. The test tissue sample is free of metastatic lesions. By inputting the image data and the tumor description into an input portion of a first machine learning model 154, the first machine learning model 154 is able to output a prediction of the occurrence of metastatic lesions originating from the tumor in the test tissue sample. The prediction is then provided to user interface 204. All of this is controlled by machine executable instructions 152.

[0127] In a practical example, Figure 3 Schematically depicted in , any suitable means may be used to detect a primary tumor 304. Here, the tumor 304 is a lung tumor. Over a certain period of time, the tumor may lead to the development of metastatic lesions in certain organs or anatomical regions (e.g., the lungs themselves, the liver, the brain, or the colon of a person). Examples of these anatomical regions are depicted by reference numeral 306. Assuming that the machine learning model is trained on the brain, inputting a description of the tumor and first image data 148 of one or more images of the still healthy anatomical region "brain" enables the first machine learning model 154 to predict metastatic lesions 300 and the probability of future metastatic lesion development. A corresponding predicted image 302 may be provided on the user interface 204.

[0128] The location or area where metastatic lesions are likely to develop can be predicted, and the location or area can be subjected to corresponding preventive irradiation treatment (reference numeral 308).

[0129] Figure 4is a block diagram illustrating the various steps for obtaining and using the first machine learning model 154. Figure 5 The flowchart in shows the corresponding method steps, wherein the left-hand boxes 500 - 506 reflect the training phase and the right-hand boxes 508 - 516 reflect the application or prediction phase.

[0130] The method begins at block 500, where a training set of multiple different training tissue samples of training tissue samples for an anatomical region is received. For example, different training sets will be received for different people with tumors. The type and location of the tumor between different people may be the same or different. However, for effective training, a large number of training sets should be received for a given tumor type and location.

[0131] Each received training set includes a corresponding training tissue sample of a second image data 400 and a third image data 402 as anatomical region image data. The second image data 400 does not have any metastatic lesions and shows a certain anatomical region, while the third image data 402 is acquired at a later time point and shows the metastatic lesions of the anatomical region. The metastatic lesions originate from a tumor 304 of a person. For a certain anatomical region, the second image data 400 and the third image data 402 may be provided with different contrasts and / or acquired as different training sets with different image acquisition techniques. In addition, each training set also includes a description 401 of a tumor in the person from which the training tissue sample originates. The description may, for example, include a description of the location of the tumor "lower right part of the lung". In another example, the description may alternatively or additionally include image data of the region where the tumor is located.

[0132] In optional block 502, the third image data 402 may be analyzed for the presence of metastatic lesions, and the locations of the metastatic lesions may be marked accordingly in the third image data 402. This may be performed by conventional means, such as in an automated manner by a neural network or other machine learning model that is typically distinct from the first machine learning model 154. Manual marking may also be performed.

[0133] Optional block 504 is used to transform the second image data 400 and the third image data 402 into a common reference image frame of the anatomical region for all training sets. This will ensure that any training and any application later preferably always is performed on the same image voxels, since different second image data 400 and third image data 402 and different first image data 148 may be acquired at different viewing angles showing different image portions of the anatomical region.

[0134] Finally, in box 506, the learning algorithm 404 is run on the second image data 400 and the third image data 402 and the description of the tumor 401. This generates the first machine learning model 154.

[0135] It is important to note that any of the above steps 500-506 may be performed on the medical analysis system 210 using the machine executable instructions 152. Steps 500-506 may be repeated multiple times using different training sets - the more training sets available, the more accurate the machine learning model.

[0136] After the first machine learning model 154 has been generated, the method continues in box 508, where the first machine learning model 154 is provided. In box 510, a description of the tumor 401 and first image data 148 of tissue (still) without metastatic lesions are received. Here, it is assumed that the description of the tumor 401 describes the type and location of the tumor 304 for which the first machine learning model 154 was obtained, so that the model can make certain predictions about the tumor 304.

[0137] Again, optional block 512 is used to transform the first image data 148 into a common reference image frame for the anatomical region. In block 514, the description 401 of the tumor and the (optionally transformed) first image data 148 are then input to the first machine learning model 154. In block 516, the machine learning model outputs a predicted location and / or probability of future occurrence of metastatic lesions. This may be in the form of, for example, image 302, which may then be provided to the user interface 204 ( Figure 2 ).

[0138] Therefore, for each type of primary tumor investigated, the network is trained using the second image data 400 of the most likely target organ, which is acquired after the primary tumor appears but before the metastatic lesions appear. The network is, for example, a convolutional neural network, so that the learning algorithm is a deep learning algorithm based on a convolutional neural network architecture.

[0139] For training purposes, the third image data 402 is annotated as to the further course of the cancer, in particular the exact location (and extent) of metastatic lesions within the imaged target organ. Thus, in the diagnostic phase, after the primary tumor appears, the second image data 400 and the third image data 402 of the most likely target organ are acquired and fed into the network.

[0140] The information subsequently delivered by the trained network in the application phase can have different levels of detail:

[0141] The overall probability that the target organ is completely affected or not affected at all (ie, a yes / no decision in the absence of localization).

[0142] A finite number N ≥ 1 of discrete locations (i.e., xn / yn / zn coordinates, 1…n…N) where the probability of a metastatic lesion occurring in the target organ is at most N.

[0143] A probability map across the target organ (anatomical region), discretized on a grid with a certain voxel size. The discrete locations described in the previous item will appear as local maxima in this probability map.

[0144] The images acquired during the training and diagnostic phases can be from a single or multiple imaging modalities, such as magnetic resonance (MR), computed tomography (CT), ultrasound (US), or positron emission tomography (PET). In addition, it can be advantageous to acquire multiple images with different contrasts using each modality (e.g., images with T1 / T2 / diffusion weighting, images before and after contrast injection, etc.). Of course, an appropriate compromise between imaging workload and prediction power must be found. In addition, the method is not limited to imaging the data, but can also incorporate information such as age, gender, laboratory values, or patient history as input to the learning algorithm 404 and the first machine learning model 154.

[0145] Even though a wide range of machine learning algorithms can be used for intra-organ prediction of metastatic lesions, deep learning techniques can be considered a preferred embodiment. Deep learning provides a unified solution for the three different use cases outlined above. In order to predict the probability of metastatic lesions in the presence of a primary tumor, a convolutional neural network (CNN) can be employed. A CNN comprises a sequence of interconnected layers (i.e., convolutional layers, max pooling layers...) (which are similar to the feature extraction steps in a classical image processing pipeline), followed by one or more fully connected layers. CNNs are typically trained to predict class labels or probabilities for natural images or to provide diagnoses for medical images, but the basic concepts can be transferred to describe the field of metastatic lesion prediction for those skilled in the art.

[0146] Even though CNNs were originally developed primarily for processing image data, it should be noted that the non-image data mentioned above (tumor descriptions, medical history data) can be easily integrated, for example with additional input connections at the level of fully connected layers.

[0147] To predict the location of metastatic lesions (including their probability), a region proposal network can be considered. Here, in contrast to conventional CNNs, the output neurons no longer correspond to discrete class labels, but to voxel locations and metastatic lesion probabilities. For this purpose, it may be beneficial if the image of the target organ (anatomical region) is transformed in a normalized reference frame.

[0148] Finally, deep learning-based techniques can also be used to generate probability maps. Given an image of a primary tumor, combined with annotated metastatic lesion regions in the target organ, a slice-by-slice analysis of the corresponding CNN (centered on the voxel of interest) can be used to estimate the probability at the voxel level. Alternatively, a more efficient network structure from the field of semantic image segmentation can be adopted.

[0149] The present invention can be extended not only to predict the location of metastatic lesions, but also to predict the time point at which metastatic lesions appear. If the training set has a plurality of third image data 402 showing the temporal development of metastatic lesions, this information can be output together with the predicted image 302. Figure 3 In the example, the output may indicate that the left predicted image 302 has an 83% probability of having a metastatic lesion 300 at the indicated location within the next 2 months, the middle predicted image 302 has a 72% probability of having a metastatic lesion 300 at the indicated location within the next 4 months, and the right predicted image 302 has a 66% probability of having a metastatic lesion 300 at the indicated location within the next 6 months.

[0150] It is important to note that the above principles can be applied to situations where: (a) a local lesion is expected in a certain area; (b) the exact location of the expected lesion is not known in advance; and (c) knowing the exact location of the expected lesion can provide a therapeutic and / or diagnostic advantage. All three of these conditions can be met by the expected metastatic lesion based on the primary tumor.

Claims

1. A computer-implemented medical analysis method for predicting metastatic lesions (300) in a test tissue sample, the method comprising: receiving a plurality of different training tissue samples of the training tissue samples for an anatomical region (306), each training set comprising, for a corresponding one of the training tissue samples, second image data (400) and third image data (402), the third image data (402) being data acquired after a metastatic lesion (300) occurs in a subject from which the training tissue sample originates, the second image data (400) representing the training tissue sample without the metastatic lesion (300), wherein each training set further comprises a description (401) of a tumor (304) in the subject from which the training tissue sample originates; Running a learning algorithm (404) on the training set to generate a trained first machine learning model (154) having an input and an output; receiving a description (401) of the tumor (304) and first image data (148) of a test tissue sample of the anatomical region (306), the anatomical region being separate from the tumor and the test tissue sample being free of metastatic lesions (300); providing the first image data (148) and the description (401) of the tumor (304) to the input of the first machine learning model (154); receiving, in response to the providing, from the output of the first machine learning model (154), a prediction of the occurrence of a metastatic lesion (300) originating from the tumor (304) in the test tissue sample; and The prediction is provided.

2. The method according to claim 1, wherein: The second image data (400) and the third image data (402) represent a 3D volume of an anatomical structure.

3. According to the method according to claim 1 or 2, the description (401) of the tumor (304) includes any one of the following items: the spatial position of the tumor (304) relative to the anatomical structure of the object carrying the tumor (304), the classification of the tumor (304), and the image data of the tumor.

4. The method according to claim 1 or 2, wherein the prediction of the occurrence of the metastatic lesion (300) comprises any one of the following: an indication of whether the test tissue sample will be affected by metastatic lesions (300) originating from the tumor (304), A probability map across the anatomical region (306) having an indication of which discrete locations of the anatomical region (306) will be affected by metastatic lesions (300) originating from the tumor (304).

5. The method according to claim 4, wherein the indication comprises any one of the following: a probability value describing the probability that the metastatic lesion (300) will be expected, A binary value describing whether metastatic lesions are to be expected (300).

6. According to the method according to claim 1 or 2, the first image data (148), the second image data (400), the third image data (402) and the image data of the tumor are any one of the following: magnetic resonance image data, computed tomography image data, ultrasound image data, positron emission tomography image data, X-ray data.

7. The method according to claim 1 or 2, further comprising: Medical history data of the subject with the tumor (304) is received and provided to the input of the machine learning model, wherein the training set also includes the medical history data of the subject with the tumor (304).

8. The method according to claim 4, wherein the third image data (402) is a time-resolved image data set, wherein: The indication is time resolved.

9. The method according to claim 1 or 2, further comprising: The third image data (402) is analyzed for the presence of metastatic lesions (300) in corresponding training tissue samples, and the third image data (402) is annotated with respect to the results of the analysis.

10. The method according to claim 1 or 2, wherein the learning algorithm is a deep learning algorithm.

11. The method of claim 10, wherein the deep learning algorithm is based on a convolutional neural network architecture, and the method further comprises: transforming the second image data (400) and the third image data (402) into a common reference image frame of the anatomical region (306) for all training sets, The running of the learning algorithm (404) is performed on the training set including the transformed image data to generate the first machine learning model (154).

12. A computer program product comprising machine executable instructions for execution by a processor, wherein: Execution of the machine executable instructions causes the processor to perform a method according to any one of the preceding claims.

13. A medical analysis system (210) for predicting metastatic lesions (300) in a test tissue sample, the medical analysis system (210) comprising a memory (202) containing machine executable instructions (152) and a processor (200) for controlling the medical analysis system (210), wherein: Execution of the machine executable instructions causes the processor (200): receiving a plurality of different training tissue samples of the training tissue samples for an anatomical region (306), each training set comprising, for a corresponding one of the training tissue samples, second image data (400) and third image data (402), the third image data (402) being data acquired after a metastatic lesion (300) occurs in a subject from which the training tissue sample originates, the second image data (400) representing the training tissue sample without the metastatic lesion (300), wherein each training set further comprises a description (401) of a tumor (304) in the subject from which the training tissue sample originates; Running a learning algorithm (404) on the training set to generate a trained first machine learning model (154) having an input and an output; receiving a description (401) of the tumor (304) and first image data (148) of a test tissue sample of the anatomical region (306), the anatomical region being separate from the tumor and the test tissue sample being free of metastatic lesions (300); providing the first image data (148) and the description (401) of the tumor (304) to the input of the first machine learning model (154); receiving, in response to the providing, from the output of the first machine learning model (154), a prediction of the occurrence of a metastatic lesion (300) originating from the tumor (304) in the test tissue sample; and The prediction is provided.

14. An MRI system (100) comprising the medical analysis system (210) according to claim 13, the MRI system being configured for acquiring the first image data (148).

Images