Method of predicting transfer site

Abstract

A method of predicting a transfer site is disclosed. The present invention describes a computer-implemented method of predicting a secondary site of a primary tumor, the method comprising obtaining a full-slide image of a primary tumor of a patient; obtaining a primary site descriptor of a primary site of the primary tumor; the full-slide image and the primary site descriptor are input to a deep learning system that is previously trained to determine an accidental transition probability from the full-slide image and the primary site descriptor of the primary tumor to identify any body site where a secondary tumor may occur, and predicting a transition probability associated with each identified body part; and outputting a secondary site prediction for each identified body site of the patient, where the secondary site prediction includes a descriptor of a body site and a transition probability associated with the body site. The invention also describes a data processing system adapted to perform such a computer-implemented method.

Description

Technical Field

[0001] The present invention relates to methods for predicting metastatic sites. Background Art

[0002] Metastasis is the main cause of cancer-related death, and a lot of effort has been invested in understanding the mechanism of metastasis. The nature of metastasis is largely based on the characteristics of cancer cells. The ability to estimate the probability that a primary cancer will undergo metastasis with a certain degree of certainty has provided valuable information to the treating physician, which can help decide the treatment plan for individual patients. However, depending on the nature of the primary tumor, secondary sites can be established at various different body regions. For example, circulating lung cancer cells can establish metastasis in bones, brain, etc.; ovarian cancer cells can establish metastasis in lungs, liver, etc. Therefore, before the metastasis at these secondary sites is identified in subsequent examinations, the treatment plan cannot include the steps of targeting specific types of metastasis, and the estimation of the location where metastasis may develop depends largely on the experience of the treating physician. For this reason, the treating practitioner can arrange a biopsy (to obtain cancer cells from the primary tumor) and laboratory analysis to obtain data on the genomics, proteomics, transcriptomics, etc. (collectively referred to as "omics") of the primary tumor cells. However, while prior art methods can be used to determine whether a primary tumor will undergo metastasis at some point, the treating physician must rely on experience to detect actual metastases ("mets") after these metastases have begun to develop at one or more secondary sites. The problem with this approach is that metastases are initially very small and therefore difficult to detect, and can establish at any number of essentially random locations. Delays in detecting metastases and corresponding delays in initiating treatment are often detrimental to patient outcomes. Summary of the invention

[0003] It is therefore an object of the present invention to provide an improved method for assessing information related to a primary tumor.

[0004] This object is achieved by the claimed computer-implemented method for predicting secondary sites of a primary tumor and by the claimed data processing system.

[0005] According to an embodiment of the present invention, a computer-implemented method includes the following steps: obtaining a whole slide image of a biopsy of a primary tumor taken from a patient; obtaining a primary site descriptor of the primary site; inputting the whole slide image and the primary site descriptor into a deep learning system (DLS) that has been previously trained to identify the probability of incidental metastasis, one or more body parts where metastasis of the primary tumor may occur, and the probability of metastasis development at each identified body part; and outputting a secondary site prediction for each identified body part of the patient to a user. Each such secondary site prediction includes a descriptor of the corresponding body part and an associated probability of metastasis development at that body part ("secondary metastasis probability" or simply "metastasis probability"). The incidental metastasis probability may also be output to the user.

[0006] According to an embodiment of the present invention, a computer-implemented method for predicting secondary sites of a primary tumor includes: obtaining a whole slide image of a patient's primary tumor, obtaining a primary site descriptor of the primary site of the primary tumor; inputting the whole slide image and the primary site descriptor into a deep learning system, the deep learning system previously trained to predict the (secondary) metastasis probability for one or more body parts of the patient based on the whole slide image of the primary tumor and the associated primary site descriptor, respectively, the (secondary) metastasis probability indicating the possibility of a secondary tumor of the primary tumor occurring in each of the one or more body parts; and outputting a secondary site prediction for at least one body part of the patient, wherein the secondary site prediction includes a descriptor of at least one body part (secondary site) and a (secondary) metastasis probability associated with the at least one body part.

[0007] The whole slide image may be a two-dimensional digital image having a plurality of pixels. The size of the whole slide image may be at least 4.000×4.000 pixels, or at least 10.000×10.000 pixels, or at least 1E6×1E6 pixels.

[0008] A full slide image can image a tissue section or slide of a patient. Preparing a tissue section from a tissue sample can include preparing a section from the tissue sample (e.g., using a punch tool), wherein the section is cut into micrometer-thick slices, i.e., a tissue section. Another word for a section is a block or punch biopsy. Under a microscope, a tissue section can show the fine tissue structure of a tissue sample, and in particular the cell structure or cells contained in the tissue sample. When observed on a larger length scale, a full slide image can show an overview of the tissue structure and tissue density. The tissue may be taken from a tumor that the patient suffers from. In particular, the tissue can show the manifestation of a patient's cancerous disease, such as cells of a tumor.

[0009] The preparation of tissue sections can also include staining the tissue sections using histopathological stains. In this case, staining can be used to highlight different structures in the tissue sections, such as, for example, cell walls or nuclei, or to test medical indications, such as, for example, cell proliferation levels. In such cases, different histopathological stains are used for different purposes.

[0010] In order to create a full slide image, the stained tissue section is digitized or scanned. For this reason, the tissue section is scanned using a suitable digitizing station such as a full slide scanner, which preferably scans the entire tissue section mounted on the object carrier and converts it into a pixel image. In order to maintain the color effect from histopathological staining, the pixel image is preferably a color pixel image. Because the overall impression of the organization and the finely resolved cell structure may be important in prediction, the full slide image usually has a very high pixel resolution. The data size of a single image can usually reach several gigabytes.

[0011] A descriptor of a primary site (hereinafter "primary site descriptor") is to be understood as information identifying the site of a primary tumor, and may be in the form of a text label, body coordinates or other suitable annotation, e.g. added to a whole body image to specify the organ or organ sub-region where the primary tumor is located. The same applies to secondary site descriptors.

[0012] The one or more body parts may include an organ or body compartment of the patient. A compartment may generally relate to an entity of the patient's body. For example, a compartment may relate to an organ, an organ part, an organ function, an anatomical structure, an anatomical body, a functional unit of the patient's body, etc.

[0013] In particular, the deep learning system can be trained to predict the probability of (secondary) metastasis in each of a plurality of different body parts of a patient. The plurality of different body parts can be predetermined. The plurality of different body parts can include organs of the patient, such as the liver, lungs, lymphatic system, brain, etc. In particular, the deep learning system can be trained to identify one or more body parts where metastasis of a primary tumor may occur, and the probability of development of a metastasis at each identified body part.

[0014] The one or more body parts may be different from the body part at which the biopsy was performed.

[0015] The output may include outputting a secondary part prediction for each of the plurality of body parts or for each of a subset of the plurality of body parts. In particular, the output may include outputting a secondary part prediction for each of the plurality of body parts whose (secondary) transition probability exceeds a predetermined threshold.

[0016] When a primary tumor metastasizes, circulating tumor cells can be transported to various other locations (organs, body parts, body compartments, etc.), and the same type of tumor can begin to develop at such new pathological sites or abnormal locations. The location where metastasis occurs is usually referred to as a "secondary site". Compared with the prior art methods that estimate the development of metastasis based solely on omics data, in the method of the present invention, the morphological characteristics of primary tumor cells in the whole slide image are considered according to the primary tumor site to obtain metastasis predictions specifically for each body part of the patient. The advantage of the computer-implemented method of the present invention is that it provides valuable information about the most likely future development of the primary tumor to users such as oncologists, radiologists, etc., helping practitioners decide on a follow-up monitoring plan even before the primary tumor undergoes metastasis. In this way, by monitoring the most likely secondary site locations, any metastasis at such a location can be identified and treated at a very early stage. Another advantage is that biopsy and subsequent pathological examination are standard procedures for cancer cases, with 95% of cancer cases having (digital) pathological examinations arranged. This is in contrast to sequencing tests that are arranged in relatively small numbers. Thus, the method of the present invention exploits readily available data and can therefore be well integrated into established clinical routine without requiring additional diagnostic costs and efforts.

[0017] According to some examples, the deep learning system is configured to determine a probability of incidental metastasis of a primary tumor based on a whole slide image of the primary tumor and / or a primary site descriptor, and to predict a probability of (secondary) metastasis associated with one or more body sites based on the probability of incidental metastasis and the primary site descriptor.

[0018] In particular, the inventors have recognized that the combination of a whole slide image of a primary tumor and the location of the primary tumor in the patient provides a good basis for predicting where secondary tumors of the primary tumor are likely to occur in the patient. Thus, a whole slide image can provide information about the "aggressiveness" of the primary tumor, while its location indicates which body parts are susceptible to being affected, for example, due to proximity to the primary tumor itself and / or typical metastatic pathways such as the blood vessels or lymphatic system. For the same reason, the location of the primary tumor may also affect the probability of incidental metastasis. Thus, according to some examples, the primary site descriptor may also be considered to predict the probability of incidental metastasis.

[0019] As an alternative to the use of a deep learning system, metastasis prediction can be rule-based, for example, involving the use of medical guidelines that can list metastasis pathways. In turn, a deep learning system can also be trained based on such medical guidelines.

[0020] As another alternative, a similar patient search module may be provided that is configured to search for patients similar to the patient under consideration. Cases of similar patients may then be searched for transferred files. For similar patient searches, patient data and optionally incidental transfer probabilities may be used as query vectors to a patient database, such as an electronic medical record system.

[0021] According to some examples, the method further comprises generating a body model of the patient and registering the primary site and any secondary sites with the body model, wherein the step of outputting comprises generating a rendering of the body model highlighting the primary site, any secondary sites and corresponding (secondary) metastasis probabilities.

[0022] According to some examples, the body model can be generated based on the patient's basic health and demographic data, such as gender, age, weight, height, etc. The basic health and demographic data can be obtained from the patient's electronic medical records. Generating the body model can include adapting a standard body model based on the basic health and demographic data.

[0023] Registering the site descriptors may include determining the location of the site in the body model. The highlighting may mean that the primary site (location), the secondary site (location) and the (secondary) transition probability are visually conceivable to the user in the rendering. For example, the (secondary) transition probability may be highlighted by inserting a number, such as a percentage, reflecting the (secondary) transition probability in the rendering. Additionally or as an alternative, the (secondary) transition probability may be highlighted in a color-coded form.

[0024] According to the present invention, a data processing system suitable for performing such a computer-implemented method comprises: an input interface for receiving a whole slide image of a primary tumor of a patient and a primary site descriptor of a primary site of the primary tumor; a deep learning system previously trained to determine the probability of incidental metastasis of the primary tumor based on the whole slide image and the primary site descriptor to identify any body site where a secondary tumor may arise and predict a (secondary) metastasis probability associated with each identified body site; and an output interface for outputting a secondary site prediction for each identified body site of the patient, wherein the secondary site prediction comprises the descriptor of the body site and the (secondary) metastasis probability associated with the body site.

[0025] The invention also describes a computer program product comprising instructions which, when the program is executed by a processor of such a data processing system, cause the data processing system to execute the steps of the method of the invention.

[0026] The units or modules of the data processing system (in particular the deep learning system) mentioned above can be fully or partially implemented as software modules running on the processor of the data processing system. The implementation mainly in the form of software modules can have the advantage that applications already installed on existing systems can be updated with relatively little effort to install and run the computer program product of the present application. In addition to the computer program, such a computer program product may also include other parts such as documents and / or additional components, and also include hardware components that facilitate access to the software, such as hardware keys (software dogs, etc.).

[0027] Computer readable media such as a memory stick, hard disk or other removable or permanently mounted carrier may be used to transfer and / or store executable parts of the computer program product so that these can be read from a processor of the data processing system.The processor unit may include one or more microprocessors or their equivalents.

[0028] As disclosed in the following description, particularly advantageous embodiments and features of the invention are given by other embodiments. Features of different embodiment categories can be appropriately combined to give other embodiments not described herein.

[0029] The data processing system of the present invention includes a first stage and a second stage, in which the whole slide image is analyzed to determine the sporadic probability of metastasis, and in the second stage, the potential secondary site locations and the probability of metastasis occurring at those secondary site locations are predicted. In the first stage of the data processing system, the deep learning system preferably includes a first machine learning algorithm, which is previously trained to predict the probability of sporadic metastasis of the primary tumor based solely on the whole slide image, that is, the probability of sporadic metastasis is derived from the morphological features of the primary tumor cells shown in the whole slide image. The first machine learning algorithm can be, for example, a residual network ("ResNet") trained to extract image data from the whole slide image and convert the image data into a feature vector. In order to improve image processing, the whole slide image can be divided into an array or matrix of "patches", and a feature vector can be extracted for each patch. The set of features can then be averaged to obtain an average feature vector for the whole slide image. For example, such a machine learning algorithm can be an off-the-shelf product that can be used in medical image processing tools.

[0030] In the second stage of the data processing system, the deep learning system preferably includes a prediction module that is previously trained to determine potential secondary sites in the patient and the probability of metastasis at each secondary site based on the primary site descriptor and the probability of incidental metastasis. The prediction module can be, for example, a regression model that is preferably based on gradient enhancement.

[0031] The deep learning system of the present invention is preferably trained using data from a patient cohort to predict the occurrence of metastases in different body parts based on knowledge of primary cancerous growths. The training data set for the deep learning system of the data processing system of the present invention preferably includes at least one full slide image of the primary tumor, a primary site descriptor, and information about the occurrence of primary tumor metastases in the patient. Preferably, many such training data sets, such as thousands of data sets, are used to train the deep learning system. For example, the training data set can be obtained from one or more picture archiving and communication systems (PACS).

[0032] Because the probability of incidental metastasis is determined based on features identified in the whole slide image in the first stage of the data processing system, this portion of the deep learning system can be referred to as the "image branch," and the probability of incidental metastasis can be considered to be a feature or vector derived from the image, or can be considered to be encoded in such a feature or vector. In the second stage of the data processing system, prediction of the location of secondary site metastases and their corresponding metastasis probabilities can be performed on this vector alone.

[0033] The prediction of secondary site metastasis or metastasis prediction of different body sites should be understood as the probability of metastasis occurring at a specific secondary site. The results of the second stage can be in the form of probabilities for each secondary site, for example, a 50% probability of metastasis forming in the liver and a 10% probability of metastasis forming in the intestine.

[0034] By predicting metastasis to different body compartments, selectable results are provided to the physician. Using the information provided by the data processing system of the present invention, the practitioner can retrieve already existing (medical) diagnostic images from the patient's EMR and examine these images in light of the predicted secondary sites. Likewise, the practitioner can promptly schedule further radiological examinations to obtain images showing the location of the identified secondary sites. According to some examples, the (medical) diagnostic images may include one or more radiological medical images.

[0035] The radiographic medical image may include one or more medical image datasets. The medical image dataset may be a two-dimensional image. Furthermore, the medical image dataset may be a three-dimensional image. Furthermore, the medical image may be a four-dimensional image, in which there are three spatial dimensions and one dimension similar to time. Furthermore, the medical image dataset may include multiple individual medical images.

[0036] Medical image data sets include image data, for example, in the form of a two-dimensional or three-dimensional array of pixels or voxels. Such an array of pixels or voxels may represent color, intensity, absorption or other parameters that vary with two-dimensional or three-dimensional position and may be obtained, for example, by appropriate processing of measurement signals obtained by a medical imaging modality or an image scanning device.

[0037] A medical image dataset may be a radiological image dataset depicting a body part of a patient. Thus, it may contain two-dimensional or three-dimensional image data of a body part of a patient. The medical image may represent an image volume or a cross-section through an image volume. A body part of a patient may be included in the image volume.

[0038] Medical imaging modalities correspond to systems for generating or producing medical image data. For example, medical imaging modalities may be computed tomography systems (CT systems), magnetic resonance systems (MR systems), angiography (or C-arm X-ray) systems, positron emission tomography systems (PET systems), etc. Specifically, computed tomography is a widely used imaging method and utilizes "hard" X-rays generated and detected by a spatially rotating instrument. The resulting attenuation data (also referred to as raw data) is processed by computer analysis software that generates detailed images of the internal structures of a patient's body part. The resulting image set is referred to as a CT scan, which may constitute a plurality of series of continuous images, presenting the internal anatomical structure in a cross section perpendicular to the axis of the human body. Magnetic resonance imaging (MRI), to provide another example, is an advanced medical imaging technique that utilizes the effect of an effective magnetic field on the movement of protons. In an MRI machine, the detector is an antenna, and the signal is analyzed by a computer to create a detailed image of the internal structure in any section of the human body.

[0039] Medical images can be stored in a standard image format such as the Digital Imaging and Communications in Medicine (DICOM) format, and stored in a memory or a computer storage system such as a picture archiving and communication system (PACS), a radiology information system (RIS), etc. Whenever DICOM is mentioned herein, it should be understood that this refers to the "Digital Imaging and Communications in Medicine" (DICOM) standard, for example, according to the DICOM PS3.1 2020c standard (or any later or earlier version of the standard). The data processing system of the present invention may include a plurality of additional "non-image branches". For example, in another embodiment of the present invention, the data processing system includes a second branch, and the DLS includes a second machine learning algorithm previously trained to derive features from omics data. The second machine learning algorithm may be a convolutional neural network (CNN) previously trained to obtain molecular markers from sequenced genomic data in the non-image branch. For example, such a trained machine learning algorithm may be a ready-made product that is already available for laboratories responsible for analyzing biopsy tissue. For example, omics data can be obtained by high-throughput sequencing of materials such as primary tumor biopsies. Morphological clues obtained from whole-slide images about the propensity of a primary tumor to metastasize can then be augmented by insights from molecular analyses.

[0040] In such an embodiment, the prediction module is configured to determine a metastasis prediction based on features derived from the image and also based on features derived from omics. The training dataset for the deep learning system thus preferably also includes any omics data associated with the whole slide images of the dataset. For example, additional omics data can be obtained from the clinic's electronic medical record (EMR) database, as omics analyses are typically stored as part of the patient's EMR.

[0041] In another embodiment of the present invention, the data processing system includes another non-image branch, and the DLS includes a third machine learning algorithm that has been previously trained to derive features from data such as medical records, preferably EMRs available to the data processing system. It is well known that factors such as age, comorbidities, dietary habits, smoking, drinking, etc. may affect the likelihood of metastasis in different body parts, and such information may be stored in the patient's EMR. Other information on the record may include digital entries, screenshots, patient-related data such as age, gender, etc. The third machine learning algorithm may be a large language model (LLM) based on a transformer architecture. Such a trained machine learning algorithm may be, for example, an off-the-shelf product that is already available for extracting information from an EMR database. In addition, if a recent report indicates that there are no findings for a particular body part, this may give an indication to the practitioner. Similarly, the likelihood of metastasis in that body part may be corrected based on the output of the deep learning system.

[0042] In such an embodiment, the prediction module is configured to determine the metastasis prediction based on features derived from the images and also based on features derived from the records, and may also make its prediction based on the above-mentioned features derived from omics. The training dataset for the deep learning system therefore preferably also includes the patient's EMR obtained, for example, from the clinic's archiving system.

[0043] Features derived from the image and optionally from the recording and / or from omics may be combined to obtain a concatenated vector as input to the second stage of the DPS or “prediction module”.

[0044] In another embodiment of the invention, the output of the second stage can be in the form of a schematic body model of the patient, and each predicted location or body compartment can be highlighted according to the likelihood of metastasis at that location. The output of the prediction module also includes an estimate of the severity of metastasis that may develop at the predicted location.

[0045] In another embodiment of the invention, the method of the invention comprises the following additional steps: querying an image database (e.g., a local PACS database) to obtain previously acquired medical diagnostic images, in particular radiological ones, for the patient; retrieving at least one image of the previously acquired radiological images based on the secondary site prediction (wherein at least one image shows the corresponding body part of the secondary site prediction); and providing at least one retrieved image. In another embodiment, the method comprises the following steps: subsequently identifying the type of each retrieved image (e.g., MRI, CT), and deploying a method for analyzing the image type to identify metastases present in the retrieved images. Algorithms such as segmentation algorithms are generally suitable for specific types of medical images, for example, image processing of CT images may depend on the image reconstruction parameters used to create the images. In this way, hitherto undetected metastases may be found in the retrieved images. In another embodiment, the method of the invention comprises the following additional steps: notifying the corresponding practitioner (the physician or clinician who originally ordered the diagnostic image) of the newly detected metastasis.

[0046] According to some examples, the query can be based on an electronic patient identifier of the patient registered in the image database. In addition, the query can be based on metadata respectively associated with the medical diagnostic images, the metadata indicating the body part depicted by the associated medical diagnostic image. For example, such metadata can include a DICOM header of the corresponding medical diagnostic image. According to some examples, the query can be limited to "recent" medical diagnostic images, which can provide insight into the progression of the patient's cancer disease. For example, recent can mean a period of one month, two months, or three months around the time point when the biopsy used to prepare the whole slide image was completed and / or any time period after the time point when the biopsy used to prepare the whole slide image was completed.

[0047] According to some examples, the method may further include indicating in the body model the medical diagnostic images included in the image database according to the respectively depicted body parts. Specifically, it may be indicated in the body model that the medical diagnostic images are available for a specific body part. Thus, the user may obtain an instant overview of the available information. On this basis, the user may selectively call up the medical diagnostic images or infer at a glance where the medical diagnostic image information is missing to assess possible metastasis.

[0048] The data processing system of the present invention is preferably also configured to determine whether all relevant body compartments (parts of the body where metastasis may occur) are covered by the images retrieved from the medical image database. If not, the data processing system informs the user of any missing diagnostic images, for example, by appropriate annotations to the body model.

[0049] Likewise, the method of the present invention may include the step of suggesting a radiological examination for the identified secondary site location for which the most recent diagnostic image is not available. This function may include the step of selecting imaging parameters with which the relevant medical imaging modality (e.g., contrast agent administration, MRI pulse sequence, etc.) should be controlled to obtain an optimal image showing the predicted secondary site. This function may also include the step of automatically scheduling corresponding imaging and examination tasks.

[0050] Specifically, according to an embodiment, if the image database does not include a medical diagnostic image showing at least one body part, the method may further include: generating a medical imaging protocol suitable for generating a medical diagnostic image showing at least one body part, and outputting the medical imaging protocol. The imaging protocol may include human-readable and / or machine-readable instructions. For example, the instructions may include steps and / or parameters for performing a medical examination intended to obtain a medical diagnostic image showing at least one body part.

[0051] According to some examples, the imaging protocol includes machine-readable instructions that are configured to control a modality configured to perform a medical examination to obtain a medical diagnostic image showing at least one body part, and the method also includes: inputting / forwarding the imaging protocol to the modality (so as to control the modality to perform the medical examination), and / or providing instructions to a user via a user interface (for viewing and further use).

[0052] According to some examples, the method also includes retrieving a medical diagnostic image of a patient showing at least one body part from an image database, applying a detection function to the retrieved medical diagnostic image to generate a detection result, wherein the detection function is configured to detect a tumor in the medical diagnostic image, and outputting the detection result.

[0053] The detection function may generally be configured to detect medical findings or anomalies in medical image data, in particular cancerous growths / tumors, in particular secondary tumors. In principle, a plethora of functions and methods are known for such computer-aided detection and classification of anomalies, all of which may be implemented in anomaly detection algorithms. For example, reference is made to US2009 / 0 092 300A1, US2009 / 0 067 693A1 or US2016 / 0 321 427A1, the contents of which are incorporated herein by reference in their entirety.

[0054] The detection result may include an indication of the presence of a finding, in particular a (secondary) tumor, in the medical diagnostic image. According to some examples, the detection result may include the location of the (secondary) tumor and / or the type of the (secondary) tumor and / or one or more features of the (secondary) tumor, such as size, contour (e.g., spiculation or lobulation level) or composition (e.g., calcification level). Furthermore, the detection result may include an indication that no finding / no secondary tumor was detected in the medical diagnostic image.

[0055] Outputting the detection result may include outputting the detection result to a user via a user interface. Further outputting the detection result may include indicating the detection result in the body model. The latter may include registering the detection result with the body model to determine a corresponding position of the detection result in the body model, and highlighting the detection result at the corresponding position in a rendering of the body model.

[0056] In another embodiment of the invention, the method of the invention comprises an additional step of grading the probability of incidental metastasis. To this end, the deep learning system may have been trained to also predict the severity of secondary tumors based on the primary tumor type and location and using other patient-related information such as the patient's age, smoking history, pre-existing conditions, etc.

[0057] In another embodiment of the present invention, the deep learning system can be configured to include a patient search module that is implemented to search the database for medical records of patients similar to the patient under consideration. Similar patient cases can then be searched for transferred files. The data of the patient under consideration and / or the probability of incidental transfer can be used as a query vector for similar patient searches. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Other objects and features of the present invention will become apparent from the following detailed description considered in conjunction with the accompanying drawings. It should be understood, however, that the drawings are designed only for the purpose of illustration and not as a definition of the limits of the present invention.

[0059] Figure 1 is a schematic diagram illustrating a prior art method;

[0060] Figure 2 is a simplified block diagram illustrating a data processing system of the present invention;

[0061] Figure 3 An exemplary flow chart illustrating the method of the present invention is shown;

[0062] Figure 4 Another schematic diagram illustrating a data processing system of the present invention is shown;

[0063] Figure 5 The use of the invention during treatment of a patient is shown.

[0064] In the drawings, like reference numerals refer to like objects throughout. Objects in the drawings are not necessarily drawn to scale. DETAILED DESCRIPTION

[0065] Figure 1 is a schematic diagram showing the prior art treatment progress of a patient 5 with a primary tumor T0 at a specific body location. The body location can largely define the nature of the primary tumor T0. In the upper part, the figure indicates the early stage of cancer detected in the diagnostic image at time t0, that is, before the primary tumor T0 metastasizes. For the oncologist treating this patient 5, it is important to monitor the development of the cancer so that appropriate treatment can be arranged in time. In some cases, a type of primary tumor can remain unspread for a long time before undergoing metastasis, while the same type of tumor in another patient may experience metastasis faster. Once circulating cancer cells reach the bloodstream or lymphatic system, the next stage of cancer begins, and eventually metastasis ("mets" or secondary tumors) will appear in the (usually multiple) secondary sites shown. At this stage of cancer, it is crucial to detect secondary tumors as early as possible. In the past, oncologists, radiologists and other practitioners relied on their experience to determine the treatment plan that best suits the patient. However, prior art treatment plans can only be put together after the secondary tumor is detected, which in turn relies on the physician to schedule a radiology exam in time to obtain diagnostic images of the corresponding body part, e.g., at a relatively late time t1. In prior art methods, the delay D between the initial detection of the primary tumor at date t0 and the detection of the metastasis at date t1 typically has a negative impact on patient outcomes.

[0066] Figure 2 is a simplified block diagram illustrating a data processing system 1 of the present invention, and Figure 3 An exemplary flow chart illustrating the method of the present invention is shown. In a first step 31, a whole slide image 2 of the primary tumor of the patient under consideration is obtained. T0 , and the primary site descriptor 20 for that primary tumor T0 . Whole slide image 2 T0 It should be understood to show tissue obtained from a biopsy of a primary tumor T0. Whole slide image 2 T0 Can be stained with standard H&E stains. Primary site descriptor 20 T0 It may be any suitable marker that clearly identifies the location of the primary tumor T0, such as the organ from which the biopsy tissue was taken, biopsy coordinates, etc.

[0067] Next, in step 32, the whole slide image 2 T0 and primary site descriptor 20 T0is input to a deep learning system 11 , 12 which has been trained to predict one or more locations where a metastasis may occur.

[0068] The deep learning system includes a first stage 11, in which the probability of incidental transitions 11 is predicted met The first stage consists of training the whole slide image 2 T0 and primary site descriptor 20 T0 Predicting the probability of incidental transfer11 met The machine learning algorithm may include the following: T0 and / or EMR data 22 T0 Additional machine learning algorithms contribute to the prediction. The output of this first stage 11 is the sporadic transition probability 11 met The deep learning system 1 includes a second stage 12 having a machine learning algorithm trained to predict any secondary site locations (i.e., body sites or regions where metastasis from a primary tumor may occur) and the probability of incidental metastasis 11 met The probability of metastasis development at each secondary site location is calculated. In the subsequent step 33, the secondary site prediction 12 SS Each secondary site prediction 12 SS Includes an identifier or descriptor of a predicted secondary site location and the probability of metastasis development at that secondary site location. Predicted probability of incidental metastasis 11 met Can also be output to the user.

[0069] In this exemplary embodiment, the workflow includes step 34 of querying a database such as a PACS database 44 to obtain medical images previously ordered for the patient. Any retrieved images may be analyzed using a suitable algorithm as described above to examine each image for hitherto undetected metastases. An appropriate report 340 may be issued to a user of the DPS 1, or to any practitioner who has ordered diagnostic images in which a secondary site has now been found.

[0070] This may be followed by another step 35 in which a radiology workflow 350 is suggested to the medical practitioner for the patient in order to schedule an imaging procedure to obtain images of predicted locations where secondary sites may develop.

[0071] Figure 4 Another block diagram illustrating the method of the present invention is shown.Here, a data processing system 1 is shown comprising an image branch and two optional non-image branches.

[0072] In the image branch, obtain a whole slide image as described above 2 T0and primary site descriptor 20 T0 . It is trained on whole slide images 2 T0 and primary site descriptor 20 T0 Machine learning algorithms that predict sporadic transition probabilities 111 such as residual networks (“ResNet”) return sporadic transition probabilities 11 met Here, the probability of accidental transition is 11 met can be considered as the vector V1 obtained from the image.

[0073] In an (optional) non-image branch, a machine learning algorithm 112 such as an LLM trained to extract information from electronic medical records stored in an EMR database 42 returns features 22 derived from the records. T0 As the vector V2 obtained from the record. In another (optional) non-image branch, a machine learning algorithm 113 such as a CNN trained to extract information from the omics data 43 returns the features 21 obtained from the omics T0 as the vector V3 obtained from omics.

[0074] To exploit all this information, the image-derived vector V1, the recording-derived vector V2, and the omics-derived vector V3 may be combined as shown here to give a concatenated vector IIV as input to the prediction module 12, which then returns its secondary site location prediction 12 SS Of course, the prediction module 12 may make its prediction based only on the vector V1 derived from the image, on a combination of the vector V1 derived from the image and the vector V3 derived from omics, or on a combination of the vector V1 derived from the image and the vector V2 derived from the recordings.

[0075] Figure 5 is a schematic diagram illustrating the application of the present invention during treatment of a patient 5 having a detected primary tumor T0. Again, the diagram indicates an early stage of cancer detected at time t0. The practitioner treating the patient 5 schedules a biopsy to obtain a whole slide image 2 T0 and compare it with the primary site descriptor 20 T0 together with the input to an instance of the data processing system 1 of the present invention, which returns one or more secondary site predictions 12 SS Each secondary site predicts 12 SS A descriptor of the body region 50 where the mets of the primary tumor T0 are likely to develop is included, as well as the probability of metastasis at that body region. A follow-up image of the patient 5 can then be scheduled in the near future, as indicated again at time t1, so that a secondary tumor can be detected early at the predicted region 50.

[0076] Likewise, already existing images 440 showing those body regions may be retrieved from the medical image database 44 as described above and analyzed to detect lesions that may be present but not yet detected, in which case there is no delay until mets are detected (t1 = t0).

[0077] Since possible secondary sites of mets can be identified even before the primary tumor T0 metastasizes, the present invention substantially eliminates the unnecessary delay D common in prior art methods and can significantly improve patient outcomes.

[0078] Although the present invention has been disclosed in the form of embodiments and variations thereon, it should be understood that numerous additional modifications and variations could be made thereto without departing from the scope of the invention.

[0079] For the sake of clarity, it should be understood that throughout this application, the use of "a" or "an" does not exclude a plurality, and "comprising" does not exclude other steps or elements. Reference to a "unit" or "module" does not exclude the use of more than one unit or module. Wording in brackets indicates optional and / or clarifying features or words. Independent of grammatical term usage, individuals with male or female identities are included. Any pronoun indicating a particular gender should be understood to apply equally to any gender identity.

Claims

1. A computer-implemented method for predicting secondary sites of a primary tumor, the method comprising: - Obtain a full slide image of the primary tumor (T0) of patient (5) (2 T0 ); - obtaining a primary site descriptor (20) of the primary site of the primary tumor (T0) of the patient (5); T0 ); - The whole slide image (2 T0 ) and the primary site descriptor (20 T0 ) are input to a deep learning system (11, 12) that has been previously trained to predict the probability of metastasis of secondary tumors occurring in one or more body parts of the patient based on whole slide images of the primary tumor and associated primary site descriptors; and - outputting a secondary site prediction (12) for at least one body site (50) of the patient (5); SS ), wherein the secondary site prediction (12 SS ) comprises a descriptor of the at least one body part (50) and a transition probability associated with the at least one body part (50).

2. A computer-implemented method according to the preceding claim, wherein: The deep learning system (11, 12) is configured to: -Based on the whole slide image of the primary tumor (2 T0 ) and / or primary site descriptors (20 T0 ) determines the probability of sporadic transitions, and - predicting transition probabilities respectively associated with one or more body parts based on the incidental transition probabilities and the primary part descriptors.

3. A computer-implemented method according to any one of the preceding claims, wherein: The secondary site prediction (12 SS ) is also based on features derived from omics (V3).

4. A computer-implemented method according to any one of the preceding claims, wherein: The secondary site prediction (12 SS ) is also based on features obtained from the recording (V2).

5. A computer-implemented method according to any one of the preceding claims, comprising the steps of: An image database (44) is queried to obtain a medical diagnostic image (440) of the patient (5) showing the at least one body part (50).

6. The computer-implemented method of claim 5, further comprising: - if the image database (44) does not include a medical diagnostic image (440) showing the at least one body part (50): generating a medical imaging protocol suitable for generating a medical diagnostic image (440) showing the at least one body part (50), and - outputting said medical imaging protocol.

7. The computer-implemented method of any preceding claim, further comprising: - retrieving a medical diagnostic image (440) of the patient showing the at least one body part from an image database (44), - applying a detection function to the retrieved medical diagnostic image (440) to generate a detection result, the detection function being configured to detect a tumor in the medical diagnostic image, and - outputting the detection result.

8. A computer-implemented method according to any one of the preceding claims, comprising the steps of: A radiological examination of the at least one body part (50) is recommended.

9. A computer-implemented method according to any one of the preceding claims, comprising the steps of: - generating a body model of the patient, and - registering the origin site and the at least one body site with the body model, wherein - The step of outputting comprises generating a rendering of the body model highlighting the location of the primary site, the at least one body part and the corresponding transition probabilities.

10. A data processing system (1) adapted to perform a computer-implemented method according to any one of claims 1 to 9, the data processing system (1) comprising: - an input interface for receiving a whole slide image (2 T0 ) and a primary site descriptor of the primary site of the primary tumor in the patient (20 T0 ); a deep learning system (11, 12) previously trained to identify at least one body site of a patient where a secondary tumor of the primary tumor may arise based on a whole slide image of the primary tumor and a primary site descriptor, and to predict a probability of metastasis associated with each identified body site; and - an output interface for outputting a secondary site prediction (12) for each identified body site (50) of the patient (5); SS ), wherein the secondary site prediction (12 SS ) comprises a descriptor of the body part (50) and a transition probability associated with the body part (50).

11. A data processing system according to the preceding claim, wherein: The deep learning system includes a first machine learning algorithm (111) that is previously trained to generate a deep learning image based on the whole slide image (2 T0 ) predicts the probability of incidental metastasis of the primary tumor (T0) (11 met ).

12. A data processing system according to claim 10 or claim 11, wherein: The deep learning system includes a prediction module (12) previously trained to predict the primary site based on the primary site descriptor (20 T0 ) and the probability of the accidental transition (11 met ) to predict one or more transfer locations.

13. The data processing system according to the preceding claim, comprising a second machine learning algorithm (112), the second machine learning algorithm (112) being previously trained to derive omics features (21) from the omics data (4). T0 ), and among them, The prediction module (12) is also configured to be based on the features (21 T0 ) to predict one or more transfer locations.

14. The data processing system according to claim 13, wherein: The second machine learning algorithm (112) includes a convolutional neural network, which is trained to obtain molecular markers (21 T0 ).

15. The data processing system according to any one of claims 10 to 14, comprising a third machine learning algorithm (113), the third machine learning algorithm (113) being previously trained to derive features (22) from the medical records (3) of the patient (5). T0 ), and among them, The prediction module (12) is also configured to be based on the features (22) obtained from the records T0 ) to predict one or more transfer locations.

16. A data processing system according to the preceding claim, wherein: The third machine learning algorithm (113) includes a large language model based on a transformer architecture.

17. The data processing system according to any one of claims 10 to 16, wherein: The training data set used for the deep learning system (11, 12) includes: - at least one whole slide image (2 T0 ); - a primary site descriptor of the primary site of the primary tumor (20 T0 );as well as - Information on the occurrence of metastases from said primary tumor (T0).

18. A computer program product comprising instructions which, when the program is executed by a data processing system (1) according to any one of claims 10 to 17, cause the data processing system (1) to perform the steps of the method according to any one of claims 1 to 9.

19. A computer-readable data carrier having stored thereon a computer program product according to claim 18.

Images