Representation results learning for treatment response prediction of risk organs and total tumor volume

Abstract

The present disclosure relates to representation learning for treatment response prediction of risk organs and total tumor volume. For predicting the response of radiation therapy, radiomics is used for unsupervised machine training of an encoder-decoder network to predict based on input of image data, such as computed tomography image data, and from segmentation. Then, the trained encoder is used to generate latent representations to be used as input to different classifiers or regressors for predicting the treatment response, such as one classifier for predicting the response for a risk organ and another classifier for predicting another type of response for a risk organ or predicting the response for a tumor.

Description

Background Technology

[0001] This embodiment relates to treatment outcome prediction. Treatment procedures typically begin by defining the target or tumor to be eradicated, while avoiding damage to adjacent organs. Deep learning-based solutions are used to automatically delineate the contours of tumors and organs at risk. While organ and tumor boundaries are important for treatment planning, these boundaries may not be readily used for evaluating treatment efficacy and predicting treatment outcomes and toxicities.

[0002] The texture characteristics of tumors and organs in images capture information that can potentially be used for treatment-related responses. Radiomics uses image-based biomarkers to provide a deeper understanding of cancer biology to better support clinical decision-making. Radiomics practice typically involves the following discrete steps: a) acquiring images, b) identifying regions of interest (ROIs) and segmenting volumes, c) extracting and identifying descriptive features from the volumes, and d) developing models to predict outcomes. A key part of radiomics is extracting high-dimensional, quantified feature data to describe the properties of the ROI. In classical radiomics analysis, image features describing various physical and geometric properties of tumors are predefined and can be computed using mathematical formulas. These features typically quantify properties related to tumor intensity, shape, and texture, as well as wavelet transforms focused on the frequency domain. However, there are several limitations of existing radiomics techniques: 1) In order to maximize the information obtained in radiomics, a very large number of features containing a lot of redundant information are usually extracted from the images; 2) All handcrafted radiomics features are in a predetermined group, so it is possible that some predictive information is not fully captured by the predetermined features; and 3) Handcrafted radiomics are fixed representations of a specific ROI, and therefore are not suitable for different patient groups or clinical problems. Summary of the Invention

[0003] This provides systems, methods, and non-transitory computer-readable media for predicting treatment responses. Radiomics is used for unsupervised machine training of encoder-decoder networks to make predictions based on image data and segmentation data, such as computed tomography image data. The trained encoder is then used to generate latent representations to serve as input to different classifiers or regressors for predicting treatment responses, such as one classifier for predicting responses to risk organs and another classifier for predicting responses to risk organs or tumors.

[0004] In a first aspect, a method for predicting treatment responses using a machine is provided. Medical image data representing a patient is acquired. Risk organs and total tumor volume are segmented from the medical image data. Latent representations are generated from the medical image data, the segmented risk organs, and the total tumor volume. The latent representations are generated by an encoder, which, together with a decoder, is machine-trained in an unsupervised manner to predict radiomics features with baseline truth from hand-crafted radiomics. In response to the latent representations, the encoder is fed into a first trained classifier and a second machine-trained classifier, which predict treatment responses for the total tumor volume and risk organs. The predicted treatment responses for the risk organs and total tumor volume are output. This method can be provided for multiple risk organs.

[0005] In one embodiment, computed tomography data is acquired as medical image data. The handcrafted radiomics used to train at least one of the encoders and at least one of the decoders includes predetermined quantizations of tumor intensity, tumor shape, tumor texture, and wavelet transform.

[0006] As another embodiment, a machine learning network is used for segmentation. The prediction is made in response to inputs from the latent representation from the encoder and the values ​​of hidden features from the segmented machine learning network. In another approach, a feature pyramid network makes predictions from the values ​​and the latent representation, where multi-scale, hierarchical features are used as hidden features.

[0007] In another embodiment, the encoder is a convolutional neural network.

[0008] In another embodiment, the encoder and decoder are machine-trained together to predict radiomics features and segmentation. As another approach, the encoder and decoder are machine-trained together to predict radiomics features, segmentation, and region of interest information.

[0009] It can predict various treatment responses. For example, it can predict negative effects and toxicity for one or more organs at risk, and predict outcomes, progression, recurrence, and / or control for total tumor volume. Different machine-trained classifiers can be used to predict multiple negative effects, toxicity, and / or outcomes, and / or multiple progression, recurrence, and / or control, using inputs from latent representations from the same encoder.

[0010] According to one embodiment, the latent representation results are used to search for similar cases. The output may include a visualization of the latent representation results for the patient and similar cases.

[0011] In a second aspect, a medical system for treatment prediction is provided. A memory is configured to store computed tomography (CT) data. An image processor is configured to: segment a tumor or an organ adjacent to the tumor; generate values ​​of a latent space by a machine learning encoder in response to input CT data and the segmentation of the tumor or organ; predict a first treatment outcome by a first machine learning classifier or regressor in response to input values ​​of the latent space; and predict a second treatment outcome by a second machine learning classifier in response to input values ​​of the latent space. A display is configured to display the first and second treatment outcomes.

[0012] In one embodiment, the encoder is trained in an unsupervised manner along with the decoder to predict predetermined radiomics. The values ​​in the latent space are the bottleneck features of the encoder and decoder.

[0013] According to another embodiment, the first treatment outcome and the second treatment outcome are different treatment outcomes among negative effects, toxicity, results, progression, relapse, and control. The first machine learning classifier or regressor and the second machine learning classifier or regressor are trained separately using the same encoder.

[0014] In another embodiment, the image processor is also configured to generate visualizations of values ​​in the latent space.

[0015] In a third aspect, a method is provided for machine training of a model for treatment prediction. A network including an encoder and a decoder is machine-trained to predict radiomics values ​​in response to input medical imaging data. A first classifier or regressor is machine-trained to output a first treatment response in response to input feature values ​​from the encoder. A second classifier or regressor is machine-trained to output a second treatment response in response to input feature values ​​from the encoder. The encoder, the first classifier or regressor, and the second classifier or regressor are stored.

[0016] In one embodiment, machine training of the network includes unsupervised training, wherein radiomics includes values ​​quantized from hand-crafted functions.

[0017] In another embodiment, machine training of the first classifier or regressor and the second classifier or regressor includes machine training the first classifier or regressor for one of the negative effects or toxicities on the at-risk organ and machine training the second classifier or regressor for tumor outcomes, progression, recurrence, or control, both using the same encoder.

[0018] According to another embodiment, machine training of the network includes machine training of a decoder to predict radiomics values ​​and segmenting tumors and / or organs of risk based on treatment outputs. Machine training of the first classifier or regressor and the second classifier or regressor includes machine training using the segmentation and medical imaging data as input to the encoder.

[0019] As another embodiment, machine training of the second classifier or regressor includes training based on the output feature values ​​and segments, to be performed by a party different from the party performing the training of the network.

[0020] Any one or more of the aspects described above may be used individually or in combination. Any aspect of one method, system, or computer-readable medium may be used in others of the same method, system, or computer-readable medium. These and other aspects, features, and advantages will become apparent from the following detailed description of preferred embodiments, which are to be read in conjunction with the accompanying drawings. The invention is defined by the appended claims, and nothing in this section should be considered as limiting those claims. Further aspects and advantages of the invention will be discussed below in conjunction with preferred embodiments and may be claimed later, independently or in combination. Attached Figure Description

[0021] Components and figures are not necessarily drawn to scale; rather, the focus is on illustrating the principles of the embodiments. Furthermore, in the figures, the same reference numerals are used throughout different views to designate corresponding parts.

[0022] Figure 1 This is a flowchart of an embodiment of a machine training method for predicting treatment responses;

[0023] Figure 2 This is a block diagram of one embodiment of a machine learning or machine learning model used for predicting treatment response;

[0024] Figure 3 This is a flowchart of an embodiment of a method for predicting treatment response in a medical system; and

[0025] Figure 4 This is a block diagram of one embodiment of a medical system for predicting treatment. Detailed Implementation

[0026] The representations of risk organs and total tumor volume used for automated contouring are learned to build models for treatment planning, evaluation, and outcome / toxicity prediction for various treatment procedures such as radiotherapy. The examples used in this paper are for radiotherapy, but can be applied to other treatments. An unsupervised system capable of decoding general image representations for various tasks is formed. This system can be further used to stratify patients based on the likelihood of treatment failure or the impact of radiotoxicity, and redirect patients at high risk of failure and toxicity to alternative or modified treatment regimens. Unlike traditional radiomics features, which are often handcrafted, deep learning-based models are entirely data-driven.

[0027] Figure 1 This is a flowchart of one embodiment of a method for machine training of a model for treatment prediction. A portion of the model is trained in an unsupervised manner to output values ​​for predetermined radiomics, segmentation, and / or region of interest information. The resulting encoder, trained on this portion, creates a latent representation or fingerprint. For treatment outcome prediction, the encoder is used in conjunction with one or more (e.g., multiple) classifiers or regressors to predict the treatment response based on the latent representation.

[0028] Training is performed by an image processor using training data (e.g., data samples and corresponding baseline truth values ​​or computational costs) stored in a database. Other devices may also be used.

[0029] These actions are performed in the order shown (numerically or from top to bottom) or other orders. For example, action 110 is performed before or simultaneously with action 120. Additional, different, or fewer actions may be provided. For example, action 120 for segmentation may not be performed. As another example, such as when the encoder is used for a classifier, one of actions 140 or 142 may not be performed. In yet another example, additional actions are provided for training additional classifiers or regressors for different treatment responses and / or regions (e.g., other OARs or GTVs) using an unsupervised trained encoder.

[0030] In action 100, medical image data is acquired. The medical image data represents the patient, such as organs, head, or torso. In one embodiment, the medical image data is computed tomography (CT) data, such as two-dimensional or three-dimensional (volume) CT data. In other embodiments, ultrasound, magnetic resonance imaging (MRI), positron emission tomography (PET), or single-photon emission computed tomography (SPECT) image data is acquired. The image data is data arranged as pixels or voxels and may or may not be formatted for display. The image data may be used to generate a display image representing the patient or may be a display image itself.

[0031] In one embodiment, medical image data is acquired from a treatment planning scan. Imaging from the treatment planning phase or from tumor detection is used. Alternatively or additionally, medical image data is acquired during and / or after treatment.

[0032] Image data is acquired from a patient's electronic medical records, such as in an image archiving system or electronic patient medical record. The image data may come from a picture archiving and communication system (PACS). In other embodiments, image data is acquired from an imager (e.g., a CT system) and / or a database (e.g., a collection of examples).

[0033] Medical image data can be preprocessed. For example, a set of slices can be transformed and / or projected onto a known reference frame. Rotation, translation, and / or scaling can be performed. Any normalization can be used, such as remapping to a given dynamic range.

[0034] Medical image data will be used for machine training. Therefore, hundreds or thousands of samples will be collected.

[0035] In action 110, the processor or user extracts radiomics. For unsupervised training, quantization is used by the trained model to optimize or minimize the loss. For treatment, quantization is radiomics. Pre-defined radiomics are used, such as those shown as predictive radiomics in publicly available studies. One, some, or all of the pre-defined radiomics can be handcrafted, such as mathematical functions selected or created by the user. Exemplary handcrafted radiomics used to represent tumor phenotypes include size, shape, texture, and / or wavelet transform information. The image processor determines the values ​​of the radiomics for each sample in the training data (e.g., for each medical image or image dataset).

[0036] Other information can be extracted. For example, segmentation or contour extraction. As another example, different radiomics values ​​can be determined for different regions of interest in a sample, such as for different risk organs and / or different tumors. Information about the regions of interest can be extracted, such as the texture of healthy regions separated from tumors in the same organs where tumors are found. The extracted information can be determined automatically, rather than relying on supervised creation based on baseline truth. For segmentation, a handcrafted or previously trained segmenter can be used.

[0037] In action 120, one or more organs at risk (OARs) and / or tumors are segmented. The total tumor volume (GTV) or other portions of the tumor are segmented. Organs adjacent to the tumor and / or along the treatment application pathway are OARs. An OAR may include the organ in which the tumor is located. In one embodiment, both the GTV and one or more OARs are segmented.

[0038] This segmentation identifies contours such as surfaces or boundaries. Alternatively, the segmentation identifies regions or volumes, such as marking all voxels belonging to a given organ or tumor.

[0039] The acquired image data samples may include segmentation. Alternatively, segmentation may be performed by a user (e.g., a radiologist) or an image processor. For example, the image processor may segment the image data represented by GTV and OAR in each sample of the training data. In one embodiment, segmentation may be performed using any method, such as full width, half-maximum (FWHM), or intensity thresholding with various standard deviations. In another embodiment, an automatic contour drawing or segmentation model using machine learning may be employed, such as an image-to-image, U-Net, or encoder-decoder network trained to output a segmentation in response to input medical image data. The machine-trained segmenter generates values ​​of features in hidden layers in response to input medical images and uses these values ​​to output a segmentation.

[0040] Segmentation can be used as the output for determining the loss function during machine training. In other embodiments, segmentation is used as input to the extraction of action 110. The radiomics engine uses both CT volume and GTV / OAR segmentation to compute handcrafted radiomics features for each sample.

[0041] In other embodiments, no training data includes segmented tissue. The boundaries, pixel locations, or voxel locations of specific tissues (e.g., scars or lesions) are not depicted and therefore not used to train the multi-task model. Action 120 is not performed.

[0042] Actions 100 and 120 result in training data to be used for unsupervised or other machine learning. The training data consists of samples, each with a corresponding input and output. Many samples of input and output are collected.

[0043] In action 130, the image processor machine trains a network comprising an encoder and a decoder to predict radiomics values ​​in response to input medical imaging data. The model is trained to receive medical image data and segmentation (e.g., from a GTV and / or an OAR) at the encoder, generate a latent representation from the encoder, and generate output radiomics from the latent representation by the decoder. Training data provides samples of the input to the encoder and samples of the output from the decoder. For example, training data includes medical images, segments of medical images from a GTV and one or more OARs, and extracted values ​​for radiomics.

[0044] To train a machine learning model, a machine learning model layout is defined. This definition is achieved through a learning configuration or programming. The number of layers or units, the type of learning, and other characteristics of the model are controlled by the programmer or user. In other embodiments, one or more aspects (e.g., the number of nodes, the number of layers or units, or the type of learning) are defined and selected by the machine during learning. Training data (including numerous samples of input data and corresponding outputs) is used for training. The relationship between the inputs and outputs is machine learning.

[0045] Figure 2 Showing the use of action 130 ( Figure 1 An exemplary model trained by a machine. The model is a neural network formed by encoder 210 and decoder 220. More than one decoder 220 may be provided, such as in cases where the model is to generate multiple different outputs (e.g., one for each radiomics feature or one for multiple radiomics features, and another for one or more segments).

[0046] Encoder 210 has one or more layers, such as ten or more layers, wherein each layer has any number of nodes. Each node can use any activation function. Any type of layer can be used. In one embodiment, a series of downsampling and convolutional layers, such as an encoder from U-Net or an image-to-image network, is used. One or more max-pooling or other pooling layers may be included. Dropout layers may be used. DensNet, convolutional neural networks, ResNet, and / or other network architectures can be used for encoder 210.

[0047] Encoder 210 increases abstraction and decreases resolution. The final layer outputs a latent representation result 212. In response to input data, such as medical image data 200 and segmentation from segmenter 202, encoder 210 outputs values ​​for one or more features. The latent representation result 212 is the value of the feature relative to the level of abstraction of the input image data 200 and / or the segmentation. This latent representation result 212 is the patient's fingerprint.

[0048] Decoder 220 has inputs for receiving latent representation results 212. Values ​​of features output by encoder 210 are input to decoder 220. Decoder 220 outputs radiomics 222, such as values ​​for each of a plurality of different radiomics. Outputs are generated in response to the input of latent representation results 212.

[0049] Decoder 220 is a neural network, but other models can be used. This neural network can be fully connected or convolutional. Any number of layers and nodes within those layers can be used. Various layer arrangements can be used, such as sequential layers or densely connected layers. In one embodiment, some of the layers are convolutional layers. Pooling, upsampling, downsampling, dropout, or other layers can be used. One layer, such as the output layer, is a Softmax layer or another layer used to output values ​​such as quantity or label (e.g., class). Other architectures can be used. Different numbers of levels, layers, parameters, and / or convolutions within layers can be used. Various kernel sizes, combinations of layers, and / or layer types can be used.

[0050] For training, the output of decoder 220 (e.g., radiomics 222) is compared with the expected output (e.g., radiomics calculated by engine 204). In unsupervised training, this comparison is performed with outputs generated without user or expert creation. For example, a predetermined radiomics function 204 is applied to image data 200 and / or segmentation 202 to generate values ​​for radiomics 222, which are then compared with the values ​​for radiomics generated by decoder 220. Unsupervised training uses radiomics 222 as values ​​quantized from handcrafted, experimentally validated, and / or other radiomics functions 204. Assuming that handcrafted radiomics (e.g., size, shape, texture, etc.) have proven efficient for representing tumor phenotypes, decoder 220 is used to reconstruct radiomics features from image latent space 212.

[0051] The model has an architecture defined as outputting one or more radiomics 222 for each sample. Samples include different types of tumors, organs, imaging, settings, and / or other variations. In other embodiments, the model is trained for specific types of tumors, organs, and / or variations.

[0052] Image processor machine training model. Training learns the values ​​of learnable parameters (e.g., weights, connections, filter kernels, and / or other learnable parameters of a defined architecture). Deep learning or another machine learning approach can be used. Weights, connections, filter kernels, and / or other parameters are the features learned. For example, convolutional kernels are features trained. Using training data, the values ​​of the model's learnable parameters are tuned and tested to determine the optimal output estimate that results in a given input. Adam or another optimization is used for training.

[0053] During training, the loss is minimized. The loss function for the values ​​of radiomics 222 is the mean squared error between the network output and the handcrafted radiomics feature values. Other losses can be used. Using optimization, different values ​​of the learnable parameters are tested across different samples of the training data to minimize the loss or difference between the values ​​of radiomics 222 generated by the predetermined radiomics function 204 and the values ​​generated by the decoder 220. For large datasets with sufficient diversity in sample instances, the encoder-decoder architecture, along with the training algorithm, generally outperforms traditional machine learning methods in terms of data dimensionality reduction and representation learning.

[0054] In one embodiment, encoder 210 and decoder 220 are trained to predict values ​​for radiomics 222 and output one or more segments (e.g., GTV and / or OAR). Latent spatial representation learning can be performed based on the entire CT volume as well as on each GTV / OAR to obtain finer granularity. A third alternative can be based on a clinically relevant ROI (e.g., chest, abdomen) comprising multiple GTVs / OARs, wherein reference radiomics generated from radiomics engine 204 is cascaded from each GTV / OAR to guide the learning process.

[0055] When additional outputs (e.g., segmentation and / or other information from the region of interest) are provided, the loss of other types of outputs is minimized. Joint losses and / or combinations of losses can be used for optimization during machine training. The segmentation loss can be L1, L2, or other image losses. In one embodiment, available segmentation is used during machine training. If some samples in the training data include segmentation, the model can be defined as outputting segmentation in one branch. For example, the decoder is tasked with adding segmentation output. The decoder includes upsampling, pooling, convolution, and / or other layers to transform the latent representation 212 into a spatial representation of the tumor or organ segmentation at the same or different resolution as the medical image data 200. Segmentation may be included for some samples in the training data.

[0056] exist Figure 1 In actions 140 and 142, the image processor machine trains a classifier or regressor to output different treatment responses. In other embodiments, only one or three or more classifiers or regressors are trained to output different treatment responses.

[0057] refer to Figure 2After training encoder 210, encoder 210 is used and / or adapted for different tasks for users to build additional models 250 to evaluate treatment response or outcome 252. For example, the latent representation outcome 212 of OAR can be used to predict the radiotoxic effects in different body regions as outcome 252. This is achieved by applying a classifier or regressor 250 to the latent features 212 according to the distribution of the target variable. Another use case is to apply a classifier or regressor 250 to the latent space 212 to predict the probability of tumor recurrence or progression-free survival after treatment as outcome 252.

[0058] Instead of outputting tissue characteristics (such as radiomics features), the output should be the treatment response. Exemplary treatment responses include negative effects, toxicity, outcomes, progression, relapse, or control (e.g., effectiveness) for a given treatment or for different treatments. Any response can be predicted for one or more time periods, such as the expected relapse probability after 1 year and 2 years. Responses to different classifiers or regressors will differ in terms of the predicted response type, differences against treatment, and / or for different time periods.

[0059] Machine training for outputting treatment responses uses input from the latent representation 212 of encoder 210. At least initially, the same encoder 210 (i.e., the same architecture with the same learnable parameter values) is used to train different classifiers or regressors 250. For various samples, encoder 210 receives medical imaging data with or without the corresponding segment 202 as input. In response, encoder 210 generates a latent representation 212, which is used as input to the trained classifier or regressor 250.

[0060] Each classifier or regressor 250 is trained by machine independently of the others, such as using various input samples from patient records and corresponding baseline truth results 252 to predict from latent representation results 212. In an alternative embodiment, multi-task machine training is used. Two or more classifiers or regressors 250 are trained simultaneously by combining corresponding losses (such as joint losses (e.g., average losses) used to train multiple classifiers or regressors 250 together).

[0061] In one embodiment, the values ​​of the learnable parameters of encoder 210 are fixed. Encoder 210, previously trained with decoder 220, is used without altering the training of one or more classifiers or regressors 250. In an alternative embodiment, the trained encoder 210 is the initialization of encoder 210. The training of the classifier or regressor (i.e., minimization of loss) is used to relearn or fine-tune encoder 210 and learn the values ​​of the learnable parameters of regressor or classifier 250.

[0062] The regressor or classifier 250 is a neural network, support vector machine, or other architecture. In one embodiment, DensNet is used. In other embodiments, a fully connected or convolutional neural network is used. Different classifiers or regressors 250 can have the same or different architectures. The classifier or regressor 250 has any number, type, and / or arrangement of layers. The classifier or regressor 250 can be the same or different architecture as the decoder 220. Upsampling may not be provided because the output quantity or class label is to be output. In one embodiment, the classifier or regressor 250 is a convolutional neural network (CNN) with pooling layers. As the classifier 250, the output layer can have: a sigmoid function for predicting the probability of toxicity or negative effects; or a softmax function for outputting class members as results 252 (e.g., toxicity level and / or pneumonia grade) based on the input of the latent representation result 212. As the regressor, the output layer can be: a fully connected layer for outputting values ​​along a continuous scale, such as the risk level of tumor recurrence.

[0063] Any optimization from various options can be used to train a regressor or classifier on a machine. For example, Adam can be used. Any loss can be used, such as cross-entropy loss for classification problems or partial log-likelihood loss for regression problems.

[0064] In one embodiment, the same image processor and / or the same entity trains each of the classifiers or regressors 250. The same image processor and / or entity trains the encoder 210 and the classifiers or regressors 250. In other embodiments, the trained encoder 210 and / or the latent representations 212 derived from the trained encoder 210 are provided to different entity and / or image processors for training one or more of the classifiers or regressors 250. For example, the output feature values ​​(i.e., latent representations 212) and segments 202 are used by a hospital or medical group to train one or more classifiers or regressors 250. The hospital or medical group is different from the service provider, manufacturer, or developer of the encoder 210. The representations 212 may be output along with masks or segments 202 for users to build their own risk stratification, treatment assessment, and / or treatment prediction models, such as hospital- or practice-specific outcome prediction models.

[0065] Typically, the machine training for actions 130, 140, and 142 is provided for a given object such as an OAR or GTV. Machine training is performed separately for different types of OARs and / or GTVs. Automatically generated OARs and GTVs are used to build representations or models of the corresponding OARs and GTVs. In other embodiments, a model is trained to provide outcome predictions for multiple OARs and / or GTVs.

[0066] In action 150, encoder 210 and one or more classifiers or regressors 250 are stored in memory. The machine-trained model is stored. The learned weights, connections, kernels, and / or other values ​​of the learnable parameters of the encoder 210 and classifiers or regressors 250 are stored along with the architecture or as part of the model. Various versions of encoder 210 are stored in cases where encoder 210 is retrained or improved for different classifiers or regressors 250.

[0067] The stored model can be used or applied. Copies of the stored model can be used or applied by different servers, image processors, computers, MR scanners, or workstations. In application, a latent representation 212 is determined based on the patient segmentation 202 and the input image data 200. The encoder 210 generates the latent representation 212 for the patient, which is used as input by a classifier and / or regressor 250 to generate different outcome predictions for the patient. The values ​​of the learned parameters of the model, fixed after training, are used to apply to the patient or different patients (i.e., unseen data).

[0068] Figure 3 This is a flowchart of one embodiment of a method for predicting treatment responses using machines. The machine learning model includes a sub-part that includes an encoder trained at least in part on predictions based on radiomics, and one or more predictors of the results trained from latent representations generated by the encoder.

[0069] The treatment response and corresponding model are for a given type and / or level (amount) of treatment. Different models are provided for different types, progressions, and / or levels of treatment. The model is used to predict patient outcomes based on treatment. In an alternative embodiment, the model is trained to output outcomes for different treatments, thereby allowing comparison of estimated patient outcomes according to treatment type, progression, and / or level.

[0070] Figure 3 The method is by Figure 4 This process is performed by a medical scanner, image processor, or other system. For example, a CT scanner acquires image data. The image processor segments the data, generates potential representations, predicts treatment responses, and performs searches. The image processor can generate images. The display shows the image based on the image created by the image processor.

[0071] The method is executed in the order shown (e.g., top to bottom or numerically), but other orders can be used. For example, actions 330, 340, and 350 are executed in reverse or other orders. Additional, different, or fewer actions can be provided. For example, action 350 is not provided. In another example, action 340 is not performed. In yet another example, action 310 is not performed in cases such as generating latent representation results and predicting results without segmentation. As another example, actions for configuring the scan are performed based on user input or other information.

[0072] In action 300, a medical scanner (e.g., a CT scanner or imaging system) acquires medical image data representing the patient. The patient is scanned. The resulting reconstructed and / or images are medical image data. In other embodiments, the patient's medical image data is acquired from memory or via transmission over a computer network. Other types of data may be acquired.

[0073] In action 310, the image processor segments one or more OARs and / or GTVs from the medical image data. It identifies the contours, surfaces, boundaries, regions, and / or volumes corresponding to the OARs and / or GTVs.

[0074] Segmentation can be performed using any procedure or function (such as intensity thresholding with low-pass filtering). In one approach, a machine learning network, such as an encoder-decoder, outputs one or more segments in response to input medical image data.

[0075] In action 320, the image processor generates a latent representation from medical image data, segmented organs of risk, and / or segmented total tumor volume. Medical image data, with or without segmentation, is input to the encoder. In response to the input, the encoder generates a latent representation as a fingerprint of the patient, such as feature values ​​at the encoder's bottleneck or output. Features from other hidden layers of the encoder can be used. Values ​​of features from any layer of the machine learning segmentation model can be added or included along with feature values ​​from the encoder's output.

[0076] The determination is performed by or using a machine learning model. The machine learning model can be a neural network, such as a deep learning neural network. Fully connected neural networks, convolutional neural networks, fully convolutional networks, dense networks, and / or another type of neural network can be used. In one embodiment, an encoder using an image-to-image network (e.g., U-net) is used, such as an encoder that increases abstraction and reduces resolution to provide values ​​for bottleneck features. For example, using... Figure 2The encoder 210 receives image data 200 and segmentation 202 as inputs and outputs a latent representation result 212. Support vector machines, clustering-based machine learning, Bayesian regressors, or other machine learning regressors can be used.

[0077] Machine learning models operate based on how they are trained. Different training methods result in different values ​​of learnable parameters, thus leading to different operations. An encoder or another machine learning model, trained together with a decoder or other model, is used to generate latent representations of the results to output radiomics, segmentation of one or more regions (e.g., OAR and / or GTV), and / or information about regions of interest (e.g., information about regions separated or isolated from OAR or GTV).

[0078] In one embodiment, a latent representation is generated by an encoder, which is machine-trained in an unsupervised manner with a decoder to predict radiomic features with baseline truth from handcrafted and / or predetermined radiomics. The handcrafted or predetermined radiomics used to train the encoder and decoder are predetermined quantizations of tumor intensity, tumor shape, tumor texture, and / or wavelet transform.

[0079] In optional actions, the decoder can be used to generate radiomics values ​​from the latent representation results. Radiomics estimates can be used to assist in diagnosis, prognosis, and / or outcome prediction.

[0080] In actions 330 and 340, the image processor predicts one or more treatment responses. Different treatment responses are predicted using different machine learning classifiers or regressors. Differences can exist in regions (e.g., OAR versus GTV), treatment and / or response types (e.g., toxicity or negative effects and relapse time).

[0081] A prediction is one or more of the various outcomes for a given treatment and / or different treatments. For example, a prediction might be the negative impact and / or toxicity of one or more risk organs, and the outcome, progression, recurrence, and / or control of the tumor or total tumor volume. Using classifiers or regressors trained on different machines, and leveraging inputs from latent representations from the same encoder, multiple of the negative impact and / or toxicity of one or more risk organs and / or multiple of the outcome, progression, recurrence, and / or control of the tumor can be predicted. Outcomes can be predicted for different regions (e.g., OAR and / or GTV).

[0082] A machine-trained classifier or regressor performs predictions. A machine learning classifier or regressor can be a neural network, such as a deep learning neural network. A machine learning classifier or regressor is any machine learning classifier or network. For example, a neural network is used to regress the relationship between the input latent representation and the output. Fully connected neural networks, convolutional neural networks, fully convolutional networks, dense networks, and / or other neural networks can be used. Support vector machines, clustering-based machine learning, Bayesian, or other machine learning regressors or classifiers can be used.

[0083] Machine learning classifiers or regressors operate based on how they are trained. Different training methods result in different values ​​for the learnable parameters, thus leading to different operations. A machine-trained classifier or regressor outputs a prediction in response to input from the latent representation of the encoder. In one embodiment, Figure 2 The classifier or regressor 250 outputs a result in response to the input of the potential representation result from the encoder 210.

[0084] In an alternative embodiment, the machine learning classifier or regressor 250 receives input in addition to the latent representation 212 generated by the encoder 210. For example, the values ​​of hidden features from the machine learning network used for segmentation are concatenated with the values ​​of features from the latent representation 212. Prediction of the result can be made using a feature pyramid network, where values ​​from the segmentation features and the latent representation are used. Any features, such as those from the segmentation features, can be multi-scale, hierarchical features used as hidden features for input to the prediction result.

[0085] In Action 350, the image processor searches for similar cases. The search can be for any number of the most similar cases or patients. For example, it can identify the ten patients most similar to the current patient from a patient database (such as the same database used for training data).

[0086] The identified similar patients can demonstrate treatments that are effective and / or ineffective. For example, treatment identification can be used to select a machine learning model to then apply to the predictions of actions 330 and 340. As another example, the identification of similar patients can allow treating physicians to conduct further research to develop treatment plans.

[0087] The search uses latent representations. A distance function based on the latent space is used to measure the similarity between each pair of cases. The latent representations of a given patient are compared with the latent representations of other patients.

[0088] In one example, the Euclid distance or cosine similarity Where, xi and x j These are features in the latent space of two GTV / OAR ROIs. Similarity metrics can also be learned in a supervised manner based on the distribution of training samples. In another example, Mahalanobis distance is used. Here, M is a matrix estimated from the covariance matrix of the training data. Another approach is to learn better representations by jointly training a Variational Autoencoder (VAE) and a Generative Adversarial Network (GAN) discriminator to measure sample similarity. A smaller distance between two image ROIs indicates higher similarity in their image content and, moreover, a higher probability of treatment outcomes for very similar individuals.

[0089] In Action 360, the predicted treatment response is output as an image. The display shows the output from the application of the machine learning model. Values ​​or classes output by the model (e.g., a classifier or regressor 250) or information derived from them may be displayed. The image processor generates an image illustrating these values. This image may include representations of patient tissue, such as CT images showing the OAR and / or GTV. These values ​​may be annotations on the CT images. In alternative or additional embodiments, the output is to a patient's medical record or radiology report that can be displayed. These values ​​are conveyed to the user through the display of the image.

[0090] The display can be text (e.g., alphanumeric), graphical, or highlighted. It can display one or more predicted outcomes with or without segmentation, radiomics values, and / or patient representations. It can output the predicted treatment response for one or more organs at risk and / or total tumor volume. Links and / or data (e.g., results) from similar patients may also be displayed.

[0091] In another embodiment, a visualization of the latent representations 212 for patients, one or more similar cases, and / or groups is output. To visualize these latent representations, dimensionality reduction techniques (such as t-SNE or UMAP) can be used to project high-dimensional latent vectors into a 2-D or 3-D space. Tools can be provided for users to visualize these representations for various patient groups (e.g., how liver representations in a group change after RT or immunotherapy). For population analysis, the data-driven latent representation learning methods described above can be deployed to a CT data pool. The latent space can be filled with existing CT data by their latent features. Assuming any unseen CT volumes, this latent representation database can be used as a reference for patients or datasets. Therefore, historical medical records, treatment plans, and outcomes for relevant patients (approximately close in the latent space) can also be retrieved.

[0092] Figure 4An embodiment of a medical system for treatment prediction is shown. The medical system includes a display 400, a memory 420, and an image processor 410. The display 400, image processor 410, and memory 420 may be part of a medical scanner 430 (e.g., a CT scanner), a computer, a server, a workstation, or other systems for image processing of medical images from patient scans. A workstation or computer without a medical scanner 430 can be used as a medical system.

[0093] Additional, different, or fewer components may be provided. For example, a computer network may be included for remote image generation using locally captured image data or for local prediction from remotely captured image data. Machine learning model 415 may be applied as a standalone application on a workstation or local device or as a service deployed on a network (cloud) architecture. As another example, user input devices (e.g., keyboard, buttons, sliders, dial pads, trackballs, mice, or other devices) may be provided for a user to change the division or placement of one or more markers (e.g., landmarks). In yet another example, a medical scanner 430 may not be provided.

[0094] Image data (e.g., CT volume or slice data), segmentation, ROI information, GTV, OAR, machine learning model definition, values ​​of learning parameters, machine learning model 415, feature values ​​(e.g., latent representation results), radiomics values, prediction results, displayed images, and / or other information are stored in a non-transitory computer-readable storage medium (such as memory 420). Memory 420 is an external storage device, RAM, ROM, database, and / or local storage (e.g., a solid-state drive or hard disk drive). The same or different non-transitory computer-readable media can be used for instructions and other data. Memory 420 can be implemented using a database management system (DBMS) and resides on memory such as a hard disk, RAM, or removable media. Alternatively, memory 420 is internal to processor 410 (e.g., cache).

[0095] Instructions for implementing the training or application processes, methods, and / or techniques discussed herein are provided on a non-transitory computer-readable storage medium or memory (such as cache, buffer, RAM, removable media, hard disk drive, or other computer-readable storage media (e.g., memory 420)). Computer-readable storage media include various types of volatile and non-volatile storage media. Functions, actions, or tasks shown in the figures or described herein are performed in response to one or more sets of instructions stored in or on the computer-readable storage medium. Functions, actions, or tasks are independent of a particular type of instruction set, storage medium, processor, or processing strategy, and can be performed by software, hardware, integrated circuits, firmware, microcode, etc., operating individually or in combination.

[0096] In one embodiment, the instructions are stored on a removable media device for reading by a local or remote system. In other embodiments, the instructions are stored at a remote location for transmission over a computer network. In still other embodiments, the instructions are stored within a given computer, CPU, GPU, or system. Because some of the system components and method steps depicted in the figures can be implemented in software, the actual connections between system components (or process steps) can vary depending on how this embodiment is programmed.

[0097] Image processor 410 is a controller, control processor, general-purpose processor, microprocessor, tensor processor, digital signal processor, 3D data processor, graphics processing unit, application-specific integrated circuit, field-programmable gate array, artificial intelligence processor, digital circuit, analog circuit, combination thereof, or other device now known or hereafter developed for processing image data. Image processor 410 is a single device, multiple devices, or a network of devices. For more than one device, parallel or sequential partitioning of processing can be used. Different devices constituting image processor 410 can perform different functions. In one embodiment, image processor 410 is a control processor or other processor of medical scanner 430. Image processor 410 operates according to and is configured by stored instructions, hardware, and / or firmware to perform the various actions described herein.

[0098] Image processor 410 or another remote processor is configured to train a machine learning architecture. Based on a network architecture and training data provided by the user or from other sources, image processor 410 learns to correlate one or more input variables (e.g., image data and segmentation) with outputs (e.g., radiomics used to train the encoder and results used to train the classifier or regressor). The result of the training is a machine learning model 415, which includes one or more encoders and one or more classifiers or regressors.

[0099] Alternatively or additionally, the image processor 410 is configured to apply a machine learning model 415 (i.e., apply an encoder and one or more classifiers and / or regressors). The machine learning model 415 may include a machine learning network for segmenting OAR and / or GTV.

[0100] Image processor 410 is configured to segment tumors and / or organs adjacent to the tumor. Image processor 410 is configured to generate values ​​in the latent space by applying a machine learning encoder. In response to input image data (e.g., CT volume data) and the segmentation of the tumor and / or organ, the machine learning encoder generates values ​​representing the latent space results. The encoder has been previously trained, at least partially or completely, with the decoder in an unsupervised manner to predict predetermined radiomics. The values ​​in the latent space are bottleneck features of the encoder and decoder, such as the encoder's output values. In other embodiments, in addition to the latent space values ​​output by the encoder, other features, such as features from the encoder's hidden layers and / or segmentation, are used.

[0101] Image processor 410 is configured to predict at least one treatment outcome. Values ​​representing the outcome are input into a machine learning classifier or regressor. In response to the input values ​​in the latent space, the machine learning classifier or regressor outputs one or more predictions. For example, predicting negative effects or toxicity for one or more OARs. As another example, predicting the effectiveness, control, or relapse of GTV. Predictions are output either for a given treatment or for different treatment options.

[0102] Image processor 410 can be configured to predict multiple different treatment outcomes using different machine learning classifiers and / or regressors in response to input values ​​in the latent space. In one embodiment, an encoder used to train different classifiers or regressors individually or jointly is used to generate latent representations of the results input to multiple classifiers or regressors. The same encoder can be used in the application, or different encoders based on the same original encoder or improved from the same original encoder can be used.

[0103] Image processor 410 is configured to generate outputs such as values ​​and / or labels of the results, suggested treatments derived from the prediction results (e.g., based on weighted combinations), and / or a list of similar cases. Image processor 410 can be configured to generate visualizations of values ​​in the latent space.

[0104] Display 400 is: a CRT, LCD, projector, plasma, printer, tablet computer, smartphone, or other display device now known or hereafter developed, for displaying output, such as images with or without patient tissue, OAR, and / or GTV (e.g., images from medical scanner 430), with or without the predicted values ​​or labels of the images.

[0105] Medical scanner 430 is a diagnostic scanner. Scanner 430 operates according to one or more settings to scan a patient 432 lying on a bed or table 434. The settings control include sending, receiving, reconstructing, and image processing of the scan. A scanning protocol is followed to generate data representing the patient 432, such as CT image data representing a portion of the patient. Scanner 430 uses the settings to image the patient 432.

[0106] The various improvements described herein can be used together or individually. Although illustrative embodiments of the invention have been described herein with reference to the accompanying drawings, it is to be understood that the invention is not limited to these precise embodiments, and that various other changes and modifications can be made therein by those skilled in the art without departing from the scope or spirit of the invention.

Claims

1. A method for predicting treatment response using a machine, the method comprising: Acquire medical image data representing the patient; The total tumor volume and adjacent risk organs are segmented from the medical image data; Potential representations are generated from the medical image data, segmented organs at risk, and the total tumor volume. These potential representations are generated by an encoder that is machine-trained in an unsupervised manner along with a decoder to predict radiomic features with baseline truth from handcrafted radiomics. In response to the latent representation results input from the encoder to a first machine-trained classifier and a second machine-trained classifier, the first machine-trained classifier predicts a treatment response for the at-risk organ and the second machine-trained classifier predicts a treatment response for the total tumor volume, wherein predicting the treatment response for the at-risk organ includes predicting negative effects or toxicity to the at-risk organ; and Output the predicted treatment response for the at-risk organ and the total tumor volume.

2. The method according to claim 1, wherein, The acquisition includes acquiring computed tomography data as the medical image data, and wherein the handcrafted radiomics used to train the encoder and decoder includes predetermined quantizations of tumor intensity, tumor shape, tumor texture, and wavelet transform.

3. The method according to claim 1, wherein, The segmentation includes segmentation using a machine learning network, and the prediction includes making a prediction in response to inputs from the latent representation from the encoder and values ​​of hidden features from the machine learning network of the segmentation.

4. The method according to claim 3, wherein, The prediction includes predicting a feature pyramid network from the values ​​and the latent representations, wherein multi-scale, hierarchical features are used as the hidden features.

5. The method according to claim 1, wherein, The generation includes generation by the encoder, which includes a convolutional neural network.

6. The method according to claim 1, wherein, Generation includes the process of generating the encoder and decoder together by machine training to predict the radiomics features and segmentation.

7. The method according to claim 6, wherein, The generation includes information generated when the encoder and decoder are trained together by a machine to predict the radiomics features, the segmentation, and the region of interest.

8. The method according to claim 1, wherein, Predicting treatment response to the total tumor volume includes predicting tumor progression, recurrence, or control.

9. The method according to claim 8, wherein, The predictions include using different machines to train classifiers, taking into account inputs from potential representations from the same encoder, to predict multiple negative effects or toxicities on the at-risk organs, as well as multiple progressions, recurrences, or control of the tumor.

10. The method of claim 1, further comprising using the potential representation results to search for similar cases.

11. The method according to claim 10, wherein, The output also includes visualizations of the potential representations of the patient and similar cases.

12. A medical system for predicting treatment response, the medical system comprising: The memory is configured to store computed tomography (CT) data. An image processor is configured to: segment a tumor and an organ of risk adjacent to the tumor; generate values ​​of a latent space by a machine learning encoder in response to input computed tomography data and the segmentation of the tumor and the organ; predict a first treatment response for the organ of risk by a first machine learning classifier or regressor in response to input of the values ​​of the latent space; and predict a second treatment response for the total tumor volume by a second machine learning classifier or regressor in response to input of the values ​​of the latent space. as well as A display is configured to show the first treatment response and the second treatment response, wherein the first treatment response includes a negative effect or toxicity on the organ at risk.

13. The medical system according to claim 12, wherein, The encoder is trained together with the decoder in an unsupervised manner to predict predetermined radiomics, and the values ​​of the latent space are bottleneck features of the encoder and decoder.

14. The medical system according to claim 12, wherein, The second treatment response includes different treatment responses in outcome, progression, recurrence, and control, wherein the first machine learning classifier or regressor and the second machine learning classifier or regressor are trained separately using the same encoder.

15. The medical system according to claim 12, wherein, The image processor is also configured to generate visualizations of the values ​​in the latent space.

16. A method for machine training of a model for predicting treatment responses, the method comprising: Machine training involves a network of encoders and decoders to predict radiomics values ​​in response to input medical imaging data; The machine trains a first classifier or regressor to output a first treatment response in response to an input of feature values ​​output by the encoder, wherein the machine training of the first classifier or regressor includes machine training the first classifier or regressor for one of the negative effects or toxicity of risk organs adjacent to the tumor. The machine trains a second classifier or regressor to output a second treatment response in response to input feature values ​​output by the encoder, wherein training the second classifier or regressor includes training the second classifier or regressor to respond to treatment responses against the total tumor volume; and The encoder, the first classifier or regressor, and the second classifier or regressor are stored.

17. The method according to claim 16, wherein, Machine training of the network includes unsupervised training, wherein the radiomics includes values ​​quantized from hand-crafted functions.

18. The method according to claim 16, wherein, Machine training of the second classifier or regressor includes machine training of the second classifier or regressor in response to the outcome, progression, recurrence, or control of the tumor, wherein both the first classifier or regressor and the second classifier or regressor use the same encoder.

19. The method of claim 16, wherein, Machine training of the network includes machine training of the decoder to predict the values ​​of the radiomics and to segment tumors and / or organs of risk according to the treatment output, wherein machine training of the first classifier or regressor and the second classifier or regressor includes machine training using the segmentation and the medical imaging data as input to the encoder.

20. The method of claim 16, wherein, Machine training of the second classifier or regressor includes training based on the output feature values ​​and segments, to be performed by a party different from the party performing the training of the network.

Images