3D Convolutional Neural Networks for Organ Segmentation in Medical Images for Radiation Therapy Planning

Abstract

Methods, systems, and apparatus for segmenting medical images, including computer programs encoded on computer storage media. In one aspect, the method includes: receiving a medical image captured using a medical imaging modality and depicting a tissue region in a human body; and processing the medical image using a segmentation neural network to generate a segmentation output, wherein the segmentation neural network includes a sequence of multiple encoder blocks, wherein: each encoder block is a residual neural network block, which includes one or more two-dimensional convolutional neural network layers, one or more three-dimensional convolutional neural network layers, or both, and each encoder block is configured to process a corresponding encoder block input to generate a corresponding encoder block output, wherein the spatial resolution of the encoder block output is lower than the spatial resolution of the encoder block input.

Description

Background Art

[0001] This specification relates to processing medical images using machine learning models.

[0002] Medical images may be captured using a variety of imaging modalities, such as a computed tomography (CT) imaging modality.

[0003] A machine learning model receives input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate output based on the received input and the values ​​of the model's parameters.

[0004] Some machine learning models are deep models that use multiple layers to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to the received input to generate an output. Summary of the invention

[0005] This specification describes a system and method implemented as a computer program on one or more computers in one or more locations that perform medical image segmentation. The segmentation generated by the system can be provided for presentation to a user, for example to assist a physician or other medical professional.

[0006] According to a first aspect, a method is provided, the method comprising: receiving a medical image, the medical image being captured using a medical imaging modality and depicting a tissue region, ie, a 3D region, in a human body. The method comprises processing the medical image using a segmentation neural network to generate a segmentation output.

[0007] The segmentation output may include a plurality of segmentation channels, each segmentation channel corresponding to a respective organ from a predetermined set of organs, and each segmentation channel defines a segmentation of the respective organ corresponding to the segmentation channel in the medical image. For each of a plurality of voxels in the medical image, the segmentation of the respective organ in the medical image may include a respective score characterizing whether the voxel corresponds to an interior of the respective organ (e.g., whether the tissue belongs to the organ). The scores may include binary scores or be processed to provide binary scores to define a segmentation mask.

[0008] In some embodiments, the segmentation neural network includes a sequence of encoder blocks. Each encoder block can be a residual neural network block including one or more two-dimensional convolutional neural network layers, one or more three-dimensional convolutional neural network layers, or both. Such a residual neural network block can have a connection that skips one or more nonlinear layers of the block, for example, a connection between the input and output of the block, i.e., a direct connection. Each encoder block can be configured to process a corresponding encoder block input to generate a corresponding encoder block output. The resolution, i.e., the spatial resolution, of the encoder block output can be lower than the (spatial) resolution of the encoder block input. For each encoder block after the initial encoder block in the encoder block sequence, the encoder block input can include the previous encoder block output of the previous encoder block in the encoder block sequence. The segmentation neural network can also include a decoder subnetwork configured to process a decoder subnetwork input including an intermediate output of each encoder block to generate a segmentation output.

[0009] In some embodiments, the segmentation neural network includes a final layer configured to process a final layer input to generate a segmentation output. Processing the final layer input to generate a segmentation output may include: processing the final layer input according to a final layer weight set to generate a transformed final layer input, and processing the transformed final layer input using a sigmoid activation function to generate a segmentation output.

[0010] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0011] The segmentation neural networks described in this specification achieve performance levels comparable to human experts (e.g., radiation therapists) in medical image segmentation. Therefore, the segmentation neural networks have the potential to be deployed in clinical settings, which can reduce healthcare costs and improve efficiency. For example, the segmentation neural network can be used as part of an automated radiation therapy planning process to segment "organs at risk," that is, normal anatomical structures that are at risk of inadvertent radiation exposure during radiation therapy. In this example, the segmentation neural network can partially or fully automate a time-consuming task previously performed by an expert.

[0012] The segmentation neural network can generate a segmentation as the output of the final neural network layer with an activation function (e.g., a Sigmoid activation function), which is applied to each component of the segmentation output separately. By applying the activation function to each component of the segmentation output separately, the segmentation neural network can characterize a single voxel of the medical image as having a high probability of belonging to multiple organs. For example, the segmentation neural network can generate a segmentation output that characterizes a single voxel as having a high probability of belonging to both "lens" and "eye". In contrast, the segmentation output generated by some conventional segmentation neural networks cannot characterize a single voxel as belonging to multiple overlapping organs with a high probability. For example, some conventional neural networks use a soft-max activation function in the final layer, where the soft-max function is jointly applied to each group of components of the segmentation output (i.e., where a group of components characterizes the probability that the voxels of the medical image belong to different organs). In this example, the sum of the soft-max outputs of the components of each group is required to be 1, resulting in the inability of conventional neural networks to characterize a single voxel as belonging to multiple organs with a high probability. Therefore, unlike some conventional segmentation neural networks, the segmentation neural network described in this specification can be used to generate accurate segmentation outputs for multiple overlapping organs.

[0013] The medical imaging modality or system may include a computed tomography (CT) medical imaging modality or system, and / or another imaging modality such as an MRI (magnetic resonance imaging) or PET (positron emission tomography) scan.

[0014] In some embodiments, the decoder subnetwork includes a sequence of multiple decoder blocks. Each decoder block may include a residual neural network block, which includes one or more two-dimensional convolutional neural network layers. Each decoder block may be configured to process a corresponding decoder block input to generate a corresponding decoder block output. The (spatial) resolution of the decoder block output is greater than the (spatial) resolution of the decoder block input. For each decoder block after an initial decoder block in the decoder block sequence, the decoder block input may include: (i) an intermediate output of the corresponding encoder block, and (ii) a previous decoder block output of the previous decoder. The three-dimensional convolutional neural network layer in the encoder block may include a padded xy-convolution and an unpadded z-convolution. The segmentation neural network may include a link block, such as a residual neural network module including a fully connected layer. The link block may be configured to process a link block input to generate a link block output, the link block input including the output of the final encoder block in the encoder block sequence, and the link block output may be provided as an input to the decoder subnetwork.

[0015] The segmentation system described in the present specification can characterize the accuracy of the predicted organ segmentation by calculating a consistency measure between (i) the surface of the predicted organ segmentation and (ii) the surface of the true organ segmentation, i.e., not the entire volume. The consistency measure between the surfaces can characterize a portion of the surface of the predicted segmentation that will need to be manually corrected by an expert, for example, to enable the segmentation to be used in radiation therapy planning. The segmentation system can train the segmentation neural network until the performance measured based on the consistency of the segmentation surfaces reaches an acceptable level. Thereafter, the trained segmentation neural network can be incorporated into a clinical workflow, where a minimal amount of expert time may be required to correct the predicted segmentation generated by the segmentation neural network.

[0016] Therefore, in some embodiments, the method further comprises calculating a surface metric, such as a surface patch metric or a surface Dice-Sorensen coefficient, as a segmentation performance indicator. The surface patch metric may be calculated between (i) a segmentation of a corresponding organ in a medical image defined by a segmentation channel from a segmentation output and (ii) a reference (a human expert segmentation of the corresponding organ in a medical image). Calculating the surface patch metric may comprise determining the number of voxels in a first intersection between (i) a surface of the segmentation of the corresponding organ; and (ii) a tolerance region around a surface of the reference segmentation of the corresponding organ; or determining the number of voxels in a second intersection between (i) a surface of the reference segmentation of the corresponding organ, and (ii) a tolerance region around a surface of the segmentation of the corresponding organ; or both. Determining both has the advantage that the false positive and false negative portions of the predicted segmentation can be penalized. Defining a tolerance region is clinically beneficial because it is an absolute deviation from the segmentation, for example in millimeters, which is important for the safety of radiotherapy.

[0017] In an embodiment of calculating the surface patch metric, the surface patch metric can also be determined as the ratio of (i) the sum of the number of voxels in the first intersection and the number of voxels in the second intersection to (ii) the sum of the number of voxels in the segmented surface of the corresponding organ and the number of voxels in the reference segmented surface of the corresponding organ.

[0018] In principle, such a surface metric can be calculated independently of the segmentation technique used. The application is not limited to the above method, and the surface metric can be applied, for example, to any segmentation of an organ in a medical image, which segmentation defines for each of a plurality of voxels in the medical image a value characterizing whether the voxel corresponds to the interior of the organ.

[0019] The segmentation neural networks described in this specification can be trained using a segmentation loss that includes contributions from only those voxels with the highest error values ​​(e.g., voxels where the segmentation neural network made a confident but incorrect prediction). By using a segmentation loss that includes contributions from only those voxels with the highest error values, the segmentation neural network can be trained to focus on the most difficult and ambiguous parts of the medical images. By selecting contributions to the segmentation loss in this way, training can also take into account class imbalance, which is often seen in scanning organs of different sizes (e.g., scanning the head, cochlea 65mm 3 , and the brain 1400000mm 3 Appears when imaging is performed.

[0020] In this way, the segmentation neural network can be trained to reach an acceptable performance level on fewer training iterations, and training the neural network can therefore consume fewer computing resources, such as memory and computing power, compared to conventional training techniques.

[0021] Therefore, a method for training a segmentation neural network performed by one or more data processing devices is also provided, the method being configured to process a medical image to generate a segmentation output, the medical image being captured using a medical imaging modality and depicting a tissue region in a human body. The segmentation output may include a plurality of segmentation channels, wherein each segmentation channel corresponds to a respective organ from a predetermined set of organs, and each segmentation channel defines a segmentation of the respective organ corresponding to the segmentation channel in the medical image. For each of a plurality of voxels in the medical image, the segmentation of the respective organ in the medical image may include a respective score characterizing whether the voxel corresponds to an interior of the respective organ. The method may include: receiving a training medical image; and processing the training medical image using a segmentation neural network to generate a training segmentation output; and determining a segmentation loss of the training medical image.

[0022] For each segmentation channel of the training segmentation output, determining the segmentation loss may include: determining a set of error values ​​for the segmentation channel, wherein each error value in the set of error values ​​for the segmentation channel corresponds to a corresponding voxel in the medical image, and is based on the error between (i) a score from the segmentation channel characterizing whether the voxel corresponds to the interior of an organ corresponding to the segmentation channel and (ii) a target score defining whether the voxel corresponds to the interior of the organ corresponding to the segmentation channel; identifying multiple highest error values ​​in the set of error values ​​for the segmentation channel; and determining the segmentation loss based on the multiple highest error values ​​identified for each segmentation channel of the training segmentation output.

[0023] The method may also include adjusting current values ​​of the segmentation neural network weights based on the segmentation loss of the training medical image.

[0024] The error may include a cross entropy loss, such as a binary cross entropy loss if the voxel scores are binarized so that the segmentation defines a mask. The method may include identifying a plurality of highest error values ​​(e.g., highest cross entropy) in a set of error values ​​for segmenting the channel as an appropriate subset of the set of error values ​​for segmenting the channel. Determining the segmentation loss may include summing the identified plurality of highest error values. Training may be performed incrementally on an image subvolume, such as determined by a central image slice and a set of image slices on either side.

[0025] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 An example segmentation system is shown.

[0027] Figure 2A-2D Illustration of an example architecture for a segmentation neural network.

[0028] Figure 3A-3B An example of calculating surface patch metrics and organ-specific tolerance thresholds is illustrated.

[0029] Figure 4 An example of the quantitative performance of a segmentation system compared to a radiation therapist is illustrated.

[0030] Figure 5 is a flowchart of an example process for training and using a segmentation neural network.

[0031] In the various drawings, like reference numerals and designations indicate like elements. DETAILED DESCRIPTION

[0032] The present specification describes a segmentation system that can process a medical image (e.g., a computed tomography (CT) image) that depicts a tissue region (e.g., a head and neck region) within a patient to generate a segmentation of one or more organs in the tissue region. More specifically, the segmentation system can generate a segmentation output that specifies, for each voxel of the medical image, whether the voxel is included in each predetermined set of organs. As used throughout this specification, "organ" can refer to any appropriate anatomical structure in the human body (e.g., the brain, the eye, or a tumor), and some organs may be partially or fully contained within other organs, e.g., a lens may be completely contained within an eye.

[0033] Organ segmentations generated by a segmentation system can be used in any of a variety of applications, for example, for planning radiation therapy (radiotherapy) treatments for cancer patients. Radiation therapy treatment planning may require accurate segmentation of a target area to be irradiated (e.g., a tumor and areas at high risk of tumor spread) as well as other organs near the target area to determine a treatment plan that minimizes the radiation dose received by the other organs. Manual organ segmentation can be time consuming (e.g., an expert may spend four hours or more on a single case, and manual segmentations generated by the same expert are not consistent, for example). In contrast, the segmentation system described in the present specification can generate automatic organ segmentations in seconds or minutes, and in some cases, achieve segmentation accuracy comparable to expert-generated manual segmentations. Thus, the segmentation system can reduce delays in radiation therapy treatments by partially automating various aspects of radiation therapy treatment planning.

[0034] These features and others are described in more detail below.

[0035] Figure 1 An exemplary partitioning system 100 is shown. Partitioning system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.

[0036] The segmentation system 100 is configured to process a medical image 102 depicting a tissue region within a patient 104 to generate a segmentation output 106 defining a segmentation of one or more organs in the medical image 102. An example set of organs in the head and neck region that may be segmented by the segmentation system 100 will be referred to as Figure 4 Describe in more detail.

[0037] The medical image 102 may be generated by the medical imaging system 108 using any suitable medical imaging modality, such as computed tomography (CT), magnetic resonance (MR), or ultrasound (US). In some cases, the segmentation system 100 may be configured to process multiple medical images of a patient, for example, using different medical imaging modalities (e.g., a CT image and an MR image) or using the same medical imaging modality, such as multiple MR images captured in rapid succession after a contrast agent is administered to the patient 104. In these cases, the multiple images may be registered before being processed by the segmentation system 100. Registering two images refers to applying one or more transformations (e.g., translation, rotation, shear, or elastic deformation operations) to one of the images to align it with the other image so that matching voxels in the two images correspond to the same portion of tissue in the patient 104.

[0038] The medical image 102 may be represented by a three-dimensional (3-D) array of voxels, where each voxel is associated with an intensity value that characterizes a property of a corresponding portion of tissue in a patient. For example, the intensity values ​​of a CT image characterize the physical density of tissue in a patient, e.g., in Hounsfield units. For convenience, the voxels of the medical image 102 may be referred to in this specification as being indexed by x, y, and z coordinates, and a set of voxels in the medical image 102 having the same z-value coordinate may be referred to as a "slice" of the medical image 102.

[0039] The segmentation system 100 processes the medical image 102 using a segmentation neural network 200, i.e., generates a segmentation output 106 according to the values ​​of a set of model parameters 110 of the segmentation neural network 200. The segmentation neural network 200 is a convolutional neural network, such as a 3-D convolutional neural network having one or more 3-D convolutional neural network layers. A 3-D convolutional layer refers to a layer that uses, for example, a 3-D convolutional filter, or performs a convolution operation by using a 2-D convolutional filter in the xy dimension and then using a 1-D convolutional filter in the z dimension. Figures 2A-2D An example architecture for a segmentation neural network is described in more detail.

[0040] The segmentation neural network 200 may be configured to receive an input of a predetermined size, such as a predetermined number of slices of the medical image 102, such as 21 slices. The segmentation output 106 may define only a proper subset of the input, such as a segmentation of a proper subset of the input of only the central slices of the 21 slices of the medical image 102 provided as input, for example, due to an unpadded convolution operation performed in the z dimension of the input. A convolution operation may be referred to as padded if a default value (e.g., zero) is added to the input of the operation, otherwise it may be referred to as unpadded. Performing an unpadded convolution operation may result in the output of the convolution operation having a resolution lower than the resolution of the input of the convolution operation. To generate an overall segmentation output 106 of the medical image 102, the segmentation system 100 may process corresponding subsets of the medical image 102 using the segmentation neural network 200 and then aggregate the corresponding segmentation outputs 106.

[0041] The segmentation output 106 may be represented by an ordered set of segmentation "channels," where each segmentation channel defines a segmentation of a corresponding organ in the medical image 102. Each segmentation channel may be represented by an ordered set of values ​​(e.g., a 3D array of values), where each value specifies a score characterizing whether a corresponding voxel in the medical image 102 is inside the organ corresponding to the channel. (Throughout the specification, "inside" of an organ should be understood to refer to the entire organ, e.g., including the surface of the organ.) The score characterizing whether a voxel is inside the organ may characterize the probability that the voxel is inside the organ and may represent a probability value probability, e.g., between 0 and 1. Optionally, the segmentation system 100 may process the segmentation output 106, e.g., by rounding the probability value specified by each channel to a corresponding binary value (e.g., 0 or 1), which indicates a hard prediction of whether the corresponding voxel in the medical image 102 is inside the corresponding organ.

[0042] The segmentation system 100 can train the model parameters 110 of the segmentation neural network 200 on a set of training data 114 using a training engine 112. The training data 114 can consist of a set of training examples, where each training example specifies: (i) a training medical image, and (ii) a target segmentation that should be generated by the segmentation neural network 200 by processing the training medical image. The target segmentation can be represented by a set of target segmentation channels, i.e., where each target segmentation channel defines a segmentation of a corresponding organ, and can be represented by a 3-D array of numerical values ​​specifying a target score for a voxel of the medical image. The target score can be a binary value, e.g., a value of 1 indicates that the voxel is included inside an organ, while a value of 0 indicates otherwise. The target segmentation can be manually generated by an expert, such as a radiation therapist or an oncologist.

[0043] The training engine 112 may train the model parameters 110 of the segmentation neural network 200 using an iterative training process (e.g., stochastic gradient descent). At each of a plurality of training iterations, the training engine 112 may select (e.g., sample) a "batch" (set) of training examples from the training data, and use the segmentation neural network 200 to process the training medical images specified by the training examples to generate corresponding segmentation outputs 106. The training engine 112 may determine a segmentation loss for each training medical image in the batch, the segmentation loss characterizing the error between (i) the segmentation output generated according to the current values ​​of the model parameters 110 for the training medical image and (ii) the corresponding target segmentation. The training engine 112 may then determine the gradient of the segmentation loss (i.e., with respect to the model parameters 110), and use the gradient to adjust the current values ​​of the model parameters 110 using any suitable gradient descent optimization algorithm (e.g., RMSprop or Adam).

[0044] The training engine 112 can determine the segmentation loss for the training medical image in any of a variety of ways. For example, the training engine 112 can determine the segmentation loss based on the most "difficult" voxel of each segmentation channel, i.e., the one with the highest error between the training segmentation channel and the target segmentation channel. More specifically, for each training segmentation channel, the training engine 112 can determine the corresponding error value between each voxel of the training segmentation channel and the corresponding voxel of the target segmentation channel. The training engine 112 can determine the error value between the corresponding voxels of the training segmentation channel and the target segmentation channel in any suitable manner, for example as a cross-entropy error. The training engine 112 can then determine the total segmentation loss based on the highest error value from each training segmentation channel (e.g., corresponding to the top 5%, top 10%, or top 15% of the error values ​​from each training segmentation channel) (e.g., the sum). Determining the segmentation loss in this manner can speed up training by enabling the segmentation neural network to focus on the most difficult (e.g., ambiguous) voxels in the training medical image.

[0045] The training engine 112 can use data augmentation techniques to increase the number of training examples used in training the segmentation neural network. In particular, the training engine 112 can generate new training examples from existing training examples by applying transformation operations to training medical images specified by the existing training examples (and potentially to target segmentations specified by the existing training examples). For example, these transformations can include translation, rotation, scaling, shearing, mirroring, elastic deformation, or a combination thereof, which are applied to both the training medical images and the target segmentations, or adding pixel-by-pixel noise to the training medical images (without modifying the target segmentations). For example, training the segmentation neural network 200 on the enhanced set of training data 114 can improve the training of the segmentation neural network 200 by reducing the probability of overfitting and increasing the robustness of the segmentation neural network 200 to changes in its inputs.

[0046] In response to determining that the training termination criteria are met, the training engine 112 can stop training the model parameters 110 of the segmentation neural network 200. For example, the training engine 112 can stop training the model parameters 110 in response to determining that the segmentation accuracy of the segmentation neural network 200 evaluated on the validation data set has met a predetermined threshold. The validation data set can include training examples that "hold out" from the training of the segmentation neural network 200, i.e., such that the model parameters 110 are not trained on training examples from the validation data.

[0047] The training engine 112 can evaluate the segmentation accuracy of the segmentation neural network 200 on the validation data in any of a variety of ways. For example, for each training medical image in the validation data, the training engine 112 can compare the training segmentation (i.e., generated by the segmentation neural network) of each organ of the training medical image with the corresponding target organ segmentation. The training engine 112 can compare two segmentations of an organ by calculating a similarity metric between the two segmentations, which similarity metric characterizes a measure of consistency between the surfaces of the two segmentations rather than the entire volume. The measure of consistency between the surface of the training segmentation of the organ and the surface of the target segmentation can characterize the fraction of the surface of the training segmentation that will need to be manually corrected by an expert, for example, to enable the segmentation to be used for radiation therapy planning.

[0048] The training engine 112 can calculate a measure of consistency between two segmented surfaces of an organ, for example as a "surface patch measure" that characterizes the fraction of the area of ​​the two surfaces that is within another tolerance threshold. The tolerance threshold (which can be expressed as a numerical value, such as millimeters) can characterize the maximum acceptable deviation between the surface of the training segmentation and the surface of the corresponding target segmentation. Figure 3A An illustration of the calculation of a surface patch metric is provided. The tolerance threshold may be an organ-specific value calculated as a percentile (e.g., the 95th percentile, or any other appropriate percentile) of a set of distances between manually segmented surfaces of the organ, such as generated by a human expert. The "distance" between two segmented surfaces may refer to the minimum distance between a point on one surface to any point on the other surface. Figure 3B Provides a graphical representation of the set of distances between two segmented surfaces.

[0049] The training engine 112 may calculate two surfaces S with a tolerance threshold τ>0 1 and S 2 Small surface patch measurements between, for example:

[0050]

[0051] in Indicates the tolerance threshold τ on the surface S 2 The surrounding "tolerance zone" The surface S 1 The area of Indicates the tolerance threshold τ on the surface S 1 The surrounding "tolerance zone" The surface S 2 The area of ​​|S 1 |Indication surface S 1 The total area and |S Z |Indication surface S 2The total area of ​​​​the surface v. The tolerance region around the surface v with a tolerance threshold τ refers to the set of points within a distance τ of the surface, that is:

[0052]

[0053] The training engine 112 may calculate the surface area referenced in equation (1), for example, by counting the number of voxels on the corresponding surface. For example, to calculate The training engine 112 can be used to train the 2 The surface S within a distance τ 1 Count the number of voxels on the surface. For example, if a voxel is adjacent to at least one voxel outside the organ and at least one voxel inside the organ, then the voxel is said to be on the surface of the organ segmentation. In some embodiments, instead of calculating the surface area referenced in equation (1) by counting voxels on the surface, the training engine 112 can use a marching cubes triangulation algorithm to represent each surface as a polygonal mesh, such as a triangle mesh, and then use the mesh to calculate the surface area. Using a polygonal mesh to calculate the surface area can reduce the probability of systematic errors that may arise from calculating the surface area by voxel counting, for example, when the voxel spacing in each spatial dimension is not equal.

[0054] This approach can also solve another problem, where representing a surface using "foreground" voxels underestimates the surface area, while using "background" voxels overestimates the surface area. Therefore, the points defining the polygonal mesh can be placed on a grid that is offset by half the grid spacing in each axis. In 3D, each point on this grid has eight neighboring voxels, and a representation is defined using the marching cubes algorithm2 8 The polygons required for each of the possible configurations are calculated to define the segmented surface and the reference segmented surface. For each surface area element, the distance to the other surface can be calculated, and then the surface area within the tolerance region can be calculated by adding the areas of the surface elements within a tolerance threshold τ.

[0055] The segmentation system 100 can present the segmentation output 106 generated by the segmentation neural network 200 to a user of the segmentation system 100, for example, by generating a visualization that overlays the organ segmentation on the medical image 102. The segmentation system 100 can enable the user to make corrections to the segmentation output 106, for example, by providing input through a user interface. The user can use the segmentation output as part of planning radiation therapy for the patient 104.

[0056] Figure 2AAn example architecture of a segmentation neural network 200 is shown. In this example, the segmentation neural network 200 has an encoder subnetwork 202, a linking block 204, and a decoder subnetwork 206. As used throughout this specification. A "block" refers to a group of one or more neural network layers. In general, the inputs and outputs of a block can be represented as arrays of values ​​indexed along multiple "spatial" dimensions (e.g., xy dimensions or zxy dimensions) and a "channel" dimension. The "resolution" of a block input / output along a dimension refers to the number of index values ​​along that dimension. If the intermediate output of the first layer of the block (i.e., the input of the block) is added to the intermediate output of the next layer of the block (i.e., the output of the block), the block may be referred to as a "residual" block.

[0057] The encoder subnetwork 202 includes a sequence of multiple encoder blocks, e.g., encoder blocks 202-AG, each encoder block including a 2-D convolutional layer, a 3-D convolutional layer, or both. Each encoder block is configured to process an encoder block input to generate an encoder block output having a lower spatial resolution than the encoder block input, i.e., such that the resolution of the encoder block output is less than the resolution of the encoder block input along at least one spatial dimension. The first encoder block 202-A may be configured to process the medical image 102, while each subsequent encoder block 202-BG may be configured to process the encoder block output of a previous encoder block in the encoder subnetwork 202. Referring to Figure 2B An example encoder block architecture is illustrated.

[0058] The linking block 204 is configured to process a linking block input, which includes the output of the final encoder block 202-G in the encoder sub-network 202, to generate a linking block output, which is provided as an input to the decoder sub-network 206. Figure 2C An example link block architecture is illustrated.

[0059] The decoder subnetwork 206 includes a sequence of multiple decoder blocks, e.g., decoder blocks 206-AG, each decoder block including a 2-D convolutional layer, a 3-D convolutional layer, or both. Each decoder block is configured to process a decoder block input to generate a decoder block output having a higher spatial resolution than the decoder block input, i.e., such that the resolution of the decoder block output is higher than the resolution of the decoder block input along at least one spatial dimension. The first decoder block 206-G may be configured to process an input comprising: (i) a link block output, and (ii) an intermediate output of a corresponding encoder block 202-G. Each subsequent decoder block 206-AF may be configured to process: (i) an intermediate output of a corresponding encoder block, and (ii) a decoder block output of a previous decoder block in the decoder subnetwork. For example, decoder block 206-F may be configured to process: (i) an intermediate output of a corresponding encoder block 202-F, and (ii) an output of decoder block 206-G. In this specification, an "intermediate output" of an encoder block refers to an output generated by any layer of the encoder block other than an input layer of the encoder block. Reference Figure 2D The figure shows an example decoder block architecture.

[0060] The segmentation neural network 200 is configured to process the output of the final decoder block 206-A using an output layer 208 ("final layer") to generate a segmentation output 106. The output layer 208 optionally processes the output of the final decoder block 206-A according to a set of output layer weights (e.g., defining one or more convolution operations), and then applies a non-linear "activation" function (e.g., a sigmoid activation function) to each component of the segmentation output 106 separately. By applying the activation function to each component of the segmentation output 106 separately, the segmentation neural network can characterize a single voxel of the medical image 102 as likely to belong to multiple organs. That is, the segmentation neural network can generate a segmentation output that assigns a "high" probability (e.g., a probability close to 1) to matching components of different segmentation channels, i.e., components corresponding to the same voxel in the medical image 102. For example, the segmentation neural network can generate a segmentation output that characterizes a single voxel as being highly likely to belong to both "spinal cord" and "vertebral canal".

[0061] Applying activations jointly to the component groups of the segmentation output 106, i.e., rather than applying them separately to each component of the segmentation output 106, may result in the segmentation output failing to characterize a single voxel as belonging to multiple overlapping organs. For example, a soft-max activation function may be jointly applied to the component groups of the segmentation output, where all components in a group correspond to the same voxel in the medical image, and the soft-max outputs of the components of each group sum to 1. Requiring that the probabilities of a given voxel belonging to each organ sum to 1 results in the segmentation output 106 failing to characterize a single voxel as being highly likely to belong to multiple organs.

[0062] Figure 2B An example architecture of an encoder block 210 of an encoder subnetwork of a segmentation neural network is illustrated. The encoder block 210 is configured to process a block input 212 to generate a block output 214, i.e., having a lower spatial resolution than the block input 212, and an intermediate output 216, as described above. In the illustration, "ReLU" refers to a rectified linear unit activation function, "1-D Conv" refers to a 1-D convolution operation, "2-D Conv" refers to a 2-D convolution operation, "3-D Conv" refers to a 3-D convolution operation, "Pool" refers to a pooling operation, "Crop" refers to cropping the block input at the center of the z dimension, and "Add" refers to an addition operation. Various illustrated features of the example architecture of the encoder block 210 are optional; for example, the encoder block 210 may exclude a 2-D, 3-D convolution operation, or a pooling operation (or a combination thereof). In one example, a 2-D convolution operation can be implemented by repeatedly applying a 1×3×3-dimensional (in zxy coordinates) convolution kernel followed by ReLU, a 3-D convolution operation can be implemented by repeatedly applying a 1×3×3-dimensional convolution kernel followed by a 3×1×1-dimensional convolution kernel and ReLU, and a pooling operation can be implemented as a 1×2×2-dimensional average pooling operation.

[0063] Figure 2C An example architecture of a link block 204 of a segmentation neural network is illustrated. The link block 204 is configured to process a block input 218 (generated by the final encoder block) to generate a block output 220 (which is provided to the first decoder block). In the illustration, "fully connected" refers to the operations performed by a fully connected neural network layer, and refers to Figure 2B Define "2-D Conv" and "ReLU". In one example, a 2-D convolution operation can be implemented by applying a 1×8×8 dimensional convolution kernel.

[0064] Figure 2D An example architecture of a decoder block 222 of a decoder subnetwork of a segmentation neural network is illustrated. The decoder block 222 is configured to process a decoder block input to generate a decoder block output 224 having a higher spatial resolution than the block input, as described above. Typically, the decoder block input includes: (i) an intermediate output 226 of the corresponding encoder block, and (ii) an output of a previous decoder block 228 or an output of a chained block. For convenience, Figure 2D The decoder block 222 is illustrated receiving the previous decoder block output 228 instead of the chained block output. In this illustration, "coarsening" refers to the operation of operating on the input to increase the spatial resolution of the input, "cropping and stacking" refers to cropping the encoder block intermediate output 226 in the z dimension and concatenating the result with the coarsened previous decoder block output 228, and referring to Figure 2B"ReLU", "1-D Conv", "2-D Conv", and "Add" are defined. In one example, a 2-D convolution operation can be implemented by applying a 1×3×3 dimensional convolution kernel.

[0065] Figure 3A An example of the calculation of a surface patch metric is shown in FIG. Figure 3A An example of calculating a surface patch metric in 2-D is shown, but more generally, a surface patch metric can be calculated in 3-D to characterize a measure of consistency between 3-D organ segmentations. In this example, the solid line 302 represents the use of a segmentation neural network (reference Figure 1 304 represents the surface of the target organ segmentation generated by, for example, a radiotherapist. The arrow 306 represents the tolerance threshold, i.e., the maximum acceptable deviation margin. The dark line segments 302-A, 302-B, 304-A, and 304-B ​​represent the portions of the surface where the distance between the surfaces exceeds the tolerance threshold. The surface patch metric characterizes the area of ​​the surfaces that are within the tolerance threshold of each other compared to the total surface area, as shown in FIG. Figure 1 As described.

[0066] Figure 3B The determination of tolerance thresholds for specific organs is illustrated. For convenience, Figure 3B (and Figure 3A ) illustrates the segmentation in 2-D, but more generally, the segmentation of the organ may be in 3-D. Lines 308 and 310 represent two segmented surfaces of the organ manually generated by different experts, and the line segment between lines 308 and 310 represents the distance between the two surfaces. The tolerance for a specific organ may be defined as, for example, the 95th percentile of the distance between the manually segmented surfaces of the organ.

[0067] Figure 4 An example of the quantitative performance of a segmentation system compared to a radiotherapist is illustrated. For each of a plurality of organs (e.g., cerebrum, brainstem, cochlea, etc.), a first bar graph shows the mean of the surface patch measure of the organ segmentation generated by the segmentation system compared to the gold standard segmentation, and a second bar graph shows the mean of the surface patch measure of the organ segmentation generated by the radiotherapist compared to the gold standard segmentation. In 19 out of 21 organs, there was no substantial difference between the performance of the segmentation system and the performance of the radiotherapist, i.e., where a substantial difference was defined as 5% or greater.

[0068] Figure 5 is a flow chart of an example process 500 for training and using a segmentation neural network. For convenience, process 500 is described as being performed by a system of one or more computers located in one or more locations. For example, a segmentation system, such as a computer programmable according to the specification, may be configured to perform the segmentation process. Figure 1 The segmentation system 100 can perform process 500.

[0069] The system receives a medical image that is captured using a medical imaging modality and depicts a tissue region in a human body (502). The medical imaging modality may be, for example, a computed tomography (CT) modality. The tissue region in the human body may be, for example, a head and neck region.

[0070] The system processes the medical image using a segmentation neural network to generate a segmentation output (504). The segmentation output may include a set of segmentation channels, wherein each segmentation channel corresponds to a corresponding organ from a predetermined set of organs and defines a segmentation of the organ in the medical image. The segmentation of the organ in the medical image specifies, for each of a plurality of voxels in the medical image, a score characterizing whether the voxel corresponds to an interior of the organ. The predetermined set of organs may include one or more organs at risk in a patient receiving radiation therapy treatment. The segmentation output may be used for radiation therapy planning.

[0071] The segmentation neural network may include an encoder subnetwork having a plurality of encoder block sequences, wherein each encoder block is a residual neural network block that includes one or more two-dimensional convolutional neural network layers, one or more three-dimensional convolutional neural network layers, or both. Each encoder block may be configured to process an encoder block input to generate an encoder block output having a lower spatial resolution than the encoder block input. Each encoder block after the first encoder block may process an input comprising an output of a previous encoder block. The three-dimensional convolutional neural network layer in the encoder block may include padded xy convolutions and unpadded z convolutions.

[0072] The segmentation neural network may include a linking block including a fully connected neural network layer. The linking block may be configured to process a linking block input including an output of the final encoder block to generate a linking block output.

[0073] The segmentation neural network may include a decoder subnetwork having a plurality of decoder block sequences, wherein each decoder block is a residual neural network block comprising one or more two-dimensional convolutional neural network layers, one or more three-dimensional convolutional neural network layers, or both. Each decoder block is configured to process a decoder block input to generate a decoder block output having a higher spatial resolution than the decoder block input. The first decoder block may be configured to process an input comprising a linking block output, and each subsequent decoder block may be configured to process an input comprising: (i) an intermediate output of a corresponding encoder block; and (ii) an output of a previous decoder block.

[0074] The segmentation neural network may include a final layer configured to process a final layer input (e.g., from a decoder subnetwork) to generate a segmentation output. The final layer may process the final layer input according to the value of the final layer weight set to generate a transformed final layer input, and then process the transformed final layer input using a sigmoid activation function to generate a segmentation output.

[0075] The system can calculate a surface patch metric between (i) a segmentation of an organ in a medical image defined by a segmentation channel of a segmentation output and (ii) a reference (target) segmentation of the organ in the medical image. For example, the system can determine a first number of voxels in an intersection of (i) the segmented surface and a tolerance region around (ii) the reference segmented surface. The system can determine a second number of voxels in an intersection of (i) the reference segmented surface, and a tolerance region around (ii) the segmented surface. The system can determine the surface patch metric as a ratio of the sum of (i) the first number of voxels and the second number of voxels to the sum of the number of voxels in (ii) the segmented surface and the target segmented surface.

[0076] During training, the system adjusts the current values ​​of the segmentation neural network weights (506). In particular, the system determines a segmentation loss for the medical image by determining a corresponding set of error values ​​for each channel of the segmentation output. Each error value in the set of error values ​​for the segmentation channel corresponds to a corresponding voxel in the medical image and is based on an error between (i) a score from the segmentation channel that characterizes whether the voxel corresponds to the interior of the organ corresponding to the segmentation channel and (ii) a target score that defines whether the voxel corresponds to the interior of the organ corresponding to the segmentation channel. The system can calculate the error, for example, as a cross-entropy error. The system identifies a corresponding set of highest error values ​​for each segmentation channel (e.g., as a "proper subset" of the total set of error values ​​for the segmentation channel) and determines a segmentation loss based on the highest error value for each segmentation channel. For example, the system can determine the segmentation loss by summing the highest error values ​​for each segmentation channel. Then, the system adjusts the current values ​​of the segmentation neural network weights based on the segmentation loss, for example, by back-propagating the gradient of the segmentation loss through the segmentation neural network weights.

[0077] This specification uses the term "configured" in connection with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0078] The embodiments and functional operations of the subject matter described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their equivalents, or in a combination of one or more thereof. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, so as to be executed by a data processing device or to control the operation of the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more thereof. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0079] The term "data processing apparatus" refers to data processing hardware, and includes various devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof.

[0080] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files storing one or more modules, subroutines, or portions of code. A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a data communications network.

[0081] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same or multiple computers.

[0082] The processes and logic flows described in this specification may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by a special purpose logic circuit (e.g., an FPGA or ASIC) or by a combination of a special purpose logic circuit and one or more programmed computers.

[0083] A computer suitable for executing a computer program can be based on a general or special microprocessor or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for performing or executing instructions and one or more storage devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit or incorporated into the special logic circuit. Typically, the computer will also include or be operably coupled to receive data from one or more mass storage devices (e.g., magnetic, magneto-optical disks or optical disks) for storing data or transfer data to one or more mass storage devices or transfer data to these two or more mass storage devices. However, the computer does not have to have such a device. In addition, the computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device, such as a universal serial bus (USB) flash drive, etc.

[0084] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM optical disks.

[0085] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, as well as a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. Moreover, a computer may interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application), and then receiving a response message from the user.

[0086] A data processing device used to implement a machine learning model may also include, for example, a dedicated hardware accelerator unit for processing common and compute-intensive portions of a machine learning training or production (i.e., inference) workload.

[0087] You can use machine learning frameworks such as the TensorFlow framework, Microsoft Cognitive Toolkit framework, Apache Singa framework, or Apache MXNet framework to implement and deploy machine learning models.

[0088] Embodiments of the subject matter described in this specification may be implemented in a computer system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0089] A computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server, for example, sends data, such as an HTML page, to a user device for displaying data to a user interacting with the device (acting as a client) and receiving user input from the user. Data generated at the user device, such as the result of a user interaction, can be received from the device at the server.

[0090] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope of the claimed protection, but should be regarded as descriptions of features specific to particular embodiments of particular inventions. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments or in any suitable sub-combination. In addition, although the features may be described above as working in certain combinations and even initially claimed as such, in some cases one or more features of the claimed combination may be cut out from the combination, and the claimed combination may be directed to a sub-combination or a variant of the sub-combination.

[0091] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations must be performed in a particular order to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0092] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more data processing devices, the method comprising: receiving a medical image captured using a medical imaging modality and depicting a tissue region in a human body; The medical image is processed using a segmentation neural network to generate a segmentation output based on the trained values ​​of the plurality of segmentation neural network parameters, wherein: The segmentation output comprises a plurality of segmentation channels, each segmentation channel corresponds to a respective organ from a predetermined set of organs, and each segmentation channel defines a segmentation of the respective organ corresponding to the segmentation channel in the medical image; For each of a plurality of voxels in the medical image, segmentation of a corresponding organ in the medical image includes a corresponding score characterizing whether the voxel corresponds to an interior of the corresponding organ; The segmentation neural network comprises a sequence of multiple encoder blocks, wherein: Each encoder block is a residual neural network block including one or more two-dimensional convolutional neural network layers, one or more three-dimensional convolutional neural network layers, or both; Each encoder block is configured to process a corresponding encoder block input to generate a corresponding encoder block output, wherein a spatial resolution of the encoder block output is lower than a spatial resolution of the encoder block input; and for each encoder block after an initial encoder block in the sequence of encoder blocks, the encoder block input comprises a previous encoder block output of a previous encoder block in the sequence of encoder blocks; The segmentation neural network comprises a decoder sub-network, wherein the decoder sub-network is configured to process a decoder sub-network input comprising an intermediate output of each encoder block to generate the segmentation output; The decoder subnetwork includes a final layer configured to process a final layer input to generate the segmentation output; The segmentation neural network has been trained by the following multiple operations, the operations comprising: Processing a training medical image using the segmentation neural network to generate a training segmentation output; Determining the segmentation loss of the training medical image includes: For each segmentation channel of the training segmentation output: determining a set of error values ​​for the segmented channel, wherein each error value in the set of error values ​​for the segmented channel corresponds to a corresponding voxel in the training medical image and is based on an error between: (i) a score from the segmented channel characterizing whether the voxel corresponds to the interior of an organ corresponding to the segmented channel, and (ii) a target score defining whether the voxel corresponds to the interior of the organ corresponding to the segmented channel; and identifying a plurality of highest error values ​​from the set of error values ​​for the segmented channels, wherein the plurality of highest error values ​​are a proper subset of the set of error values ​​for the segmented channels; and determining the segmentation loss based on the plurality of highest error values ​​identified for each segmentation channel of the training segmentation output; and Current values ​​of the plurality of segmentation neural network parameters of the segmentation neural network are adjusted based on the segmentation loss of the training medical image.

2. The method according to claim 1, wherein: The medical imaging modality is a computed tomography (CT) medical imaging modality.

3. The method according to claim 1, wherein: The tissue region in the human body depicted by the medical image includes the head and neck region.

4. The method according to claim 1, wherein: The predetermined set of organs includes one or more organs at risk in a patient receiving radiation therapy treatment.

5. The method of claim 1, further comprising using the segmentation output in radiation therapy treatment planning.

6. The method according to claim 1, wherein: The decoder subnetwork comprises a sequence of multiple decoder blocks, where: Each decoder block is a residual neural network block including one or more two-dimensional convolutional neural network layers; Each decoder block is configured to process a corresponding decoder block input to generate a corresponding decoder block output, wherein a spatial resolution of the decoder block output is greater than a resolution of the decoder block input; and For each decoder block following an initial decoder block in the sequence of decoder blocks, the decoder block input includes: (i) an intermediate output of the corresponding encoder block, and (ii) a previous decoder block output of the previous decoder block.

7. The method according to claim 1, wherein: The 3D convolutional neural network layer in the encoder block includes padded xy-convolutions and unpadded z-convolutions.

8. The method according to claim 1, wherein: The segmentation neural network comprises a linking block, wherein: The link block is a residual neural network block including a fully connected layer; The linking block is configured to process a linking block input including an output of a final encoder block in the sequence of encoder blocks to generate a linking block output; and The linking block output is provided as input to the decoder subnetwork.

9. The method of claim 1 , further comprising calculating a surface patch metric between (i) a segmentation of a corresponding organ in the medical image defined by the segmentation channels from the segmentation output and (ii) a reference segmentation of the corresponding organ in the medical image, comprising: determining a number of voxels in a first intersection between (i) the segmented surface of the corresponding organ and (ii) a tolerance region around a reference segmented surface of the corresponding organ; determining a number of voxels in a second intersection between (i) the reference segmented surface of the corresponding organ and (ii) a tolerance region around the segmented surface of the corresponding organ; as well as The surface patch metric is determined as a ratio of (i) the sum of the number of voxels in the first intersection and the number of voxels in the second intersection to (ii) the sum of the number of voxels in the segmented surface of the corresponding organ and the number of voxels in the reference segmented surface of the corresponding organ.

10. The method according to any one of claims 1 to 9, wherein: Generating the segmentation output includes processing the output of a final neural network layer of the segmentation neural network using a sigmoid activation function.

11. A method for training a segmentation neural network performed by one or more data processing devices, the segmentation neural network configured to process a medical image to generate a segmentation output, the medical image being captured using a medical imaging modality and depicting tissue regions in a human body, wherein: The segmentation output comprises a plurality of segmentation channels, each segmentation channel corresponds to a respective organ from a predetermined set of organs, and each segmentation channel defines a segmentation of the respective organ corresponding to the segmentation channel in the medical image; and For each of a plurality of voxels in the medical image, the segmentation of the corresponding organ in the medical image includes a corresponding score characterizing whether the voxel corresponds to the interior of the corresponding organ, the method comprising: receiving a training medical image; processing the training medical images using the segmentation neural network to generate a training segmentation output; Determining the segmentation loss of the training medical image includes: For each segmentation channel of the training segmentation output: determining a set of error values ​​for the segmentation channel, wherein each error value in the set of error values ​​for the segmentation channel corresponds to a corresponding voxel in the medical image and is based on an error between (i) a score from the segmentation channel characterizing whether the voxel corresponds to an interior of an organ corresponding to the segmentation channel and (ii) a target score defining whether the voxel corresponds to an interior of the organ corresponding to the segmentation channel; and identifying a plurality of highest error values ​​in a set of error values ​​for the segmented channels; and determining the segmentation loss based on the plurality of highest error values ​​identified for each segmentation channel of the training segmentation output; and Current values ​​of segmentation neural network weights are adjusted based on the segmentation loss of the training medical image.

12. The method according to claim 11, wherein: The error between (i) a score from the segmented channel characterizing whether the voxel corresponds to the interior of the organ corresponding to the segmented channel and (ii) a target score defining whether the voxel corresponds to the interior of the organ corresponding to the segmented channel comprises: The cross entropy between (i) a score from the segmentation channel characterizing whether the voxel corresponds to the interior of the organ corresponding to the segmentation channel and (ii) a target score defining whether the voxel corresponds to the interior of the organ corresponding to the segmentation channel.

13. The method according to claim 11, wherein: Identifying a plurality of highest error values ​​in a set of error values ​​for the segmented channels comprises: The plurality of highest error values ​​in the set of error values ​​for the segmented channels are identified as a proper subset of the set of error values ​​for the segmented channels.

14. The method according to claim 11, wherein: Determining the segmentation loss includes summing the plurality of highest error values ​​identified for each segmentation channel of the training segmentation output.

15. The method according to claim 11, wherein: Adjusting the current values ​​of the segmentation neural network weights involves: Back-propagating the gradient of the segmentation loss of the training medical images.

16. The method according to claim 11, wherein: The medical imaging modality is a computed tomography (CT) medical imaging modality.

17. The method according to claim 11, wherein: The tissue region in the human body depicted by the medical image includes the head and neck region.

18. The method according to any one of claims 11 to 17, wherein: The predetermined set of organs includes one or more organs at risk in a patient receiving radiation therapy treatment.

19. A method performed by one or more data processing devices, the method comprising: receiving a medical image captured using a medical imaging modality and depicting a tissue region in a human body; The medical image is processed using a segmentation neural network to generate a segmentation output, wherein: The segmentation output comprises a plurality of segmentation channels, the plurality of segmentation channels comprising at least two segmentation channels, each segmentation channel corresponding to a respective organ from a predetermined set of organs, and each segmentation channel defines a segmentation of the respective organ corresponding to the segmentation channel in the medical image; For each of a plurality of voxels in the medical image, segmentation of a corresponding organ in the medical image includes a corresponding score characterizing whether the voxel corresponds to an interior of the corresponding organ; The segmentation neural network comprises a plurality of sequences of encoder blocks, wherein: Each encoder block is a residual neural network block including one or more two-dimensional convolutional neural network layers, one or more three-dimensional convolutional neural network layers, or both; Each encoder block is configured to process a corresponding encoder block input to generate a corresponding encoder block output, wherein a spatial resolution of the encoder block output is lower than a spatial resolution of the encoder block input; and for each encoder block after an initial encoder block in the sequence of encoder blocks, the encoder block input comprises a previous encoder block output of a previous encoder block in the sequence of encoder blocks; The segmentation neural network comprises a decoder sub-network, wherein the decoder sub-network is configured to process a decoder sub-network input comprising an intermediate output of each encoder block to generate the segmentation output; The decoder subnetwork comprises a final layer configured to process a final layer input to generate the segmentation output; and Processing the final layer input to generate the segmentation output comprises: processing the final layer input according to a final layer parameter set to generate a transformed final layer input; and applying a non-linear activation function to each component of the transformed final layer input to generate the segmented output, wherein the non-linear activation function is configured to process an input consisting of a single scalar value to generate an output consisting of a single scalar value, wherein, for one or more voxels of the medical image, the segmentation output defines that the voxel is included in each of a plurality of overlapping organs; The segmentation neural network has been trained through a plurality of operations, the operations comprising: Processing a training medical image using the segmentation neural network to generate a training segmentation output; Determining the segmentation loss of the training medical image includes: For each segmentation channel of the training segmentation output: determining a set of error values ​​for the segmented channel, wherein each error value in the set of error values ​​for the segmented channel corresponds to a corresponding voxel in the training medical image and is based on an error between: (i) a score from the segmented channel characterizing whether the voxel corresponds to the interior of an organ corresponding to the segmented channel, and (ii) a target score defining whether the voxel corresponds to the interior of the organ corresponding to the segmented channel; and identifying a plurality of highest error values ​​from the set of error values ​​for the segmented channels, wherein the plurality of highest error values ​​are a proper subset of the set of error values ​​for the segmented channels; and determining the segmentation loss based on the plurality of highest error values ​​identified for each segmentation channel of the training segmentation output; and Current values ​​of segmentation neural network weights of the segmentation neural network are adjusted based on the segmentation loss of the training medical image.

20. The method according to claim 19, wherein: The medical imaging modality is a computed tomography (CT) medical imaging modality.

21. The method according to claim 19, wherein: The tissue region in the human body depicted by the medical image includes the head and neck region.

22. The method according to claim 19, wherein: The predetermined set of organs includes one or more organs at risk in a patient receiving radiation therapy treatment.

23. The method of claim 19, further comprising using the segmentation output in radiation therapy treatment planning.

24. The method of any one of claims 19-23, further comprising calculating a surface patch metric between (i) a segmentation of a corresponding organ in the medical image defined by the segmentation channels from the segmentation output and (ii) a reference segmentation of the corresponding organ in the medical image, comprising: determining a number of voxels in a first intersection between (i) the segmented surface of the corresponding organ and (ii) a tolerance region around a reference segmented surface of the corresponding organ; determining a number of voxels in a second intersection between (i) the reference segmented surface of the corresponding organ and (ii) a tolerance region around the segmented surface of the corresponding organ; as well as The surface patch metric is determined as a ratio of (i) the sum of the number of voxels in the first intersection and the number of voxels in the second intersection to (ii) the sum of the number of voxels in the segmented surface of the corresponding organ and the number of voxels in the reference segmented surface of the corresponding organ.

25. A system for processing medical images, comprising one or more computers and one or more storage devices storing instructions, which when executed by the one or more computers cause the one or more computers to perform the operations of the method according to any one of claims 1-24.

26. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1-24.

Images