Multi-scale sub-network convergence with adaptive data sampling for brain injury detection and segmentation

Abstract

The invention relates to multi-scale sub-network convergence with adaptive data sampling for brain injury detection and segmentation. Systems and methods for segmenting one or more lesions from a medical image patch are provided. An input medical image patch depicting one or more lesions is received. One or more impairments are segmented from an input medical image patch using a plurality of machine learning based segmentation networks to generate a plurality of initial segmentation masks, respectively. Each of a plurality of machine learning based segmentation networks is trained to segment impairments from patches having different field-of-view sizes. A final segmentation mask of the one or more impairments is generated based on the plurality of initial segmentation masks. And outputting a final segmentation mask of the one or more damages.

Description

[0001] STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT This invention was made with government support awarded by the National Institutes of Health under Grant No. R01CA262182. The government has certain rights in this invention. Technical Field

[0002] The present invention relates generally to medical imaging analysis, and more particularly to multi-scale sub-network fusion with adaptive data sampling for brain lesion detection and segmentation. Background Art

[0003] Brain injury is an area of ​​damaged brain tissue due to, for example, illness, injury, disease or infection. Brain metastasis is a type of brain injury formed when cancer cells spread to the brain. Current treatment options for brain injury, such as, for example, stereotactic radiosurgery, can benefit from accurate and consistent automatic detection and segmentation of brain injury. However, brain metastasis and other brain injuries have great variability in position, size, shape and pathology. Due to this variability, it may be challenging to accurately detect small brain injuries and segment large complex injuries simultaneously. Summary of the invention

[0004] According to one or more embodiments, a system and method for segmenting one or more lesions from a medical image patch is provided. An input medical image patch depicting one or more lesions is received. One or more lesions are segmented from the input medical image patch using multiple machine learning-based segmentation networks to generate multiple initial segmentation masks, respectively. Each of the multiple machine learning-based segmentation networks is trained to segment lesions from patches with different field of view sizes. A final segmentation mask of one or more lesions is generated based on the multiple initial segmentation masks. A final segmentation mask of the one or more lesions is output.

[0005] In one embodiment, multiple initial segmentation masks are combined, and a final segmentation mask is generated based on the combined initial segmentation masks. The multiple initial segmentation masks can be combined by combining the intensity values ​​of corresponding voxels in the multiple initial segmentation masks into corresponding scores. A machine learning-based fusion network can be used to generate the final segmentation mask based on the scores.

[0006] In one embodiment, a plurality of machine learning based segmentation networks are trained by extracting training patches from training medical images depicting at least one lesion. The training patches are extracted from the training medical images based on a size of the at least one lesion.

[0007] In one embodiment, at least some of the multiple machine learning based segmentation networks are trained with different loss functions. In one embodiment, at least some of the multiple machine learning based segmentation networks are implemented with different network architectures. In one embodiment, at least some of the multiple machine learning based segmentation networks are implemented with different hyperparameters.

[0008] In one embodiment, the one or more lesions include one or more brain lesions located in the patient's brain.

[0009] These and other advantages of the present invention will become apparent to those of ordinary skill in the art by reference to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A method for detecting and segmenting one or more lesions from an input medical image patch using multiple machine learning based segmentation networks according to one or more embodiments is shown;

[0011] Figure 2 illustrates a workflow for training a machine learning based network for generating a final segmentation mask in accordance with one or more embodiments;

[0012] Figure 3 shows a performance table of the embodiments described herein;

[0013] Figure 4 An exemplary artificial neural network that can be used to implement one or more embodiments is shown;

[0014] Figure 5 shows a convolutional neural network that can be used to implement one or more embodiments; and

[0015] Figure 6 A high-level block diagram of a computer that can be used to implement one or more embodiments is shown. DETAILED DESCRIPTION

[0016] The present invention generally relates to methods and systems for multi-scale subnetwork fusion with adaptive data sampling for brain injury detection and segmentation. Embodiments of the present invention are described herein to give a visual understanding of such methods and systems. A digital image is typically composed of digital representations of one or more objects (or shapes). Here, the digital representation of an object is typically described in terms of the recognition and manipulation of the object. Such manipulation is a virtual manipulation performed in the memory or other circuits / hardware of a computer system. Therefore, it should be understood that embodiments of the present invention can be performed within a computer system using data stored within the computer system. In addition, unless otherwise stated, references to image pixels herein are equivalent to references to image voxels, and vice versa.

[0017] The embodiments described herein provide an AI / ML (artificial intelligence / machine learning) based system for both detecting small brain lesions and segmenting large complex brain lesions from medical images of a patient's brain. The AI / ML based system includes multiple independent machine learning based segmentation networks, each trained to perform segmentation on image patches with different field of view sizes using adaptive data sampling, thereby effectively optimizing the sensitivity of each segmentation network to a specific target lesion size. By combining the initial segmentation masks from multiple independent segmentation networks, the detection sensitivity of small lesion detection and the accuracy of large lesion segmentation are improved. Advantageously, the embodiments described herein mitigate the technical challenges of traditional methods for brain lesion detection and segmentation, thereby supporting clinicians in effectively diagnosing and treating patients with neurological diseases.

[0018] Figure 1 A method 100 for detecting and segmenting one or more lesions from an input medical image patch using multiple machine learning based segmentation networks according to one or more embodiments is shown. The steps of the method 100 may be performed by one or more suitable computing devices, such as, for example, Figure 6 Computer 602.

[0019] exist Figure 1 In step 102, an input medical image patch depicting one or more lesions is received. The one or more lesions may include a tumor, a nodule, or any other abnormality. In one embodiment, the one or more lesions include one or more brain lesions located in the patient's brain. However, the one or more lesions may be located on any other suitable anatomical object of interest to the patient, such as, for example, other organs, bones, blood vessels, etc.

[0020] In one embodiment, the input medical image patch comprises an MR (magnetic resonance) image patch. However, the input medical image patch may be any other suitable modality, such as, for example, CT (computed tomography), US (ultrasound), x-ray, or any other medical imaging modality or combination of medical imaging modalities. The input medical image patch may be a 2D (two-dimensional) patch or a 3D (three-dimensional) patch. The input medical image patch may be extracted from an acquired medical image of the patient (e.g., around one or more lesions), or the input medical image patch may be the entire acquired medical image. The input medical image patch may have any suitable field of view size.

[0021] The input medical image patch may be received directly from an image acquisition device (such as, for example, an MR scanner) as the input medical image patch is acquired, or may be received by loading a previously acquired image patch from a storage device or memory of a computer system, or may be received by receiving an image patch from a remote computer system.

[0022] exist Figure 1 In step 104, one or more lesions are segmented from the input medical image patch using a plurality of machine learning based segmentation networks to generate a plurality of initial segmentation masks, respectively. Each of the plurality of machine learning based segmentation networks is trained to segment lesions from a patch having a different field of view size. The field of view size refers to the size of the field of view of the patch. For example, the field of view size can be expressed in terms of voxels (or pixels), such as, for example, 32×32×32 for smaller lesions and 128×128×128 for larger lesions.

[0023] In one embodiment, the plurality of machine learning based segmentation networks include a plurality of 3D patch based encoder-decoder based networks. The encoder network of each of the plurality of encoder-decoder based networks receives an input medical image patch as input and encodes the input medical image patch to generate an embedding representing a latent feature of the input medical image patch as output. The decoder network of each of the plurality of encoder-decoder based networks receives an embedding output from its corresponding encoder network as input and decodes the embedding to generate a corresponding initial segmentation mask. The plurality of machine learning based segmentation networks may be any other suitable network architecture.

[0024] Each of the plurality of machine learning-based segmentation networks is independent and different. For example, in one embodiment, at least some of the plurality of machine learning-based segmentation networks are trained differently. For example, at least some of the plurality of machine learning-based segmentation networks may be trained to segment lesions from patches with different field of view sizes (e.g., using different training data corresponding to the field of view sizes). In another example, at least some of the plurality of machine learning-based segmentation networks may be trained to segment lesions from patches according to the same or different loss functions (e.g., prioritizing the execution of patches with a specific field of view size). Exemplary loss functions include cross entropy loss, volume adaptive soft block (softdice) loss, boundary loss, etc. In one embodiment, at least some of the plurality of machine learning-based segmentation networks may be implemented with different network architectures. In one embodiment, at least some of the plurality of machine learning-based segmentation networks may be implemented with the same or different hyperparameters (e.g., depending on the target application). For example, some of the plurality of machine learning-based segmentation networks may have fewer layers, while other networks of the plurality of machine learning-based segmentation networks may have larger layers.

[0025] A plurality of machine learning based segmentation networks are trained using the training data during a previous offline or training phase for segmenting lesions from the patch. In one embodiment, the plurality of machine learning based segmentation networks are as follows: Figure 2Once trained, multiple machine learning based segmentation networks are applied in an online or inference phase to segment lesions from the slices, such as in Figure 1 Step 104.

[0026] In one embodiment, the plurality of initial segmentation masks may be probability maps, wherein each corresponding voxel (or pixel) has an intensity value ranging from, for example, 0 to 1, which indicates the probability that the corresponding voxel in the input medical image patch depicts a lesion. In another embodiment, the plurality of initial segmentation masks may be binary maps generated by, for example, applying a threshold value (e.g., .5) to the probability maps, wherein each corresponding voxel of the binary maps has an intensity value, for example, 1, which indicates that the corresponding voxel in the input medical image patch depicts a lesion, or for example, 0, which indicates that the corresponding voxel in the input medical image patch does not depict a lesion. The segmentation masks may be in any other suitable form.

[0027] exist Figure 1 In step 106, a final segmentation mask of one or more lesions is generated based on the multiple initial segmentation masks. The final segmentation mask can be a probability map, a binary map, or can be represented in any other suitable form. The final segmentation mask can be generated according to any suitable method. In one embodiment, the final segmentation mask is generated by combining the multiple initial segmentation masks and generating the final segmentation mask based on the combined initial segmentation masks.

[0028] The multiple initial segmentation masks can be combined by a multi-scale feature generation module, which computationally combines the multiple initial segmentation masks. For example, in one embodiment, the multi-scale feature generation module respectively combines the intensity values ​​of corresponding voxels in the multiple initial segmentation masks into corresponding scores. Any suitable method can be used to combine the intensity values ​​of corresponding voxels, such as, for example, mean, median, mode, addition, etc. The score can be represented by a feature vector representing the combined initial segmentation mask.

[0029] A machine learning based fusion network is then used to generate a final segmentation mask based on the feature vector. The machine learning based fusion network receives the scored feature vector as input and generates a final segmentation mask as output. In one embodiment, the machine learning based fusion network is implemented as a random forest or boosted tree to perform fusion by weighted voting. In another embodiment, the machine learning based fusion network is implemented as an encoder-decoder network. However, the machine learning based fusion network can be implemented according to any other suitable network architecture.

[0030] The machine learning based fusion network is trained using training data during a previous offline or training phase. In one embodiment, the machine learning based fusion network is as follows: Figure 2Once trained, the machine learning based fusion network is applied during an online or inference phase to generate the final segmentation mask, such as in Figure 1 Step 106.

[0031] exist Figure 1 In step 108, a final segmentation mask of one or more lesions is output. For example, the final segmentation mask of one or more lesions can be output by displaying the final segmentation mask on a display device of the computer system, storing the final segmentation mask on a memory or storage device of the computer system, or by evaluating the final segmentation mask to a remote computer system.

[0032] Figure 2 A workflow 200 for training machine learning-based networks to generate final segmentation masks is shown in accordance with one or more embodiments. The workflow 200 is performed during a previous offline or training phase to train various machine learning-based networks for generating final segmentation masks. Once trained, the trained machine learning-based networks are applied during an online or inference phase. In one example, the scale-dependent subnetworks trained in steps 206-A, 206-B, and 206-C in the workflow 200 are Figure 1 Step 104 utilizes multiple machine learning based segmentation networks, and the final predictor network trained in step 210 is Figure 1 Step 106 utilizes a fusion network based on machine learning.

[0033] In workflow 200, training data 202 is received. The training data 202 includes training medical images of a patient population depicting at least one lesion. Adaptive data sampling 204 based on lesion size is performed to extract training patches depicting one or more lesions from the training medical images of the training data 202 as training samples. The training patches are extracted based on the size of at least one lesion depicted in the training medical images. In one embodiment, the adaptive data sampling 204 can be performed by annotating the bounding box of the lesion with a pre-computed ground truth or an online connected component analysis of the lesion.

[0034] Then, in steps 206-A, 206-B, and 206-C, the scale-dependent subnetworks are trained separately on the training samples (collectively referred to as scale-dependent subnetwork training 206) to generate initial segmentation masks as scale-dependent probability masks. The scale-dependent subnetworks can be implemented as encoder-decoder networks. Each of the scale-dependent subnetworks is independent and different. In one embodiment, one or more scale-dependent subnetworks are trained differently in scale-dependent subnetwork training 206. For example, one or more scale-dependent subnetworks can be trained to segment lesions from patches with different field of view sizes (e.g., using different training data corresponding to the field of view sizes). In another example, one or more scale-dependent subnetworks can be trained to segment lesions from patches according to the same or different loss functions (e.g., prioritizing the execution of patches with a specific field of view size). The scale-dependent subnetworks can be first trained with unsupervised learning and then refined with supervised learning using annotated samples. In one embodiment, one or more scale-dependent subnetworks can be implemented with different network architectures. In one embodiment, one or more scale-dependent subnetworks can be implemented with the same or different hyperparameters (e.g., depending on the target application).

[0035] The multi-scale feature generation module 208 then computationally combines the scale-dependent probability masks by combining the intensity values ​​of each group of corresponding voxels in the scale-dependent probability masks into scores, respectively, and the scores of each group of corresponding voxels are represented in a feature vector. Then, the final predictor network is trained 210 to generate a final predicted lesion mask 212 from the feature vector. The final predictor network can be trained 210 using supervised learning using the training data 202 with the ground truth segmentation. In order to characterize the biased predictions and false positives of each subnetwork, the predicted lesions (in the scale-dependent probability masks) output from the scale-dependent subnetworks can be used to generate features using a predefined window kernel (representing the patch area of ​​the medical image) for training 210 the final predictor network. The features can be sampled from the window kernels of both the lesion area and the normal area. In one embodiment, the final predictor network can be implemented as a random forest or boosted tree trained on the generated features to perform fusion by weighted voting. In another embodiment, the final predictor network can be implemented as an encoder-decoder network.

[0036] The embodiment described here is experimentally validated. Using 1856 MRI volumes (1539 for training, 183 for validation, and 134 for testing) of 1791 patients treated with SRS (stereotactic radiosurgery) from 7 institutions, a 3D U-Net system was trained and evaluated to detect and segment intraparenchymal metastases with a size greater than 2mm (millimeter). All patients underwent 3D post-Gd (gadolinium) T1w (T1 weighted) MRI scans before SRS. The total tumor volume (GTV) of brain metastases from SRS was first screened by each institution. Then, additional efforts were spent to create GTVs for untreated and / or uncontoured metastases, including centralized review by 2 radiologists to improve the accuracy of the ground truth. Using a 3D generation pipeline, the training data set was augmented with synthetic metastases from 3773 MRIs. The proposed system consists of two U-Nets, one of which uses a small 3D patch dedicated to detecting small metastases and the other one uses a large 3D patch for segmenting large metastases, and a random forest-based fusion module for combining the outputs of the two networks. 3 The first U-Net was trained on 3D patches of metastases with a size of (cubic centimeters). For detection performance, sensitivity at the metastasis level and false positive (FP) rate at the case level were measured. For segmentation performance, Dice similarity coefficient (DSC) and 95th percentile Hausdorff distance (HD95) at the brain metastasis level were measured. Performance was also stratified according to the size of the brain metastases.

[0037] Figure 3 Table 300 showing the performance of the embodiments described herein according to brain metastasis size, where N is the number of brain metastases in each category. For 739 brain metastases in 134 test cases, the overall lesion level sensitivity was 0.870, with an average case level FP of 1.34 ± 1.92 (95% CI: 1.02-1.67). For brain metastases larger than 0.1 cm, the overall lesion level sensitivity was 0.870, with an average case level FP of 1.34 ± 1.92 (95% CI: 1.02-1.67). 3 The sensitivity of brain metastases is 0.969, but for those smaller than 0.1 cm 3 The sensitivity of all brain metastases detected was 0.755 (as shown in Table 300). The average DSC and HD95 of all brain metastases detected were 0.786 and 1.35 mm, respectively. 3 The poor performance of brain metastases is caused by 3 Cases caused by artifacts in the GTV and MRI volumes.

[0038] For more than 0.1cm 3 The detection sensitivity and segmentation accuracy of small brain metastases (less than 0.1 cm) were 3), using a large multi-institutional dataset with controlled FP rate, with good performance.

[0039] The embodiments described herein are described with respect to the claimed systems and the claimed methods. Features, advantages, or alternative embodiments herein may be assigned to other claimed objects, and vice versa. In other words, the claims for the system may be improved with features described or claimed in the context of the method. In this case, the functional features of the method are embodied by the object unit providing the system.

[0040] In addition, certain embodiments described herein are described with respect to methods and systems for utilizing trained machine learning-based models, and with respect to methods and systems for training machine learning-based models. Features, advantages, or alternative embodiments herein may be assigned to other claimed objects, and vice versa. In other words, claims to methods and systems for training machine learning-based models may be improved with features described or claimed in the context of methods and systems for utilizing trained machine learning-based models, and vice versa.

[0041] In particular, the trained machine learning-based models applied in the embodiments described herein can be adjusted by the methods and systems for training machine learning-based models. In addition, the input data of the trained machine learning-based models can include the advantageous features and embodiments of the training input data, and vice versa. In addition, the output data of the trained machine learning-based models can include the advantageous features and embodiments of the output training data, and vice versa.

[0042] Generally speaking, trained machine learning-based models mimic the cognitive functions that humans associate with other human minds. In particular, trained machine learning-based models are able to adapt to new environments and detect and infer patterns through training based on training data.

[0043] Typically, the parameters of a machine learning based model can be adjusted by training. In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning and / or active learning can be used. In addition, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of a trained machine learning based model can be iteratively adapted through several training steps.

[0044] In particular, the trained machine learning based model may include a neural network, a support vector machine, a decision tree and / or a Bayesian network, and / or the trained machine learning based model may be based on k-means clustering, Q learning, a genetic algorithm and / or an association rule. In particular, the neural network may be a deep neural network, a convolutional neural network or a convolutional deep neural network. In addition, the neural network may be an adversarial network, a deep adversarial network and / or a generative adversarial network.

[0045] Figure 4 An embodiment of an artificial neural network 400 is shown in accordance with one or more embodiments. Alternative terms for "artificial neural network" are "neural network," "artificial neural network," or "neural network." Artificial neural network 400 may be used to implement the machine learning networks described herein, such as, for example, in Figure 1 The multiple segmentation networks based on machine learning used in step 104 and the fusion network based on machine learning used in step 106, and Figure 2 The scale-dependent subnetwork trained at step 206 and the final predictor network trained at step 210 in workflow 200 of FIG.

[0046] The artificial neural network 400 includes nodes 402-422 and edges 432, 434, ..., 436, wherein each edge 432, 434, ..., 436 is a directed connection from a first node 402-422 to a second node 402-422. Typically, the first node 402-422 and the second node 402-422 are different nodes 402-422, but it is also possible that the first node 402-422 and the second node 402-422 are the same. For example, in Figure 4 , edge 432 is a directed connection from node 402 to node 406, and edge 434 is a directed connection from node 404 to node 406. Edges 432, 434, ..., 436 from a first node 402-422 to a second node 402-422 are also denoted as "incoming edges" of the second node 402-422 and "outgoing edges" of the first node 402-422.

[0047] In this embodiment, the nodes 402-422 of the artificial neural network 400 may be arranged in layers 424-430, wherein the layers may include an inherent order introduced by edges 432, 434, ..., 436 between the nodes 402-422. In particular, the edges 432, 434, ..., 436 may only exist between adjacent node layers. Figure 4In the illustrated embodiment, there is an input layer 424 that includes only nodes 402 and 404 and no incoming edges, an output layer 430 that includes only nodes 422 and no outgoing edges, and hidden layers 426, 428 between the input layer 424 and the output layer 430. In general, the number of hidden layers 426, 428 can be selected arbitrarily. The number of nodes 402 and 404 within the input layer 424 is generally related to the number of input values ​​of the neural network 400, and the number of nodes 422 within the output layer 430 is generally related to the number of output values ​​of the neural network 400.

[0048] In particular, a (real) number may be assigned as a value to each node 402-422 of the neural network 400. Here, x (n) i represents the value of the i-th node 402-422 of the n-th layer 424-430. The value of the node 402-422 of the input layer 424 is equivalent to the input value of the neural network 400, and the value of the node 422 of the output layer 430 is equivalent to the output value of the neural network 400. In addition, each edge 432, 434, ..., 436 may include a weight as a real number, in particular, the weight is a real number in the interval [-1, 1] or in the interval [0, 1]. Here, w (m，n) i，j represents the weight of the edge between the i-th node 402-422 of the m-th layer 424-430 and the j-th node 402-422 of the n-th layer 424-430. (n) i，j The weight w is defined as (n，n+1) i，j .

[0049] Specifically, to calculate the output value of the neural network 400, the input value is propagated through the neural network. Specifically, the value of the nodes 402-422 of the (n+1)th layer 424-430 can be calculated based on the value of the nodes 402-422 of the nth layer 424-430 by the following formula:

[0050] Here, function f is a transfer function (another term is "activation function"). Known transfer functions are step functions, sigmoid functions (e.g., logistic function, generalized logistic function, hyperbolic tangent function, inverse tangent function, error function, smoothed step function), or rectifier functions. Transfer functions are mainly used for normalization purposes.

[0051] In particular, the values ​​are propagated layer by layer through the neural network, wherein the value of the input layer 424 is given by the input of the neural network 400, wherein the value of the first hidden layer 426 can be calculated based on the value of the input layer 424 of the neural network, wherein the value of the second hidden layer 428 can be calculated based on the value of the first hidden layer 426, and so on.

[0052] To set the value of edge w (m，n) i，j , training data must be used to train the neural network 400. In particular, the training data includes training input data and training output data (denoted as t i ). For the training step, the neural network 400 is applied to the training input data to generate the calculated output data. In particular, the training data and the calculated output data include a certain number of values, which is equal to the number of nodes of the output layer.

[0053] In particular, the comparison between the calculated output data and the training data is used to recursively adjust the weights within the neural network 400 (back propagation algorithm). In particular, the weights vary according to the following formula where γ is the learning rate, and the number δ (n) j Based on δ (n+1) j Recursively computed as If the (n+1)th layer is not the output layer, and If the (n+1)th layer is the output layer 430, where f' is the first derivative of the activation function, and y (n+1) j is the comparison training value of the j-th node of the output layer 430.

[0054] Figure 5 A convolutional neural network 500 is shown in accordance with one or more embodiments. The convolutional neural network 500 may be used to implement the machine learning networks described herein, such as, for example, in Figure 1 The multiple segmentation networks based on machine learning used in step 104 and the fusion network based on machine learning used in step 106, and Figure 2 The scale-dependent subnetwork trained in step 206 and the final prediction network trained in step 210 in the workflow 200 of FIG.

[0055] exist Figure 5 In the illustrated embodiment, the convolutional neural network 500 includes an input layer 502, a convolutional layer 504, a pooling layer 506, a fully connected layer 508, and an output layer 510. Alternatively, the convolutional neural network 500 may include several convolutional layers 504, several pooling layers 506, and several fully connected layers 508, as well as other types of layers. The order of the layers can be selected arbitrarily, and the fully connected layer 508 is usually used as the last layer before the output layer 510.

[0056] In particular, within the convolutional neural network 500, the nodes 512-520 of a layer 502-510 can be considered to be arranged as a d-dimensional matrix or a d-dimensional image. In particular, in the two-dimensional case, the value of the node 512-520 indexed by i and j in the nth layer 502-510 can be represented as x (n) [i，j] However, the arrangement of the nodes 512-520 of a layer 502-510 as such has no effect on the computations performed within the convolutional neural network 500, as these are given solely by the structure and weights of the edges.

[0057] In particular, the convolution layer 504 is characterized by the structure and weights of the incoming edges that form a convolution operation based on a specific number of kernels. In particular, the structure and weights of the incoming edges are selected so that the value x of the node 514 of the convolution layer 504 is (n) k is calculated as the value x based on the node 512 at the previous layer 502 (n-1) The convolution of x (n) k =K k *x (n-1) , where convolution* is defined in two dimensions as

[0058] Here, the kth kernel K k is a d-dimensional matrix (a two-dimensional matrix in this embodiment), which is typically small compared to the number of nodes 512-518 (e.g., a 3×3 matrix or a 5×5 matrix). In particular, this means that the weights of the incoming edges are not independent, but are chosen so that they produce the convolution equation. In particular, for the kernel as a 3×3 matrix, there are only 9 independent weights (each entry of the kernel matrix corresponds to an independent weight), regardless of the number of nodes 512-520 in the corresponding layer 502-510. In particular, for a convolutional layer 504, the number of nodes 514 in the convolutional layer is equivalent to the number of nodes 512 in the previous layer 502 multiplied by the number of kernels.

[0059] If the nodes 512 in the previous layer 502 are arranged as a d-dimensional matrix, the use of multiple kernels can be interpreted as adding an additional dimension (denoted as the "depth" dimension) so that the nodes 514 of the convolution layer 504 are arranged as a (d+1)-dimensional matrix. If the nodes 512 in the previous layer 502 are already arranged as a (d+1)-dimensional matrix including the depth dimension, the use of multiple kernels can be interpreted as expanding along the depth dimension so that the nodes 514 of the convolution layer 504 are also arranged as a (d+1)-dimensional matrix, where the size of the (d+1)-dimensional matrix relative to the depth dimension is a multiple of the number of kernels in the previous layer 502.

[0060] The advantage of using convolutional layers 504 is that the spatial local correlation of the input data can be exploited by enforcing local connectivity patterns between nodes in adjacent layers, in particular by having each node only connected to a small region of nodes in the previous layer.

[0061] exist Figure 5 In the illustrated embodiment, the input layer 502 includes 36 nodes 512 arranged in a two-dimensional 6×6 matrix. The convolution layer 504 includes 72 nodes 514 arranged in two two-dimensional 6×6 matrices, each of which is the result of the convolution of the values ​​of the input layer with the kernel. Equivalently, the nodes 514 of the convolution layer 504 can be interpreted as being arranged in a three-dimensional 6×6×2 matrix, where the last dimension is the depth dimension.

[0062] The pooling layer 506 can be characterized by the structure and weights of the incoming edges and the activation functions of its nodes 516, thereby forming a pooling operation based on a nonlinear pooling function f. For example, in the two-dimensional case, the value x of the node 516 of the pooling layer 506 (n) The value x of the node 514 in the previous layer 504 may be (n-1) To calculate x (n) [i，j] =f(x (n-1) [id1, jd2], ..., x (n-1) [id1+d1-1, jd2+d2-1]).

[0063] In other words, by using the pooling layer 506, the number of nodes 514, 516 can be reduced by replacing the number d1·d2 of neighboring nodes 514 in the previous layer 504, where a single node 516 is calculated as a function of the values ​​of the number of neighboring nodes in the pooling layer. In particular, the pooling function f can be a maximum function, an average value, or an L2 norm. In particular, for the pooling layer 506, the weights of the incoming edges are fixed and are not modified by training.

[0064] The advantage of using the pooling layer 506 is that the number of nodes 514, 516 and the number of parameters are reduced. This results in a reduction in the amount of computation in the network and controls overfitting.

[0065] exist Figure 5 In the embodiment shown, the pooling layer 506 is a maximum pooling, which replaces four neighboring nodes with only one node, the value of which is the maximum of the values ​​of the four neighboring nodes. Maximum pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, maximum pooling is applied to each of the two two-dimensional matrices, thereby reducing the number of nodes from 72 to 18.

[0066] The fully connected layer 508 is characterized by the fact that most, in particular all, edges exist between the nodes 516 of the previous layer 506 and the nodes 518 of the fully connected layer 508, and wherein the weight of each edge can be adjusted individually.

[0067] In this embodiment, the nodes 516 of the previous layer 506 of the fully connected layer 508 are displayed as a two-dimensional matrix, and are also displayed as irrelevant nodes (indicated as a row of nodes, where the number of nodes is reduced for better presentability). In this embodiment, the number of nodes 518 in the fully connected layer 508 is equal to the number of nodes 516 in the previous layer 506. Alternatively, the number of nodes 516, 518 may be different.

[0068] Furthermore, in this embodiment, the values ​​of the nodes 520 of the output layer 510 are determined by applying the Softmax function to the values ​​of the nodes 518 of the previous layer 508. By applying the Softmax function, the sum of the values ​​of all the nodes 520 of the output layer 510 is 1, and all the values ​​of all the nodes 520 of the output layer are real numbers between 0 and 1.

[0069] The convolutional neural network 500 may also include a ReLU (rectified linear unit) layer or activation layer having a nonlinear transfer function. In particular, the number of nodes and the node structure included in the ReLU layer are equivalent to the number of nodes and the node structure included in the previous layer. In particular, the value of each node in the ReLU layer is calculated by applying a rectification function to the value of the corresponding node in the previous layer.

[0070] The inputs and outputs of different CNN blocks can be wired using sums (residual / dense NNs), element-wise multiplications (attention), or other differentiable operators. Thus, if the entire pipeline is differentiable, the CNN architecture can be nested instead of sequential.

[0071] In particular, the convolutional neural network 500 can be trained based on a back-propagation algorithm. To prevent overfitting, regularization methods can be used, such as dropping of nodes 512-520, random pooling, use of artificial data, weight decay based on L1 or L2 norm or maximum norm constraints. Different loss functions can be combined for training the same neural network to reflect joint training objectives. A subset of neural network parameters can be excluded from optimization to retain weights pre-trained on another dataset.

[0072] The systems, apparatus, and methods described herein may be implemented using digital circuits, or using one or more computers using known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include or be coupled to one or more mass storage devices, such as one or more magnetic disks, internal hard disks and removable disks, magneto-optical disks, optical disks, and the like.

[0073] The systems, devices, and methods described herein can be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computer is remote from the server computer and interacts through a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers.

[0074] The systems, devices, and methods described herein can be implemented in a network-based cloud computing system. In such a network-based cloud computing system, a server or another processor connected to the network communicates with one or more client computers via the network. For example, a client computer can communicate with a server via a web browser application resident and running on the client computer. The client computer can store data on the server and access the data via the network. The client computer can send a request for data or a request for an online service to the server via the network. The server can perform the requested service and provide the data to the (multiple) client computers. The server can also transmit data suitable for causing the client computer to perform a specified function, such as performing a calculation, displaying specified data on a screen, etc. For example, the server can transmit a request suitable for causing the client computer to perform one or more steps or functions of the methods and workflows described herein, including Figure 1 or one or more steps or functions of 2. Certain steps or functions of the methods and workflows described herein include Figure 1 One or more steps or functions of or 2 may be performed by a server or another processor in a network-based cloud computing system. Certain steps or functions of the methods and workflows described herein include Figure 1 One or more steps of or 2 may be performed by a client computer in a network-based cloud computing system. The steps or functions of the methods and workflows described herein include Figure 1 One or more steps of or 2 may be performed by a server and / or a client computer in a network-based cloud computing system in any combination.

[0075] The systems, apparatus, and methods described herein may be implemented using a computer program product tangibly embodied in an information carrier, such as in a non-transitory machine-readable storage device, for execution by a programmable processor; and the methods and workflow steps described herein, including Figure 1 One or more steps or functions of or 2 may be implemented using one or more computer programs executable by such a processor. A computer program is a set of computer program instructions that can be used directly or indirectly in a computer to perform a specific activity or produce a specific result. A computer program may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0076] Figure 6 A high-level block diagram of an example computer 602 that can be used to implement the systems, apparatus, and methods described herein is depicted in . The computer 602 includes a processor 604 operably coupled to a data storage device 612 and a memory 610. The processor 604 controls the overall operation of the computer 602 by executing computer program instructions that define these operations. The computer program instructions may be stored in the data storage device 612 or other computer-readable medium and loaded into the memory 610 when execution of the computer program instructions is required. Thus, Figure 1 The method and workflow steps or functions of the method 600 or 601 may be defined by computer program instructions stored in the memory 610 and / or the data storage device 612 and controlled by the processor 604 executing the computer program instructions. For example, the computer program instructions may be implemented as a computer executable code programmed by a person skilled in the art to execute Figure 1 or 2. Thus, by executing the computer program instructions, the processor 604 performs Figure 1 The computer 602 may also include one or more network interfaces 606 for communicating with other devices via a network. The computer 602 may also include one or more input / output devices 608 (e.g., a display, keyboard, mouse, speaker, buttons, etc.) that enable user interaction with the computer 602.

[0077] The processor 604 may include both general and special purpose microprocessors, and may be the sole processor or one of multiple processors of the computer 602. For example, the processor 604 may include one or more central processing units (CPUs). The processor 604, the data storage device 612, and / or the memory 610 may include, be supplemented by, or incorporate one or more application specific integrated circuits (ASICs) and / or one or more field programmable gate arrays (FPGAs).

[0078] The data storage device 612 and the memory 610 each include a tangible, non-transitory computer-readable storage medium. The data storage device 612 and the memory 610 may each include a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM), digital versatile disk read-only memory (DVD-ROM) disks, or other non-volatile solid-state storage devices.

[0079] Input / output devices 608 may include peripheral devices such as printers, scanners, display screens, etc. For example, input / output devices 608 may include a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor for displaying information to a user, a keyboard, and a pointing device such as a mouse or trackball through which a user can provide input to computer 602.

[0080] The image acquisition device 614 can be connected to the computer 602 to input image data (e.g., medical images) to the computer 602. It is possible to implement the image acquisition device 614 and the computer 602 as one device. The image acquisition device 614 and the computer 602 can also communicate wirelessly over a network. In a possible embodiment, the computer 602 can be located remotely relative to the image acquisition device 614.

[0081] Any or all of the systems and devices discussed herein may be implemented using one or more computers, such as computer 602 .

[0082] Those skilled in the art will recognize that the actual computer or computer system implementation may have other structures and may also include other components, and Figure 6is a high-level representation of some of the components of such a computer for illustration purposes.

[0083] Independent of the use of the grammatical term, individuals with either male or female gender identities are also included in the term.

[0084] The foregoing detailed description should be understood to be illustrative and exemplary in every respect, rather than restrictive, and the scope of the invention disclosed herein is not determined by the detailed description, but by the claims interpreted in accordance with the full breadth allowed by patent law. It should be understood that the embodiments shown and described herein are merely illustrative of the principles of the invention, and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Various other feature combinations may be implemented by those skilled in the art without departing from the scope and spirit of the invention.

[0085] The following is a list of non-limiting illustrative embodiments disclosed herein:

[0086] Illustrative embodiments 1. A computer-implemented method comprising: receiving an input medical image patch depicting one or more lesions; segmenting the one or more lesions from the input medical image patch using multiple machine learning-based segmentation networks to respectively generate multiple initial segmentation masks, each of the multiple machine learning-based segmentation networks being trained to segment lesions from patches with different field of view sizes; generating a final segmentation mask of one or more lesions based on the multiple initial segmentation masks; and outputting the final segmentation mask of the one or more lesions.

[0087] Illustrative embodiment 2. The computer-implemented method of illustrative embodiment 1, wherein generating a final segmentation mask of one or more lesions based on multiple initial segmentation masks comprises: combining the multiple initial segmentation masks; and generating a final segmentation mask based on the combined initial segmentation masks.

[0088] Illustrative embodiment 3. The computer-implemented method of any of illustrative embodiments 1-2, wherein combining a plurality of initial segmentation masks comprises combining intensity values ​​of corresponding voxels in the plurality of initial segmentation masks into corresponding scores.

[0089] Illustrative embodiment 4. The computer-implemented method of any of illustrative embodiments 1-3, wherein generating a final segmentation mask based on the combined initial segmentation masks comprises: generating the final segmentation mask based on the scores using a machine learning-based fusion network.

[0090] Illustrative embodiment 5. A computer-implemented method according to any of illustrative embodiments 1-4, wherein the multiple machine learning-based segmentation networks are trained by extracting training patches from training medical images depicting at least one lesion, wherein the training patches are extracted from the training medical images based on the size of the at least one lesion.

[0091] Illustrative embodiment 6. The computer-implemented method of any of illustrative embodiments 1-5, wherein at least some of the plurality of machine learning based segmentation networks are trained using different loss functions.

[0092] Illustrative embodiment 7. The computer-implemented method of any of illustrative embodiments 1-6, wherein at least some of the plurality of machine learning based segmentation networks are implemented using different network architectures.

[0093] Illustrative embodiment 8. The computer-implemented method of any of illustrative embodiments 1-7, wherein at least some of the plurality of machine learning-based segmentation networks are implemented using different hyperparameters.

[0094] Illustrative Embodiment 9. The computer-implemented method of any of Illustrative Embodiments 1-8, wherein the one or more lesions include one or more brain lesions located in the patient's brain.

[0095] Illustrative embodiment 10. An apparatus comprising: a device for receiving an input medical image patch depicting one or more lesions; a device for segmenting one or more lesions from the input medical image patch using multiple machine learning-based segmentation networks to respectively generate multiple initial segmentation masks, each of the multiple machine learning-based segmentation networks being trained to segment lesions from patches with different field of view sizes; a device for generating a final segmentation mask of one or more lesions based on the multiple initial segmentation masks; and a device for outputting the final segmentation mask of the one or more lesions.

[0096] Illustrative embodiment 11. An apparatus according to illustrative embodiment 10, wherein the device for generating a final segmentation mask of one or more lesions based on multiple initial segmentation masks includes: a device for combining multiple initial segmentation masks; and a device for generating a final segmentation mask based on the combined initial segmentation masks.

[0097] Illustrative embodiment 12. The apparatus of any of illustrative embodiments 10-11, wherein the means for combining a plurality of initial segmentation masks comprises means for combining intensity values ​​of corresponding voxels in the plurality of initial segmentation masks into corresponding scores.

[0098] Illustrative embodiment 13. The apparatus of any of illustrative embodiments 10-12, wherein the means for generating a final segmentation mask based on the combined initial segmentation masks comprises: means for generating the final segmentation mask based on the scores using a machine learning based fusion network.

[0099] Illustrative embodiment 14. An apparatus according to any one of illustrative embodiments 10-13, wherein the multiple machine learning-based segmentation networks are trained by extracting training patches from training medical images depicting at least one lesion, and the training patches are extracted from the training medical images based on the size of the at least one lesion.

[0100] Illustrative embodiment 15. A non-temporary computer-readable medium storing computer program instructions which, when executed by a processor, cause the processor to perform operations including: receiving an input medical image patch depicting one or more lesions; segmenting the one or more lesions from the input medical image patch using a plurality of machine learning-based segmentation networks to respectively generate a plurality of initial segmentation masks, each of the plurality of machine learning-based segmentation networks being trained to segment lesions from patches having different field of view sizes; generating a final segmentation mask of the one or more lesions based on the plurality of initial segmentation masks; and outputting the final segmentation mask of the one or more lesions.

[0101] Illustrative embodiment 16. The non-transitory computer-readable medium of illustrative embodiment 15, wherein generating a final segmentation mask of one or more lesions based on a plurality of initial segmentation masks comprises: combining the plurality of initial segmentation masks; and generating a final segmentation mask based on the combined initial segmentation masks.

[0102] Illustrative embodiment 17. The non-transitory computer readable medium of any of illustrative embodiments 15-16, wherein at least some of the plurality of machine learning based segmentation networks are trained using different loss functions.

[0103] Illustrative embodiment 18. The non-transitory computer readable medium of any of illustrative embodiments 15-17, wherein at least some of the plurality of machine learning based segmentation networks are implemented using different network architectures.

[0104] Illustrative embodiment 19. The non-transitory computer-readable medium of any of illustrative embodiments 15-18, wherein at least some of the plurality of machine learning-based segmentation networks are implemented using different hyperparameters.

[0105] Illustrative Embodiment 20. The non-transitory computer readable medium of any of Illustrative Embodiments 15-19, wherein the one or more lesions include one or more brain lesions located in the patient's brain.

Claims

1. A computer-implemented method comprising: receiving an input medical image patch depicting one or more lesions; segmenting the one or more lesions from the input medical image patch using a plurality of machine learning-based segmentation networks to respectively generate a plurality of initial segmentation masks, each of the plurality of machine learning-based segmentation networks being trained to segment lesions from patches having different field of view sizes; generating a final segmentation mask of one or more lesions based on the plurality of initial segmentation masks; as well as Output a final segmentation mask of one or more lesions.

2. The computer-implemented method of claim 1 , wherein generating a final segmentation mask of one or more lesions based on a plurality of initial segmentation masks comprises: Combining multiple initial segmentation masks; as well as A final segmentation mask is generated based on the combined initial segmentation masks.

3. The computer-implemented method of claim 2 , wherein combining a plurality of initial segmentation masks comprises: The intensity values ​​of corresponding voxels in multiple initial segmentation masks are combined into a corresponding score.

4. The computer-implemented method of claim 3, wherein generating a final segmentation mask based on the combined initial segmentation masks comprises: The final segmentation mask is generated based on the scores using a machine learning based fusion network.

5. The computer-implemented method of claim 1 , wherein the plurality of machine learning-based segmentation networks are trained by extracting training patches from training medical images depicting at least one lesion, the training patches being extracted from the training medical images based on a size of the at least one lesion.

6. The computer-implemented method of claim 1, wherein at least some of the plurality of machine learning based segmentation networks are trained using different loss functions.

7. The computer-implemented method of claim 1 , wherein at least some of the plurality of machine learning based segmentation networks are implemented using different network architectures.

8. The computer-implemented method of claim 1 , wherein at least some of the plurality of machine learning based segmentation networks are implemented using different hyperparameters.

9. The computer-implemented method of claim 1, wherein the one or more lesions comprise one or more brain lesions located in a patient's brain.

10. A device comprising: means for receiving an input medical image patch depicting one or more lesions; means for segmenting one or more lesions from an input medical image patch using a plurality of machine learning-based segmentation networks to respectively generate a plurality of initial segmentation masks, each of the plurality of machine learning-based segmentation networks being trained to segment lesions from patches having different field of view sizes; means for generating a final segmentation mask of one or more lesions based on a plurality of initial segmentation masks; as well as Means for outputting a final segmentation mask of the one or more lesions.

11. The apparatus of claim 10, wherein the means for generating a final segmentation mask of one or more lesions based on a plurality of initial segmentation masks comprises: means for combining a plurality of initial segmentation masks; as well as Means for generating a final segmentation mask based on the combined initial segmentation masks.

12. The apparatus of claim 11, wherein the means for combining a plurality of initial segmentation masks comprises: Means for combining intensity values ​​of corresponding voxels in a plurality of initial segmentation masks into corresponding scores.

13. The apparatus of claim 12, wherein the means for generating a final segmentation mask based on the combined initial segmentation masks comprises: Means for generating a final segmentation mask based on the scores using a machine learning based fusion network.

14. The apparatus according to claim 10, wherein: The plurality of machine learning based segmentation networks are trained by extracting training patches from training medical images depicting at least one lesion, the training patches being extracted from the training medical images based on a size of the at least one lesion.

15. A non-transitory computer readable medium storing computer program instructions which, when executed by a processor, cause the processor to perform operations comprising: receiving an input medical image patch depicting one or more lesions; segmenting the one or more lesions from the input medical image patch using a plurality of machine learning-based segmentation networks to respectively generate a plurality of initial segmentation masks, each of the plurality of machine learning-based segmentation networks being trained to segment lesions from patches having different field of view sizes; generating a final segmentation mask of one or more lesions based on the plurality of initial segmentation masks; as well as Output a final segmentation mask of one or more lesions.

16. The non-transitory computer readable medium of claim 15, wherein generating a final segmentation mask of one or more lesions based on a plurality of initial segmentation masks comprises: Combining multiple initial segmentation masks; as well as A final segmentation mask is generated based on the combined initial segmentation masks.

17. The non-transitory computer-readable medium of claim 15, wherein at least some of the plurality of machine learning based segmentation networks are trained using different loss functions.

18. The non-transitory computer-readable medium of claim 15, wherein at least some of the plurality of machine learning based segmentation networks are implemented using different network architectures.

19. The non-transitory computer-readable medium of claim 15, wherein at least some of the plurality of machine learning based segmentation networks are implemented using different hyperparameters.

20. The non-transitory computer-readable medium of claim 15, wherein the one or more lesions comprise one or more brain lesions located in a patient's brain.

Images