Segmenting tubular features

Abstract

In a method of segmenting a tubular feature in an image, a series of overlapping portions of the image are segmented using a trained model. The overlapping portions are positioned along a length of the tubular feature and combined to determine a segmentation of the tubular feature.

Description

Technical Field

[0001] The present disclosure relates to image segmentation. Particular, but not exclusive, embodiments herein relate to segmenting images comprising tubular structures, such as blood vessel structures in medical images. Background Art

[0002] Image segmentation involves dividing or partitioning an image into multiple segments, with each segment corresponding to a different portion of the image content. For example, in medical imaging, an image of an anatomical feature can be segmented into different parts, where each segment or part represents a different anatomical feature in the medical image. Segmentation enables the extraction / isolation of different features in the image for further analysis and / or modeling. Segmentation can be performed automatically or (partially) interactively and has a wide range of applications in medical image processing.

[0003] Neural networks, such as U-net or F-net (as described in references Ronneberger O et al. (2015) "U-Net: Convolutional Networks for Biomedical Image Segmentation"; MICCAI 2015 Lecture Notes in Computer Science, Vol. 9351, pp. 234-241; and Brosch T., Saalbach A. (2018) "Foveal fully convolutional nets for multi-organ segmentation" SPIE Vol. 10574, 105740U) have been shown to be useful for segmenting anatomical structures, and when compared with other segmentation methods, they are often among the best performing methods. Neural networks can also be used to segment vascular structures.

[0004] For an up-to-date overview of vascular segmentation techniques, see S. Moccia et al., “Blood vessels segmentation algorithms—Review of methods, datasets and evaluation metrics,” Computer Methods and Programs in Biomedicine 2018, 158, 71-91.

[0005] An object of the present disclosure is to provide an improved method for segmenting tubular structures such as blood vessels in an image. Summary of the Invention

[0006] As briefly outlined above, neural networks have shown promise in image segmentation, including the segmentation of medical images. However, segmenting tubular structures (e.g., such as blood vessels) using standard network architectures is challenging (especially in 3D) due to the relative size of the vessels (which are very small) and the background (which is much larger). Consequently, very large training datasets are required to achieve good results.

[0007] One solution is to use data augmentation techniques, such as generative adversarial networks, to generate simulated data that can be used to train neural networks. However, these can be difficult to generalize from 2D to 3D.

[0008] It is therefore an object of embodiments herein to provide improved segmentation techniques for segmenting tubular structures, such as vascular structures.

[0009] Thus, according to a first aspect, there is provided a method of segmenting a tubular feature in an image. The method comprises segmenting a series of overlapping portions of the image using a trained model, the overlapping portions being located along the length of the tubular feature, and combining the segmentations of the series of overlapping portions of the image to determine a segmentation of the tubular feature.

[0010] By breaking the segmentation into a series of smaller parts in this way, the segmentation task for the trained model is simplified. The amount of content or background information in each part is less (possibly significantly less) than the amount of information in the entire image. In other words, the tubular features in each part are likely to stand out more from the background than in the entire image, and therefore easier to segment. Based on a smaller dataset, the model can be trained faster and easier. The resulting trained model can be more reliable (due to the simplified task described above). Furthermore, this method can be more easily generalized to three dimensions than some prior art methods.

[0011] In some embodiments, segmenting a series of overlapping portions of an image may include: segmenting a first portion of the image, the first portion of the image including a first portion of the tubular feature; determining a first point included in the tubular feature based on segmenting the first portion of the image; determining a first vector that is parallel to the length of the tubular feature at the first point; and determining a second portion of the series of overlapping portions of the image based on the first point and the first vector.

[0012] In some embodiments, the second portion of the image may be determined such that the tubular feature is rotated in the second portion of the image based on the first vector.

[0013] In some embodiments, the tubular feature may be rotated so that the first vector is in a predetermined orientation in the second portion of the image.

[0014] This further simplifies the task of the trained model because blood vessels have roughly the same orientation across the image portion. Furthermore, this type of orientation alignment allows for 3D data augmentation by rotating the training data around a predetermined orientation. In 2D, augmentation can be achieved by mirroring the image portion using a predetermined orientation.

[0015] According to a second aspect, a system for segmenting tubular features in an image is provided. The system includes a memory and a processor, the memory including instruction data representing a set of instructions, the processor configured to communicate with the memory and execute the set of instructions. When executed by the processor, the set of instructions causes the processor to segment a series of overlapping portions of an image using a trained model, the overlapping portions being located along the length of the tubular feature, and to combine the segmentations of the series of overlapping portions of the image to determine a segmentation of the tubular feature.

[0016] According to a third aspect, there is provided a computer program product comprising a computer readable medium, the computer readable medium embodying computer readable code, the computer readable code being configured to, when executed by a suitable computer or processor, cause the computer or processor to perform the method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to better understand and more clearly show how the embodiments herein may be implemented, the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0018] Figure 1 Methods according to some embodiments of the present invention are shown;

[0019] Figure 2 is a schematic diagram illustrating how positions of a series of overlapping portions are determined according to some embodiments;

[0020] Figure 3 shows an example architecture of a neural network according to some embodiments;

[0021] Figure 4 is a block diagram illustrating a method for training a neural network according to some embodiments;

[0022] Figure 5 is a block diagram illustrating a method of segmenting tubular features in an image according to some embodiments;

[0023] Figure 6a illustrates an example output segmentation of a vessel structure segmented using an embodiment of the method described herein;

[0024] Figure 6b The figure shows the image segmented according to the prior art segmentation method. Figure 6a Segmentation of the same vascular structures in

[0025] Figure 7 A system according to some embodiments herein is illustrated. DETAILED DESCRIPTION

[0026] As mentioned above, using trained models, such as neural networks, for image segmentation, such as medical image segmentation, offers various advantages. However, segmenting tubular structures, such as blood vessels, typically requires very large training datasets because the area or volume of these structures is small relative to the background area or volume of the image. This makes training models for this task difficult, especially in the case of 3D images.

[0027] The embodiments herein aim to improve previous segmentation methods to better segment tubular structures such as blood vessels in medical images.

[0028] Figure 1 A computer-implemented method 100 according to some embodiments of the present invention is illustrated. Briefly, in a first block 102, the method 100 includes segmenting a series of overlapping portions of an image using a trained model, the overlapping portions being located along the length of a tubular feature. In a second block 104, the method 100 includes combining the segmentations of the series of overlapping portions of the image to determine a segmentation of the tubular feature.

[0029] In this way, the segmentation of long and / or thin tubular features can be broken down into a series of smaller segmentations of smaller portions of the image. This simplifies the task of the trained model because the ratio of tubular features to background in each image portion is increased, thereby reducing the amount of contextual information in each image portion. As will be described in more detail below, a smaller training dataset may be required to train a model to segment tubular features in a portion of an image compared to the entire image. The resulting model can be more reliable and easier to train.

[0030] In more detail, generally, the image may include any type of image acquired using any imaging modality. For example, in some embodiments, the image may include a medical image. Examples of medical images include, but are not limited to, computed tomography (CT) images (e.g., from a CT scan), such as a C-arm CT image, a spectral CT image, or a phase-contrast CT image, an x-ray image (e.g., from an x-ray scan), a magnetic resonance (MR) image (e.g., from an MR scan), an ultrasound (US) image (e.g., from an ultrasound scan), a fluoroscopic image, a nuclear medicine image, or any other medical image.

[0031] In some embodiments where the image comprises a medical image, the tubular feature may comprise, for example, a portion of a vascular structure, such as a blood vessel or artery. In other examples, the tubular feature may comprise a lung bronchial structure.

[0032] However, it should be understood that although the method herein is described with respect to medical imaging, the teachings herein are equally applicable to the segmentation of any other tubular structure in any other type of image. Various examples of segmentation tasks to which method 100 may be applied include, but are not limited to: segmentation of rivers or tributary systems in satellite images; segmentation of road networks in satellite images; and segmentation of tree branches in tree images.

[0033] Therefore, in general, it should be understood that the teachings of this article can be applied to the segmentation of any tubular feature. In some embodiments of this article, tubular can be understood as "tube-like", such as a slender or substantially cylindrical feature. Tubular features can include, for example, hollow tubes, such as the above-mentioned vascular and / or bronchial structures. However, tubular features are not necessarily hollow (for example, see the example of tributary systems in satellite images). Typically, tubular features can bend, bend or deform. In addition, tubular features can narrow in width along their length, for example, blood vessels in a vascular system narrow as they move away from the main veins and arteries. The embodiments of this article are also applicable to branched tubular features (for example, a tubular feature is divided into two or more sub-branches), such as tubular features observed in vascular and / or bronchial structures. It should also be understood that in a two-dimensional image, tubular features can include lines, such as curves.

[0034] In any embodiments described herein, an image can be a two-dimensional image, a three-dimensional image, or an image of any other dimension. An image can include a plurality (or a group) of image elements. In embodiments where the image is a two-dimensional image, the image elements are pixels. In embodiments where the image is a three-dimensional image, the image elements are voxels.

[0035] Returning to method 100, in block 102, overlapping portions of the image may be sequentially segmented using a trained model. The trained model may comprise a trained neural network, such as a trained F-net or a trained U-net. Those of skill in the art will be familiar with neural networks, but in brief, a neural network is a supervised machine learning model that can be trained to predict a desired output given input data. A neural network is trained by providing training data, which includes example input data and the desired corresponding "correct" or true value results. A neural network comprises multiple layers of neurons, each representing a mathematical operation applied to the input data. The output of each layer in the neural network is fed to the next layer to produce an output. For each piece of training data, the weights associated with the neurons are adjusted until an optimal weight is found that produces a prediction that reflects the corresponding true value for the training example.

[0036] In embodiments where the trained model comprises a neural network, the neural network may comprise a convolutional neural network, such as a U-net or F-net, suitable for classifying pixels / voxels into vessels and background. Such networks may be supplemented with additional output channels to estimate vessel orientation and / or vessel diameter. The network structure may reflect the directional information used to construct the sub-images and may, for example, be convolutional in one direction and fully connected in an orthogonal direction. Furthermore, the network structure may be combined with a recurrent network structure to address subsequent segmentation along the vessel's path.

[0037] In embodiments where the trained model comprises a neural network, the model may be trained using a number j = 1, ..., M images and corresponding ground truth segmented tubular features (e.g., binary labeled as tubular or non-tubular). The segmentation is used to generate Nj points xj,i for each training image and the corresponding direction dj,i of the tubular feature at each point xj,i. The points and directions may be used to generate sub-images or portions of images that may be used to train the neural network. Reference will be made to Figure 4 This is described in more detail.

[0038] Furthermore, data augmentation can be applied during training. For example, slightly shifted sub-images, slightly rotated sub-images around their centers, or sub-volumes rotated by a large angle around their centers using an orientation corresponding to the sub-images can be used for data augmentation.

[0039] It should be understood that although neural networks are described in the examples herein, these principles can be equally applied to any machine learning model that can be trained to take an image as input and predict a segmentation of the image as output. Other trained models that can be used in the embodiments herein include, but are not limited to, training a random forest model. Further details on how to train the model will be discussed below.

[0040] Typically, a trained model takes an image (which, as described herein, may be a portion of a larger image) as input and produces a segmentation of the image as output. The segmentation may provide an indication of the likely locations of tubular features in the input image.

[0041] In some embodiments, the trained model takes as input an image (which, as described herein, may be a portion of a larger image) and produces as output a probability map comprising probability values ​​describing the likelihood that each image element in the input image comprises a partial tubular feature.

[0042] Alternatively, in some embodiments, the trained model may take as input an image (which may be part of a larger image as described herein) and produce as output a map that classifies each image element as part of a tubular feature or part of a non-tubular feature (e.g., vessel or non-vessel).

[0043] In addition to segmentation, the trained model can provide further output estimates of other features of the tubular features in the image. For example, the trained model can be trained to output, for each image element (e.g., pixel / voxel), a vector parallel to the length of the tubular feature at that image element (e.g., a direction vector describing the direction or centerline of the blood vessel at each point). Other features, such as the diameter or curvature of the tubular feature at each image element, can also be predicted.

[0044] In some embodiments, segmenting a series of overlapping portions of an image using the trained model 102 may include a recursive method using a trained model such as a neural network to segment tubular features such as blood vessels. Taking a blood vessel as an example, the recursive method may use a point x on or near the centerline of the blood vessel. i and the approximate centerline direction d i The centerline direction includes the direction parallel to or along the length of the tubular feature at the first point. This information is used to generate the centerline direction relative to point x. i A reformatted portion or sub-image is shifted in the direction of the vessel (e.g., along the vessel). A trained model (e.g., a neural network or other learned filter) is then applied to segment the vessels in the volume. Based on the segmentation of the reformatted portion of the image, new points are selected and new centerline directions are determined at the new points. This process is then repeated to produce a series of overlapping segmented portions of the image.

[0045] This is Figure 2 is shown in Figure 2 The diagram illustrates how two overlapping portions 204 and 206 of a series of overlapping portions are determined or located in the image 200 of the blood vessel 202. In this example, the blood vessel segmentation is performed at a point x near the centerline of the blood vessel. i and point x i The approximate centerline direction d i (‖d i = 1) as input. As described above, the centerline direction comprises a direction parallel to (eg, along) the length of the tubular feature at the first point.

[0046] Using this input, the image 200 is generated with image element y i+1 =x i +ad i A portion 204 (e.g., a sub-image) centered on the image is then segmented. An "offset" distance along the vessel between each portion is determined. The portion (or sub-volume) can have a fixed size and resolution. The trained model is then applied to the portion 204 to segment the portion 204.

[0047] In this embodiment, the trained model includes a trained neural network that outputs a probability map describing the relative likelihood that each image element in portion 204 contains a vessel (e.g., a "vascularity" value). The neural network can also calculate a vector for each image element that describes the centerline vector as described above. The probability map (possibly along with additional orientation information) is then converted back to the coordinate system of the 2D image.

[0048] Based on the probability map, determine the new point x that may correspond to the blood vessel i+1 And determine the point x i+1 The corresponding center line direction d i+1 For example, point x i+1 The image element with the highest probability of containing a blood vessel in the segmentation of portion 204 can be selected. This process is repeated to i+1 and vector d i+1 The next portion in sequence is obtained, namely portion 206. In this manner, with each iteration, a new portion of the vessel is segmented without the need to segment the entire image 200.

[0049] To describe the above in more general terms, in some embodiments, step 102 may include iteratively performing the following blocks: i) segmenting a first portion of the image (e.g., portion 204), the first portion of the image including a first portion of the tubular feature, ii) determining a first point (e.g., x) included in the tubular feature based on the segmentation of the first portion of the image. i+1 ), iii) determining a first vector (e.g., d i+1 ), and iv) based on the first point (e.g., x i+1 ) and the first vector (e.g., d i+1 ) determines a second portion (e.g., portion 206) of a series of overlapping portions of the image.

[0050] In some embodiments, in block i), the first portion 204 may be provided as an input parameter, eg the first portion may be manually determined, eg by a user. The user may provide the position of the first portion as input to the method.

[0051] In other embodiments, the input parameters of the method may include the initial starting point x i and the initial starting direction d i , according to the initial starting point x i and the initial starting direction d i , can be calculated according to the above equation y i+1 =x i +ad i To determine the position of the first part 204. Initial starting point x i and the initial starting direction d iIt may be input by a user (eg, manually input), or determined in an automatic process, for example, based on an initial (eg, rough) segmentation of the image 200 .

[0052] Now turning to box ii), the first point (e.g., x i+1 For example, in embodiments where a probability map is generated for the segmentation of the first portion (e.g., embodiments where a probability value is determined for each image element in the first portion of the image to indicate the probability that the image element is included in the tubular feature), determining the first point included in the tubular feature based on the segmentation of the first portion of the image may include determining the first point based on image elements in the segmentation of the first portion of the image that have a probability value above a predetermined threshold. The predetermined threshold may include a threshold (probability) corresponding to a high likelihood that the image element contains a blood vessel. In other words, the first point may be selected from the image elements indicated as being most likely to form part of the tubular feature in the segmentation of the first portion.

[0053] The predetermined threshold value can be set to a constant value, or can be set / changed based on the properties of the tubular features in different parts of the image. For example, the predetermined threshold value can decrease as the size of the tubular feature decreases to reflect that the tubular feature may appear blurrier in the image as its size decreases. As described above, in some embodiments, the trained model can be trained to output an estimate of the diameter of the tubular feature at each image element of the image, and thus the size of the predetermined threshold value can be set / changed based on the estimated diameter of the tubular feature at each point.

[0054] In other embodiments, the first point may be determined as an image element having a highest probability of including a blood vessel based on segmenting the first portion of the image (eg, a pixel or voxel in the image portion that is most likely to be part of a blood vessel).

[0055] Going to box iii), the first vector (e.g., Figure 2 The d shown i+1 ; also referred to herein as the centerline direction or direction vector) parallel to the starting point x i In other words, the first vector is along the major axis of the tubular feature and therefore represents the distance at point x. i+1 It should be understood that the vector d i+1 The same can be reversed to determine the direction along the tubular feature 202. Figure 2 A series of tubular features in opposite directions as shown. Typically, d i+1 The direction can be chosen so that (d i ,d i+1 )=cosα>=0(where α is d i and d i+1In such an example, the tracking direction may then be defined by the initial values ​​(eg, the first point and the first direction).

[0056] In some embodiments, block iii) may comprise determining a first vector d i+1 , for example, by analyzing the probability values ​​in the segmentation of the first part using a Hessian filter and selecting i Most parallel direction ±d H i+1 to confirm.

[0057] In other embodiments, block iii) may comprise using a neural network to estimate the orientation d of each image element in the portion. i+1 For example, in some embodiments, a trained model may be trained to take as input a portion of an image and output a segmentation of the portion and, for each image element in the portion, a vector indicating an estimated direction of the tubular feature at that image element.

[0058] Go to box iv), based on the first point (e.g., x i+1 ) and the first vector (e.g., d i+1 ) determining a second portion (e.g., portion 206) of the series of overlapping portions of the image may include determining a center of the second portion of the image according to the following equation:

[0059] y i+2 =x i+1 +ad i+1 ;

[0060] where y i+2 Contains the coordinates of the center of the second part of the image, x i+1 The coordinates of the first point, a, determine the distance of "offset" between the segments along the vessel, d i+1 Include the first vector.

[0061] As described above, a represents the distance, such as an offset or shift, between the first and second portions 204, 206 of the image and can be provided as a configuration or input parameter. In some embodiments, parameter a and the desired width and / or height of each portion can be provided as input parameters to method 100.

[0062] In some embodiments, a is a constant. In other embodiments, a may vary, for example, depending on a property of the tubular feature such as diameter or curvature (an estimate of which may be obtained as an additional output of the segmentation performed by the trained model as described above).

[0063] The size (number of pixels / voxels in each dimension) and resolution (geometric dimensions of pixels / voxels) of each portion of the image can be set as constant input parameters, or can also be varied. For example, when tracking a tubular structure such as a blood vessel along the direction of blood flow, the diameter of the vessel will generally decrease. Thus, the size (number of pixels / voxels in each dimension) and resolution (geometric dimensions of pixels / voxels) of each portion of the image can be varied based on, for example, the diameter or curvature (an estimate of which can be obtained as an output of the trained model as described above).

[0064] For example, in one embodiment, as a tubular feature narrows, the size (number of pixels / voxels in each dimension) can be maintained along the tubular feature while increasing the resolution (i.e., reducing the geometric size of the pixels / voxels). This provides a more computationally efficient approach than always choosing a higher resolution.

[0065] Turning now to other embodiments, in some embodiments, the second portion 206 may be determined such that the tubular features in the second portion of the image are based on the first vector d i+1 For example, a tubular feature in each section can be rotated relative to the original image.

[0066] In some embodiments, the first vector (eg, d i+1 ) may define the coordinate axes of the (reformatted) second image portion 206. For example, the second portion 206 may be rotated or selected from the image 200 so that the first vector d i+1 parallel to the edge of the second portion 206. By rotating or selecting the image portions in this manner, the orientation of the tubular features in each image portion is changed so that the tubular features are in approximately the same orientation in each image portion. For example, if the second portion is selected so that the first vector d i+1 If the x-axis of the second portion is parallel to the x-axis of the second portion, then the tubular features will be roughly parallel to the edge of the second image portion. This has the technical effect of further reducing the complexity of the problem of training the model, because the orientation and appearance of the tubular features are more regular in each image portion. As a result, the model can be trained to higher accuracy based on less training data.

[0067] It should be understood that the specific orientation of the tubular feature in the image is not important. Instead, the advantages described above can be achieved if the portions of the image are selected or rotated so that the tubular feature is in substantially the same orientation (horizontally, diagonally, or vertically across the image portion) in each image portion. Thus, in some embodiments, the tubular feature is rotated so that the first vector is in a predetermined orientation in the second portion of the image.

[0068] More generally, in some embodiments, the tubular feature is rotated so that the tubular feature is in a predetermined orientation in the second portion of the image. For example, some of a series of overlapping portions may be selected or rotated so that the tubular feature therein is (substantially) in a specific orientation.

[0069] In some embodiments, the tubular feature may be rotated so that the tubular feature is within, for example, + / - 45 degrees of a particular direction or orientation in each portion of the image.

[0070] It should also be understood that due to the curved nature of many tubular features considered herein (e.g., vascular or bronchial structures), the tubular features may not be perfectly aligned in a particular direction relative to the edges of the corresponding portion of the image (e.g., Figure 2 In the example above, although the direction vector d i+1 is aligned with the edge of image portion 206, but due to the tubular feature at the point x i+1 Starting direction d i+1 The curvature of the image is such that the tubular feature is at an angle to the edge. However, rotating / selecting the portion in this manner still reduces the number of degrees of freedom in the orientation of the tubular feature within the image portion compared to the case where the tubular feature is randomly oriented, and thus reduces the complexity of the problem of training the model, increasing its accuracy and reducing the amount of training data required. This is particularly true in three dimensions, where the aforementioned rotation can ensure that the tubular feature enters the volume of each three-dimensional portion at a similar location in each image portion.

[0071] In embodiments where portions of an image are rotated, a trained model can be trained based on training data comprising a) a plurality of example images including portions of tubular features, wherein each example image is rotated based on a predetermined orientation, and b) a corresponding ground truth segmentation for each example image. In other words, the image portions included in the training data can be rotated such that the tubular features in the training data are (substantially) in a particular orientation. As described above, in embodiments where image portions are rotated such that at least a portion of the tubular structure is in a common orientation, less training data is required because the variation in the position and orientation of the tubular features is reduced.

[0072] Turning now to other embodiments, it will be appreciated that blocks i)-iv) may be repeated in an iterative manner along the length of the tubular feature. For example, in some embodiments, the method may include determining a second point included in the tubular feature based on segmentation of the second portion of the image, determining a second vector parallel to the length of the tubular feature at the second point, and determining a third portion in a series of overlapping portions of the image based on the second point and the second vector.

[0073] When the tubular feature ends, for example, if the segmentation of a portion indicates that there are no tubular features in the portion, block 102 of method 100 can terminate. For example, the method can include stopping segmenting other portions of the image if the segmentation of a portion in a series of overlapping portions includes probability values ​​that are all below a predetermined threshold (e.g., if the probability value is below a threshold for selecting the next point on the tubular feature in a particular image portion).

[0074] The predetermined threshold value can be set to a constant value, or can be set / changed based on the properties of the tubular features in different parts of the image. For example, the predetermined threshold value can decrease as the size of the tubular feature decreases to reflect that the tubular feature may appear blurrier in the image as its size decreases. As described above, in some embodiments, the trained model can be trained to output an estimate of the diameter of the tubular feature at each image element of the image, and thus the size of the predetermined threshold value can be set / changed based on the estimated diameter of the tubular feature at each point.

[0075] As also described above, the tubular feature may branch, such that the tubular feature splits into two or more branches or separate tubular features. Identifying more than one point that is a predetermined distance from a known point in the tubular feature and has a probability value above a threshold probability in the segmentation may indicate different branches of the bifurcation.

[0076] Thus, more generally, in some embodiments, the method may include determining a third point in the segmentation of the first portion of the image, the third point having a probability value above a predetermined threshold and being at a distance from the first point greater than a predetermined distance, and determining a bifurcation in the tubular feature based on the positions of the first point and the third point.

[0077] Thus, the predetermined distance is the minimum separation between two points above a predetermined threshold (indicating that both points may form part of a tubular feature), which is required for the method to mark the points as being in separate branches / generate a second series of overlapping images.

[0078] The predetermined distance can also be set and / or varied based on properties of the tubular features in different parts of the image. For example, the predetermined distance can decrease as the size of the tubular feature decreases. In embodiments where the trained model outputs a diameter estimate of the tubular feature at each image element of the image, the size of the predetermined distance can be set / varied based on the estimated diameter of the tubular feature at each point.

[0079] If two (or more) image elements are determined in this way, the method may comprise stepping along two branches (eg overlapping portions of the second or subsequent series of images may be processed in parallel).

[0080] Now back Figure 1In block 104, the method includes combining the segmentations of the series of overlapping portions of the image to determine a segmentation of the tubular feature. The segmentations of the series of overlapping portions of the image may be averaged or otherwise superimposed on each other to form a combined segmentation of the complete tubular feature.

[0081] In some embodiments, block 104 may include averaging the image element values ​​in the segmentations of a series of overlapping portions of the image in the overlapping segmented regions. For example, in embodiments where the segmentations produced by the trained model include probability maps, if two or more probabilities are determined for a particular image element (e.g., if the element is included in the overlapping region between two image portions), the probabilities may be averaged (e.g., the mean, mode, or median). In some embodiments, a threshold majority voting process may be used to determine the probability of a particular image element.

[0082] Combining the segmentations in overlapping regions, for example by averaging two or more estimates of the probability that an image element includes a vessel, has the technical effect of improving the estimate of the probability that the image element includes a vessel and, therefore, improving the segmentation. The segmentation process is more efficient overall because only regions of the larger image that are known / determined to contain the tubular feature in question are segmented.

[0083] In some embodiments, a weighted mask may be applied when combining a series of segmentations in block 104. Applying a weighted mask may weight image elements closer to the center of the image portion more than image elements further away from the center of the image portion.

[0084] Turning now to other embodiments, in a particular embodiment, the tubular features include blood vessels in a cardiac computed tomography angiography (CCTA) image 300. In this embodiment, the model includes Figure 3 The neural network shown in Figure 3 In the example of [ 3 ], the architecture used is F-Net. F-Net1 is similar to U-Net2 in that it is able to view different resolutions in order to combine local and global image features. The feature extraction path in F-Net includes three CBR (Convolution, Batch Normalization, ReLu) blocks 302. The feature integration path in F-Net includes CBRU (Convolution, Batch Normalization, ReLu, Upsampling) blocks 304 (in addition to the highest resolution level). In this embodiment, the number of resolution levels used (as indicated by the row including the CBR blocks) is 4. The neural network outputs a segmentation 306 of the image 300.

[0085] A neural network was trained to segment vessels in CCTA images using a training dataset of 39 images. Thirty of these images were used to train the neural network and were divided into a series of rotational segments positioned along the length of tubular features, such as those described above. Nine of these images were used as test / validation images to test the accuracy of the trained neural network. Using data augmentation (e.g., automated methods for generating simulated test data), the number of training samples (e.g., training image segments) was increased to as many as 26,500 samples.

[0086] exist Figure 4 An example training strategy is shown in the flowchart of . The part center and centerline direction in boxes 402 and 404 can be used to determine the sample part in box 406 (in a manner similar to how the first part is determined as outlined above with respect to box 102 of method 100). In box 408, the sample part is reformatted or rotated / selected so that the direction vector is in a specific orientation in the part. This defines a new coordinate system (u, v, w) for the part relative to the coordinate system (x, y, z) of the training image. In box 410, the enhanced part is used to train the model. In box 412, the model generates an output probability map in the coordinate system (u, v, w) and is used to update the gradient 414 (according to a gradient descent process) by comparing the probability map generated by the model with the true value probability map of the image part. In box 416, the model parameters are updated. Typically, the output of all image parts (or a subset thereof) can be accumulated, and the gradient update can be performed based on the accumulated value.

[0087] like Figure 5 Using the trained model, the coordinates of the center position of the portion described in block 502 are provided (e.g., as Figure 2 The y shown i+1 ) and the direction in box 504 (e.g., as Figure 2 The d shown i ) as input, in block 506, a first portion of the image is determined based on the input. As described above, the first portion of the image has a coordinate system (u, v, w), and one of the axes is aligned with the vector d i At block 508, the first portion is provided as input to the trained model. The trained model has been described above with reference to Figure 4 The trained model segments the first portion and outputs a probability map at block 510. The output probability map is in the same coordinate system (u, v, w) as the first portion. At block 512, the probability map is converted (e.g., rotated and / or shifted) to the coordinate system (x, y, z) of the (primary or entire) image.

[0088] Based on the segmentation of the first portion, a first point included in the tubular feature is determined based on the segmentation of the first portion of the image. At block 514, a first vector parallel to the length of the tubular feature at the first point is also determined. This is as described above with respect to Figure 1 At block 516, the first point and the first vector are used to determine the center coordinates of the next portion of the series of portions of the image (e.g., using the equation y i+2 =x i+1 +ad i+1 , as described above with respect to block 102 of method 100 .) The location of the next portion in the series of portions is then sampled or determined in block 506 using the center coordinates 514 of the next portion and the determined vector 516 .

[0089] Blocks 506, 508, 510, 512, 514, 516 are repeated in this manner to produce a series of overlapping segmentations positioned along the length of the vessel. These segmentations are merged or combined (e.g., with the previous segmentation 520 from the previous cycle of blocks 506, 508, 510, 512, 514, 516) to form a final segmentation of the entire vessel 522, as described above with respect to block 104 of method 100.

[0090] Figure 6a Shown using the above Figure 3-5 The process described herein is an example output segmentation 600 of an image of a branch vessel 602. In this example, the starting point x i and d i Generated from ground truth data (e.g., user-entered starting position and direction vectors). Figure 6b The segmentation of the same vessel using the prior art segmentation method is shown 604. In this example, the dice scores of the two methods are 0.81 and 0.7 respectively.

[0091] Now turning to other embodiments, such as Figure 7 As shown, in some embodiments, a system 700 for segmenting tubular features in an image is provided. The system can form part of a dedicated device (e.g., a dedicated medical device). Alternatively, the system can form part of a computer system (e.g., a laptop, desktop computer, other device, or computing cloud). The system includes a memory 704 and a processor 702. The memory 704 includes instruction data representing a set of instructions. The processor 702 is configured to communicate with the memory and execute the set of instructions. Generally, when executed by the processor, the set of instructions can cause the processor to perform any of the embodiments of methods 100, 400, or 500 described above.

[0092] More specifically, when executed by the processor, the set of instructions causes the processor to segment a series of overlapping portions of an image using the trained model, wherein the overlapping portions are located along the length of the tubular feature, and combine the segmentations of the series of overlapping portions of the image to determine a segmentation of the tubular feature.

[0093] Segmenting a series of overlapping portions of an image has been described in detail above with respect to block 102. The method 100 and the details therein should be understood to be equally applicable to the configuration of the processor 702. Segmenting a series of overlapping portions of a combined image has been described above with respect to block 104 of the method 100, and the details therein should be understood to be equally applicable to the configuration of the processor 702.

[0094] The processor 702 may include one or more processors, processing units, multi-core processors, or modules configured or programmed to control the system 700 in the manner described herein. In particular embodiments, the processor 702 may include multiple software and / or hardware modules, each of which is configured to perform or be used to perform a single or multiple steps of the methods described herein.

[0095] The memory 704 is configured to store program code that can be executed by the processor 102 to perform the methods described herein. Alternatively or additionally, one or more memories 704 can be located outside the system 700 (i.e., separated from or away from the system 700). For example, one or more memories 704 can be part of another device. The memory 704 can be used to store images, information, data, signals, and measurements obtained or generated by the processor 702 of the device 700 or from any interface, memory, or device outside the system 700. The processor 702 can be configured to control the memory 704 to store images, information, data, signals, and measurements.

[0096] It should be understood that the system 700 may include other components, such as one or more user interfaces, such as a screen or display for displaying images and / or segmentation of tubular features. The system may also include a user input device, such as a keyboard, mouse, or other input device, that enables a user to interact with the system, such as to provide initial input parameters to be used in the methods described herein.

[0097] It should be understood that Figure 7 Only components necessary to illustrate this aspect of the invention are shown, and in actual implementations, system 700 may include additional components than those shown. For example, system 700 may include a battery or other power source for powering system 700, or a means for connecting system 700 to a mains power source.

[0098] In another embodiment, a computer program product is provided comprising a computer-readable medium, the computer-readable medium embodying computer-readable code, the computer-readable code being configured to, when executed by a suitable computer or processor, cause the computer or processor to perform one or more of the methods described herein. It will therefore be understood that the present disclosure also applies to computer programs, in particular computer programs on or in a carrier, suitable for putting the embodiments into practice. The program may be in the form of source code, object code, a code intermediate source and object code, such as a partially compiled form, or any other form suitable for implementing the methods according to the embodiments described herein.

[0099] It should also be understood that such a program can have many different architectural designs. For example, the program code that implements the functionality of the method or system can be subdivided into one or more subroutines. Many different ways of distributing functionality between these subroutines are apparent to those skilled in the art. The subroutines can be stored together in an executable file to form a self-contained program. Such an executable file can include computer-executable instructions, such as processor instructions and / or interpreter instructions (e.g., Java interpreter instructions). Alternatively, one or more or all subroutines can be stored in at least one external library file and, for example, statically or dynamically linked to a main program at runtime. The main program includes at least one call to at least one subroutine. The subroutines can also include function calls between each other.

[0100] The carrier of a computer program may be any entity or device capable of carrying the program. For example, the carrier may include a data storage device, such as a ROM, such as a CD ROM or a semiconductor ROM, or a magnetic recording medium, such as a hard disk. In addition, the carrier may be a transmissible carrier, such as an electrical or optical signal, which may be transmitted via an electrical or optical cable or by radio or other means. When the program is contained in such a signal, the carrier may be constituted by such a cable or other device or apparatus. Alternatively, the carrier may be an integrated circuit embedded with the program, the integrated circuit being suitable for executing the relevant method or for executing the relevant method.

[0101] Variations of the disclosed embodiments may be understood and implemented by those skilled in the art from a study of the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may perform the functions of several items listed in the claims. The reference to certain measures in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. The computer program may be stored / distributed on a suitable medium, such as an optical storage medium or solid-state medium provided with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A method for segmenting tubular features in an image, the method comprising: segmenting the image into a series of overlapping portions using the trained model, the overlapping portions being located along a length of the tubular feature; as well as combining the segmentations of the series of overlapping portions of the image to determine a segmentation of the tubular feature, wherein segmenting the series of overlapping portions of the image comprises: segmenting a first portion of the image, the first portion of the image including a first portion of the tubular feature; determining a first point included in the tubular feature based on the segmentation of the first portion of the image; determining a first vector parallel to the length of the tubular feature at the first point; and A second portion of the series of overlapping portions of the image is determined based on the first point and the first vector, wherein the second portion of the image is determined such that the tubular feature is rotated in the second portion of the image based on the first vector, wherein the tubular feature is rotated such that the first vector is in a predetermined orientation in the second portion of the image.

2. The method of claim 1 , wherein the center of the second portion of the image is determined according to the following formula: y i+1 =x i +a d i ; where y i+1 The coordinates of the center of the second portion of the image, x i includes the coordinates of the first point, a represents the magnitude of the offset between the first point and the center of the second portion, and d i Including the first vector.

3. The method of claim 1 , wherein the trained model has been trained on training data, the training data comprising: i) a plurality of example images comprising portions of the plurality of tubular features, wherein each example image is rotated based on the predetermined orientation, and ii) a corresponding ground truth segmentation for each example image.

4. The method according to any one of claims 1 to 3, further comprising: determining a second point included in the tubular feature based on segmenting the second portion of the image; determining a second vector parallel to the length of the tubular feature at the second point; as well as A third portion of the series of overlapping portions of the image is determined based on the second point and the second vector.

5. The method according to any one of claims 1 to 3, wherein segmenting the first portion of the series of overlapping portions of the image comprises: determining a probability value for each image element in the first portion of the image, the probability value indicating a probability of the image element being included in the tubular feature; and Wherein determining a first point included in the tubular feature based on the segmentation of the first portion of the image comprises: The first point is determined based on image elements in the segmentation of the first portion of the image having a probability value above a predetermined threshold.

6. The method according to claim 5, further comprising: determining a third point in the segmentation of the first portion of the image, the third point having a probability value above the predetermined threshold and being located at a distance from the first point greater than a predetermined distance; as well as Based on the positions of the first point and the third point, a bifurcation in the tubular feature is determined.

7. The method according to claim 5, further comprising: If the segmentation of a portion of the series of overlapping portions includes probability values ​​that are all below the predetermined threshold, then segmenting other portions of the image is stopped.

8. The method of any one of claims 1 to 3, wherein combining the segmentations of the series of overlapping portions of the image to determine a segmentation of the tubular feature comprises: In regions of overlapping segmentations, the image element values ​​in the segmentations of the series of overlapping portions of the image are averaged.

9. The method of any one of claims 1 to 3, wherein the image comprises a medical image and the tubular features comprise blood vessels or bronchial structures.

10. The method of any one of claims 1 to 3, wherein the trained model comprises a trained neural network model or a trained random forest model.

11. A system for segmenting tubular features in an image, the system comprising: a memory including instruction data representing a set of instructions; as well as a processor configured to communicate with the memory and to execute the set of instructions, wherein the set of instructions, when executed by the processor, causes the processor to: segmenting the image into a series of overlapping portions using the trained model, the overlapping portions being located along a length of the tubular feature; as well as combining the segmentations of the series of overlapping portions of the image to determine a segmentation of the tubular feature, wherein the step of segmenting the series of overlapping portions of the image using the trained model comprises: segmenting a first portion of the image, the first portion of the image including a first portion of the tubular feature; determining a first point included in the tubular feature based on the segmentation of the first portion of the image; determining a first vector parallel to the length of the tubular feature at the first point; and A second portion of the series of overlapping portions of the image is determined based on the first point and the first vector, wherein the second portion of the image is determined such that the tubular feature is rotated in the second portion of the image based on the first vector, wherein the tubular feature is rotated such that the first vector is in a predetermined orientation in the second portion of the image.

12. A computer program product comprising a computer-readable medium, the computer-readable medium embodying computer-readable code, the computer-readable code being configured such that when executed by a suitable computer or processor, the computer or processor is caused to perform the method according to any one of claims 1 to 10.

Images