Automated vessel segmentation from image sequences

Abstract

Machine learning image segmentation model to: receive image frames as input to a 3D encoder; provide the 3D encoder’s output as input to temporal extraction nodes; generate, by the temporal extraction nodes, 2D temporal output based on extracted temporal information; provide the 2D temporal output data to a 2D convolution node and a 2D decoder; combine, at the 2D decoder, the 2D temporal output and the output of the 2D convolution node to generate a predicted vessel tree segmentation; use the predicted segmentation to generate output representing the segmented vessel. Also disclosed: providing the 3D encoder’s output as input to a 3D convolution node and a temporal extraction node; providing the 3D convolution node’s output as input to a temporal extraction node; extracting temporal information from the output of the 3D encoder or the 3D convolution node; generating 2D temporal outputs which are combined and used to segment the vessel. Nodes may be arranged in a layered pyramidal network. An energy loss function associated with a vessel boundary velocity of the segmentation may be minimised using gradient backpropagation. May be used for invasive coronary angiography (ICA) to segment coronary blood vessels and diagnose coronary artery stenosis or coronary artery disease.

Description

TECHNICAL FIELD

[0001] Examples of the present disclosure relate to automated vessel segmentation. More particularly, examples of the present disclose relate to the use of machine learning image segmentation models for segmentation of a vessel tree from a series of image frames, such as x-ray angiographic image sequences. Further examples of the disclosure relate to training the machine learning model, and methods of use thereof. BACKGROUND

[0002] Invasive coronary angiography (ICA) is an invasive diagnostic procedure providing information about blood flow of the coronary vessels. During ICA, a catheter is inserted and guided through blood vessels of the body as a series of x-ray images are taken, acquiring a number of angiograms. A contrast agent (“contrast”) is generally applied during the procedure to highlight the blood vessels. ICA is widely used in clinical practice for analysis of coronary artery disease (CAD), and serves as the gold standard for diagnosing coronary artery stenosis.

[0003] Segmentation of coronary vessels enables clear visualisation and calculation of various quantitative metrics of the vessels, such as fractional flow reserve measurement, FFR, relating to achievable blood flow, for evaluation of blockage severity. Accurate segmentation of coronary vessels in ICA is important for evaluation. Machine learning methods, such as neural networks, have become widely utilised in most medical image analysis applications, including segmentation.

[0004] However, when obtaining the images in ICA, the acquired vascular structures - i.e. representing the coronary vessels - may overlap with various background structures. These images may also be affected by body motion and unevenly distributed contrast, which may result in segmentation artefact, such as misaligned boundaries and disconnected vessels. This reduces the accuracy of such representative images for performing evaluation.

[0005] Manual identification e.g. the gold standard, looks to address and / or mitigate the abovedetailed problems, by manual identification of vessels between the sequence of image frames. However, such methods are increasingly time-inefficient, and may still be prone to error.

[0006] In addition, coronary vessel segmentation presents a number of challenges, in view of the contrast distribution variance, and large variations in the sizes of blood vessels of the coronary network. Consequently, current methods, models or algorithms looking to automate image segmentation may fail to distinguish thin vessels and / or may obtain inaccurate or misaligned vessel boundaries, which may render the obtained images less effective for identification and diagnosis, such as relating to blockage severity. BRIEF SUMMARY

[0007] The present invention looks to address and / or mitigate at least some of the problems described above. Aspects of the present invention and disclosure relate to examples for an automated approach for accurately and effectively delineating vessels (e.g. coronary vessels) from a sequence of images.

[0008] To address and / or mitigate the problems identified above, aspects of the present disclosure relate to a machine learning segmentation model.

[0009] According to an aspect of the invention there is provided machine learning image segmentation model for automatically identifying and segmenting structural features of a vessel tree from image frames (e.g. a series of contrast x-ray image frames). The model comprises a 3D encoder and 2D decoder, the 3D encoder and 2D decoder connected by at least one interlinked convolution node and a plurality of temporal extraction nodes therebetween, and wherein the model is configured to identify and segment structural features of a vessel tree from the plurality of image frames, by the model being configured to: receive a plurality of image frames as an input to the 3D encoder; provide an output of the 3D encoder as an input to the plurality of temporal extraction nodes to extract temporal information; generate, by the plurality of temporal extraction nodes, a 2D temporal output based on the extracted temporal information; provide the generated 2D temporal output to the at least one interlinked 2D convolution node and 2D decoder; generate, at the 2D decoder, a combined temporal output based on an output of the at least one interlinked 2D convolution node and at least one temporal extraction node, wherein the combined temporal output represents a predicted segmentation of the vessel; and generate an output representative of the segmented structural features of the vessel tree based on the predicted segmentation.

[0010] In further aspects, the model may further comprise that, to extract temporal information, each of the plurality of temporal extraction nodes of the model are configured to: apply 3D convolution to the received 3D input (e.g. kernel size T x 1 x 1); and dimensionally compress the 3D input so as to produce a temporally compressed 2D feature map representing the extracted temporal information.

[0011] In further aspects, each of the at least one interlinked 2D convolution node may comprise a plurality of interlinked 2D convolution nodes connecting the 3D encoder and 2D decoder, and the plurality of interlinked 2D convolution nodes connecting the 3D encoder and 2D encoder comprise one or more skip connections therebetween.

[0012] In further aspects, the model may further comprise that the 3D encoder, 2D decoder, and the at least one interlinked 2D convolution node are arranged as nodes in a layered pyramidal network; and the plurality of temporal extraction nodes may be arranged between the respective 3D and 2D nodes of network.

[0013] In further aspects, the model may be further configured to: minimise an energy loss associated with the predicted segmentation of the vessel based on determining a difference between the predicted segmentation and the labelled input, wherein the determined divergence is associated with an energy loss function representing an elastic energy of a system comprising the predicted segmentation and the labelled input.

[0014] In further aspects, the model may be further configured to: determine the difference between a velocity of a vessel boundary of the predicted segmentation and a vessel boundary of the labelled input; and minimise the energy loss function based on the determined difference.

[0015] In further aspects, thee 3D encoder, 2D decoder, and the at least one interlinked 2D convolution node are arranged as nodes in a layered pyramidal network, and wherein, to minimise the energy loss, the model is further configured to: combine the predicted segmentation with an output of at least one interlinked 2D convolutional node arranged in a top layer of the network to output a refined predicted segmentation; and determine the difference between a velocity of a vessel boundary of the refined predicted segmentation and the vessel boundary of the labelled input; and minimise the energy loss function based on the determined difference.

[0016] In further aspects, wherein the 3D encoder, 2D decoder, and the at least one interlinked 2D convolution node are arranged as nodes in a layered pyramidal network, and the model is further configured to: backpropagate a determined gradient of the energy loss function at each of the nodes in the one or more layers of the network to update an assigned weighting of each respective node, wherein the gradient of the energy loss function is associated with a velocity of a vessel boundary of the predicted segmentation.

[0017] According to a further aspect of the invention, there is provided a machine learning image segmentation model for automatically identifying and segmenting structural features of a vessel tree from image frames (e.g. a series of contrast x-ray image frames), the model comprising a 3D encoder and 2D decoder, the 3D encoder and 2D decoder connected by at least one 3D interlinked convolution node and a plurality of temporal extraction nodes therebetween, and wherein the model is configured to identify and segment structural features of a vessel tree from the plurality of image frames, by the model being configured to: receive a plurality of image frames as an input to the 3D encoder; provide an output of the 3D encoder as an input to at least one interlinked 3D convolution node and at least one temporal extraction node of the plurality of temporal extraction nodes, wherein an output of the at least one interlinked 3D convolution node is provided as an input to at least one temporal extraction node; extract, by each of the plurality of temporal extraction nodes, temporal information of the received 3D input from one of the 2D encoder or at least one interlinked 3D convolution node; generate, at the 2D decoder, a combined 2D temporal output based on an output of the plurality of temporal extraction nodes, wherein the combined temporal output represents a predicted segmentation of the vessel; and generate an output representative of the segmented structural features of the vessel based on the predicted segmentation.

[0018] In further aspects, to extract temporal information, each of the plurality of temporal extraction nodes of the model are configured to: apply 3D convolution to the received 3D input (e.g. kernel size T x 1x 1); and dimensionally compress the 3D input so as to produce a temporally compressed 2D feature map representing the extracted temporal information.

[0019] In further aspects, the at least one interlinked 3D convolution node comprises a plurality of interlinked 3D convolution nodes connecting the 3D encoder and 2D decoder, and the plurality of interlinked 3D convolution nodes connecting the 3D encoder and 2D encoder comprise one or more skip connections therebetween.

[0020] In further aspects, the 3D encoder, 2D decoder, and the at least one interlinked 3D convolution node are arranged as nodes in a layered pyramidal network architecture; and wherein the plurality of temporal extraction nodes are arranged between the respective 3D and 2D nodes of network.

[0021] In further aspects, the model is further configured to minimise an energy loss associated with the predicted segmentation of the vessel based on determining a difference between the predicted segmentation and the labelled input, wherein the determined divergence is associated with an energy loss function representing an elastic energy of a system comprising the predicted segmentation and the labelled input.

[0022] In further aspects, the model is further configured to: determine the difference between a velocity of a vessel boundary of the predicted segmentation and a vessel boundary of the labelled input; and minimise the energy loss function based on the determined difference.

[0023] In further aspects, to minimise the energy loss, the 3D encoder, 2D decoder, and the at least one interlinked 3D convolution node are arranged as nodes in a layered pyramidal network, and the model is further configured to: combine the predicted segmentation with an output of at least one interlinked 3D convolutional node arranged in a top layer of the network to output a refined predicted segmentation; determine the difference between a velocity of a vessel boundary of the refined predicted segmentation and the vessel boundary of the labelled input; and minimise the energy loss function based on the determined difference.

[0024] In further aspects, the model is performed in mixed precision assigned memory, wherein the steps of the image segmentation model are performed in half tensor, by assigning a memory size of 16 floating points, except for the minimisation of the energy loss function which is performed in full tensor, by assigning a memory size of 32 floating points.

[0025] In further aspects, the 3D encoder, 2D decoder, and the at least one interlinked 3D convolution node are arranged as nodes in a layered pyramidal network, and the model is further configured to: backpropagate a determined gradient of the energy loss function at each of the nodes in the one or more layers of the network to update an assigned weighting of each respective node, wherein the gradient of the energy loss function is associated with a velocity of a vessel boundary of the predicted segmentation.

[0026] According to a further aspect of the present invention, a method of training a machine learning image segmentation model for identifying and segmenting structural features of a vessel tree from image frames (e.g. a series of contrast x-ray image frames) is provided, the method comprising: receiving a labelled training set for the machine learning image segmentation model, wherein the labelled training set comprises a plurality of image frames (e.g. sequential contrast x-ray image frames) of a vessel tree; training the image segmentation model, the model comprising a 3D encoder and 2D decoder, and wherein the 3D encoder and 2D decoder are connected by at least one interlinked 2D convolution node and a plurality of temporal extraction nodes therebetween, and wherein training the image segmentation model comprises: using the received labelled training set as an input to the 3D encoder; providing an output of the 3D encoder as an input to the plurality of temporal extraction nodes to extract temporal information; generating, by the plurality of temporal extraction nodes, a 2D temporal output based on the extracted temporal information; providing the generated 2D temporal output to the at least one interlinked 2D convolution node and 2D decoder; generating, at the 2D decoder, a combined temporal output based on an output of the at least one interlinked 2D convolution node and at least one temporal extraction node, wherein the combined temporal output represents a predicted segmentation of the vessel tree; and outputting the trained image segmentation model usable for identifying and segmenting structural features of a vessel tree from image frames.

[0027] According to a further aspect of the present invention, a method of training a machine learning image segmentation model for automatically identifying and segmenting structural features of a vessel tree from image frames is provided, the method comprising: receiving a labelled training set for the machine learning image segmentation model, wherein the labelled training set comprises a plurality of image frames (e.g. sequential contrast x-ray image frames) of a vessel tree; training the image segmentation model, the model comprising a 3D encoder and 2D decoder, and wherein the 3D encoder and 2D decoder are connected by at least one interlinked 3D convolution node and a plurality of temporal extraction nodes therebetween, and wherein training the image segmentation model comprises: using the received labelled training set as an input to the 3D encoder, providing an output of the 3D encoder as an input at least one interlinked 3D convolution node and at least one temporal extraction node, wherein an output of the at least one interlinked 3D convolution node is provided as an input to at least one temporal extraction node; extract, by each of the plurality of temporal extraction nodes, temporal information of the received input from one of the 3D encoder or at least one interlinked 3D convolution node; generating, by the plurality of temporal extraction nodes, a 2D temporal output based on the extracted temporal information; generating, at the 2D decoder, a combined 2D temporal output based on an output of the plurality of temporal extraction nodes, wherein the combined temporal output represents a predicted segmentation of the vessel tree; and outputting the trained image segmentation model usable for identifying and segmenting structural features of a vessel tree from image frames.

[0028] According to a further aspect of the present invention, a method of automatically identifying and segmenting structural features of a vessel tree in a plurality of image frames is provided, the method comprising: receiving a plurality of x-ray images; providing the received images to a trained machine learning image segmentation model for automatically identifying and segmenting structural features of a vessel tree from image frames; and receiving, from the trained model, an output representative of the segmented structural features of the vessel tree of the plurality of images.

[0029] In further aspects, the trained model is trained according to the above aspects of the invention.

[0030] According to a further aspect of the invention, a trained image segmentation model is provided for automatically identifying and segmenting structural features of a vessel tree from image frames, wherein the model is trained in accordance with the above-detailed aspects.

[0031] According to a further aspect of the invention, there is provided a computer-readable medium having stored thereon computer-readable code representative of a machine learning machine segmentation model in accordance with any of the aspects detailed above.

[0032] According to a further aspect of the invention, there is provided a computer-readable medium having stored thereon computer-readable code representative of an image segmentation model trained in accordance with any of the aspects detailed above.

[0033] According to a further aspect of the invention, there is provided a computer-readable medium having stored thereon computer-readable code executable so as to perform the method in accordance with any of the aspects detailed above. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Embodiments of the invention will now be described by way of example only, with reference to the accompanying Figures, in which:

[0035] FIG. 1 illustrates a model for identifying and segmenting structural features of a vessel tree from image frames in accordance with examples of the present disclosure;

[0036] FIG. 2 illustrates a model for identifying and segmenting structural features of a vessel tree from image frames in accordance with examples of the present disclosure;

[0037] FIG. 3A illustrates a 3D convolution node in accordance with examples of the present disclosure;

[0038] FIG. 3B illustrates a 2D convolution node in accordance with examples of the present disclosure;

[0039] FIG. 3C illustrates a temporal extraction node in accordance with examples of the present disclosure;

[0040] FIG. 4 illustrates an image plane relevant to an energy loss function in accordance with examples of the present disclosure;

[0041] FIG. 5A illustrates a method of training a machine learning image segmentation model in accordance with examples of the present disclosure;

[0042] FIG. 5B illustrates a method of training a machine learning image segmentation model in accordance with examples of the present disclosure;

[0043] FIG. 6 illustrates a method of automatically identifying and segmenting structural features in a vessel tree, in accordance with examples of the present disclosure;

[0044] FIG. 7 illustrates a computer readable medium having stored thereon computer-readable code representative of a machine learning image segmentation model in accordance with examples of the present disclosure;

[0045] FIG. 8 illustrates skeletisation metrics for vessel centreline in accordance with examples of the present disclosure;

[0046] FIG. 9 illustrates a qualitative evaluation of segmentation output samples in accordance with examples of the present disclosure;

[0047] Fig. 10 represents a qualitative evaluation of segmentation and skeleton output samples using 3D(T) UNet++ in accordance with examples of the present disclosure;

[0048] FIG. 11 illustrates vessel segmentation performance using 3D(T) UNet++ with deep and without deep supervision in accordance with examples of the present disclosure;

[0049] FIG. 12 illustrates re-segmented samples in accordance with examples of the present disclosure;

[0050] FIG. 13 illustrates a cumulative distribution of the Dice coefficient scores for test samples in accordance with examples of the present disclosure;

[0051] FIG. 14 illustrates re-segmented samples in accordance with examples of the present disclosure;

[0052] FIG. 15 illustrates results of a qualitative evaluation of re-segmentation and skeleton output sample in accordance with the present disclosure. DETAILED DESCRIPTION

[0053] Aspects and examples of the present disclosure relate to a machine learning algorithm or model, for example, an image segmentation model, which may automatically identify and segment structural features of a vessel tree from a series of image frames. The present disclosure also relates to ways of training the machine learning model for automatically identifying and segmenting structural features of a vessel tree. The present disclosure further discloses methods of identifying and segmenting structural features of a vessel (e.g. a vessel tree) from a series of images, for example, a series of contrast x-ray frames in invasive coronary angiography, ICA.

[0054] In the following examples and disclosure, conventional, or manual annotation of annotation of ICA images, from which segmentation and vessel tree re-construction into diagnostic images may be performed, may throughout be interchangeably referred to as a manual annotation, gold standard, or labelled input. In some examples the gold standard may be referenced, or discussed, as a ground truth, in respect of training one or more examples of models of the disclosure.

[0055] Whilst various embodiments are described below, the invention is not limited to these embodiments, and variations of these embodiment may fall within the scope of the invention, which is set out by the appended claims.

[0056] FIG. 1 relates to a machine learning image segmentation model 100 for identifying and segmenting structural features of a vessel tree from a series of image frames.

[0057] In the following examples, the model 100 will be discussed in relation to identifying and segmenting structural features of a vessel tree (e.g. one or more vessels, particularly, one or more branched vessels), in accordance with a coronary vessel network, wherein a vessel tree may refer to one or more main vessels having one or more subbranches. The described series of image frames may be discussed in terms of a series, or sequence, of contrast x-ray image frames, and may be described in relation to a series of images obtained from invasive coronary angiography, or ICA.

[0058] However, it will be appreciated that the model may otherwise be used to identify and segment any such vessel(s) or vessel trees. For example, the model(s) described may be used in view of other tubular, or tube-like, structures within the body (e.g. biliary tree, for example), wherein a series of images, or video, are obtained. In some examples, these images or video may be x-ray images, or contrast x-ray images, but this is not intended to be limiting, and in some examples, may be used to segment structures in other image or video sequences. That is, whilst examples are discussed herein, the techniques and examples discussed may apply to any image sequence for segmenting complex objects or structures affected by motion, and the present disclosure is not intended to be limited as such.

[0059] Herein, model 100 may alternatively be described as an algorithm, a network, a framework, an architecture, or other relevant terminology as known to the skilled person.

[0060] Referring to again to FIG. 1, the model 100 comprises an encoder-decoder structure with connections therebetween, and comprises (or is arranged as) a layered pyramidal network architecture. The model 100 may be described as a deep-learning architecture, and specifically, may be described in relation to U-Net++. That is, the above-described model 100 may be considered (or herein termed) a modified U-Net++ architecture. Model 100 may be herein termed as 3D(T) UNet++, and reference herein to this terminology in the examples discussed may be considered in view of model 100.

[0061] Model 100 comprises an encoder 102 and a decoder 104, wherein the encoder 102 and decoder 104 (that is, encoder, or decoder sub-networks) are connected by, or herein comprise, at least one interlinked convolution node 106 (alternatively, extra convolution node 106, or additional convolution nodes, or blocks) and a plurality of temporal extraction nodes 108 therebetween. Here “interlinked” may be considered to refer to the nested links or connections of said nodes to one or more nodes within the network between the encoder 102 and decoder 104. That is, an interlinked convolution node may be linked between one or more further interlinked convolution nodes by one or more pathways, and the at least one interlinked convolution node(s) 106 are connected between the encoder 102 and the decoder 104.

[0062] It is noted that in the descriptions and examples herein that the term “node” and “block” may be interchangeably used to represent the features of the model(s).

[0063] As indicated, the interlinked nodes provide a number of nested connections between the encoder 102 and decoder 104, as shown by the connections in FIG. 1. In particular, the encoder 102 and decoder 104, and the at least one Interlinked convolution node 106 may comprise one or more skip connections (skip pathways, or nested skip pathways) therebetween. Herein, skip connections, or skip connected nodes, are understood as conventionally known, whereby an output of any skip-connected node may be connected to at least one non-adjacent node and / or an adjacent node within the network. In FIG. 1, examples of such skip connections are illustrated by the dashed arrow connections.

[0064] The interlinked convolution nodes may connect (comprise a pathway or connection) to nodes of a same and / or upper or lower layer within the model 100. For example, and as shown in FIG. 1, each of the interlinked nodes may perform up sampling (up-sampling layer, e.g. scale factor (2, 2)) to provide an up sampled output a higher (upper) layer of the network. It will be appreciated that, as shown in FIG. 1, the inclusion of these further nodes, the nested connections, and one or more skip connections between the encoder and decoder blocks (or nodes), at one or more levels or resolutions, provide a dense network for multi-scale structure concatenation.

[0065] As shown within FIG. 1, the model 100 comprises a number of both 3D and 2D convolution nodes, LFj, that is, either 3D or 2D in terms of the data dimension, or data format, processed by the convolution layer. In FIG. 1, the 3D convolution blocks (that is, convolution process on 3D data, e.g. encoder 102) are shown represented by the dark shading and 2D convolution blocks (convolution process on 2D data, e.g. nodes 106, decoder 104) are herein represented in the lighter shading. In model 100, within FIG. 1, it may be seen that the interlinked convolution nodes 106 are 2D interlinked convolution nodes 106, that is, they receive, and process, data of 2D input. Further reference to the steps of 3D and 2D convolution performed by the respective nodes is shown in FIG. 3A and FIG 3B.

[0066] Encoder 102 may be termed herein, and in the following examples, as 3D encoder 102, or backbone, of the network, and may be comprised of a plurality of connected 3D convolution nodes, which are arranged in respective layers of the network above-described. That is, as shown in FIG. 1, encoder 102 may refer to (or comprise) connected 3D convolution nodes U° °, U1 °, U2 °, U3 0 and U4 0, in respective layers of the network, and an output of encoder 102 may refer to an output of at least one, or all, of these nodes. The encoder 102, or the nodes of the 3D encoder, obtained or captures features of a respective input (e.g. an image input), through convolution and pooling layers. That is, and referring to FIG. 1, the encoder 102, or each node of the encoder, performs pooling on a respective input, as set out in FIG. 1 ("Pooling 3D"), which in the example shown, using a kernel size (1, 2, 2), stride (1, 2, 2). However, it may be appreciated that in some examples, a different kernel size and or stride may be set, in consideration of dimension, resolution or computer load.

[0067] Decoder 104 may be termed herein as 2D decoder 104, and may be comprised of a plurality of connected convolution 2D nodes, in respective layers of the network above-described. That is, as shown in FIG. 1, decoder 104 may refer to (or comprise) connected 2D convolution nodes U31, U2 2, U1-3, and U0 4, and an input to decoder 104 may refer to at least one, or all, of these nodes. The decoder 104, or the plurality of nodes of the decoder, may perform up-sampling (e.g. “Up-Sampling 2D”, at scale factor (2,2)) of a respective received input to generate an image segmentation map (e.g. based on the features received form the encoder sub-network), to a final node output (U0 4), which outputs a predicted segmentation, fs(), 122.

[0068] As shown in FIG. 1, the at least one interlinked convolution node 106 may comprise a plurality of interlinked convolution nodes 106. In respect of the example shown in FIG. 1, the interlinked convolution nodes 106 may be termed interlinked convolution 2D nodes 106, comprising 6 interlinked 2D convolution nodes when the network comprises 5 layers (a 5 layer pyramidal network, or framework, UNet++ architecture). However, in other examples, a network may comprise a different number of layers (e.g. 4 layers, 6 layers), in which case, a different number of interlinked convolution nodes may be used. It will be appreciated that the number of layers, and therefore the number of interlinked convolution nodes therebetween, may be considered in terms of a balance between increasing the denseness of the network, without necessitating significant drain on resource in view of training and utilising the model.

[0069] The layers of the model 100 of FIG. 1 may be described as a top to bottom layers, for descriptive purposes. A top layer 112, or upper layer, is represented, or named, by nodes “UO x”, where x references the number of the node along this layer. For example, the top layer 112 may comprise at least a node of 3D encoder 102, 2D decoder 104 and at least one interlinked convolution node 106. In view of the structure described in UNet++, and with regard to FIG. 1, the bottom layer 114 may comprise a single node, e.g. Node "U4 °". In examples, the bottom layer 114, here, Node (U4 0), is described for convenience and clarity as an encoder (or part of encoder 102, or as an encoder node), as the downhill (or down sampling) stops at this node. However, this node may be alternatively considered as half decoder, or half encoder, and divided by a second convolution (e.g. second convolution layer) in the node.

[0070] The convolution nodes (that is, both the 2D and 3D convolution nodes, having respective convolution layers) in the network are represented, or shown, by IF J, where i represents the number of pooling or down sampling operations, or steps, that the node has performed from U° °, and j representing the number of received skip connections of the node, in the 5 layer-pyramidal structure as shown. In this example, an output channel (e.g. as represents the number of layers of the output features) of every node equals 2i+5, and the output (e.g. output size) of the bottom node is 512.

[0071] With the 6 interlinked convolution nodes 106 shown in FIG. 1 (U01, U0 2, U0 3, U1-1, U12, U21), the connections originating from U° ° may be seen as a dense convolution block. For example, node U1-2, has two skip connection inputs and one up-sampling output that are concatenated together, where these outputs are generated in different convolution layers and pyramidal layers. This brings the semantic gap between the encoder 102 and decoder 104 closer, which consequently aids the optimisation process of the network.

[0072] Mathematically, for example, the above may be expressed as: ( V^i) j = 0 yij = J / = 1 v ([[¢ / ^^^((7^),5((7^1^1)]) / >1

[0073] In the above-equation, the functions V(), v(), S(j, F() and [] represent the 3D and 2D convolution blocks, up-sampling layer, temporal extraction node, and concatenation layer, respectively. That is, j=0 nodes only process 3D convolution (e.g. encoder nodes 102), j=1 nodes are 2D convolving concatenated results of temporal feature extraction of the previous node and up-sampled node. That is, the j=1 nodes receive the 2D temporal output from or more temporal extraction nodes 108 (that is, the extracted input of a previous 3D node, either skip connected or up-sampled). The j>1 nodes convolve concatenated temporal features extracted from the first nodes.

[0074] As above, the terms “2D” and “3D” are used to describe nodes, or blocks, this is used to refer to the dimensionality, or format, of the data, and operations performed at the blocks, or nodes therein, within the model. For example, an input, or 3D input, may represent data in view of a series of image frames acquired during ICA, where input data (images) comprises having image dimensional information (e.g. h, w), and temporal information (t), comprising, or corresponding to, temporal information relating to the sequence of images. However, the output of the network is processed in a 2D format. Consequently, the data between the input to the model 100, and the output to the model, is different in format. This loss of information may result in errors in connectivity, or lossiness, in the segmentation output.

[0075] In order to manage the difference in these two formats, part of the network, or model, has to manage 3D data, and the remaining part of the network manages 2D data. However, the abovedetailed 3D and 2D parts of the network, cannot be directly linked.

[0076] To address the above, the model 100 of FIG. 1 comprises a plurality of temporal extraction nodes 108 between the 3D encoder 102 and 2D decoder 104 to extract (e.g. retain) temporal information of the 3D nodes within the 2D output.

[0077] In detail, and referring to FIG. 1, the model 100 is configured, or arranged, to receive a data, X, which may be a plurality of image frames. In particular, data, X, may be a plurality of sequential, or series data, as an input 110 to the 3D encoder 102. In an example, these may be a series of images (or data frames) obtained during ICA, that is, a series of sequential x-ray contrast image frames as an input 110.

[0078] Herein, the encoder 102 generates an output, which may be provided as an input to the plurality of temporal extraction nodes 108 which may extract temporal information from the 3D output of encoder 102. In other words, the output of encoder 102, which may comprise one or more connected 3D convolution nodes, generates an output which is provided to the plurality of temporal extraction nodes 108.

[0079] In further examples, and referring to FIG. 1, each of the connected 3D convolution nodes of the encoder in a respective layer may generate a respective output which is output to one or more temporal extraction nodes 108, or temporal 3D-2D blocks.

[0080] Each of the plurality of temporal extraction nodes 108 may generate a 2D output based extracted temporal information from the received 3D input. In particular, each of the plurality of temporal extraction nodes 108 may produce, or generate, a 2D feature map from the received 3D input (e.g. from the encoder 102) by squeezing (dimensionally compressing) a feature map in a time axis, so as to retain temporal information. Herein, this 2D output from temporal extraction nodes and described in the examples herein may be termed as a “temporal”, “2D temporal output”, temporally compressed 2D feature map, or 2D temporally fused output, by way of describing the retained temporal information. Further aspects of the temporal extraction node are illustrated in FIG. 3C.

[0081] The 2D temporal output of the plurality of temporal extraction nodes 108 may be provided to at least one 2D interlinked convolution node 106 and 2D decoder. That is, each of temporal extraction nodes 108 may provide an output (e.g. 2D temporal output) to one of the 2D interlinked convolution nodes 106 or the 2D decoder within the network. In other words, an output from each 3D convolution node in the network is provided to at least one temporal extraction node to extract the temporal information, and in some examples, the output of a 3D node (e.g. a node of encoder 102) may be provided to a plurality of temporal extraction nodes 108. Each temporal extraction node generates a 2D temporal output based on this information, and provides this 2D output to one 2D convolution node (e.g. a 2D convolution node of the plurality of interlinked convolution nodes 106 or a 2D node of decoder 104).

[0082] For example, in FIG. 1, an output of node U°0 in top layer 112 is provided to 4 temporal extraction nodes 108, each of which provide an output to one 2D convolution node. Herein, the output is provided, by a respective temporal extraction node, to each of a plurality of interlinked convolution nodes 106 in top layer 112 and decoder 104. In examples, the temporal extraction nodes 108 may be arranged so that the 2D temporal output is provided to a 2D convolution node of an adjacent node in a same layer, or a non-adjacent node in a same layer, or may be provided to an adjacent node in a different layer. That is, the 3D and 2D parts of the network have at least one temporal extraction node 108 therebetween. It will be appreciated, therefore, that temporal extraction nodes 108 therefore provide the link, or connection, between the 3D and 2D parts of the network, and allow for retaining of temporal information.

[0083] At the decoder 104, a combined temporal output may be generated based on an output from at least one interlinked convolution node 106 and at least one temporal extraction node 108, wherein the combined temporal output represents a predicted segmentation of the vessel tree. That is, in the example of Fig. 1, the decoder 104, which may comprise the plurality of 2D convolution nodes arranged in layers of the network, may receive a 2D output from the plurality of 2D interlinked convolution nodes 106, and 2D temporal output from one or more of the plurality of temporal extraction nodes. Each node of decoder 104 arranged in the network may receive input from one or more temporal extraction nodes, and additionally, an input from any 2D interlinked convolution nodes 106 arranged in a same layer of the network. Each node of decoder 104 may up sample the received inputs (e.g. combined inputs) through a next layer node of the decoder 104 arranged in the network to the final node U0 4to generate a combined temporal output.

[0084] Herein, the predicted segmentation, is represented in FIG. 1 by fs(.). That is, the combined temporal output is an output representative of the segmented structural features of the vessel tree, based on predicted segmentation and identification performed by the model 100. In some examples, the output may be presented (or output) as an image representative of the segmented structural features of the vessel tree.

[0085] In the example shown in FIG. 1, the model 100 is constructed with the interlinked convolution nodes 106 as 2D convolution nodes, between the 3D encoder 102 and the 2D decoder 104. As indicated, this construction of the model 100 may be termed a 3D(T) UNet++, where the temporal extraction nodes 108 are arranged to receive input from the encoder 102. In other words, the temporal extraction feature is performed directly after the encoder, such that the 3D convolution is performed only in the pooling phase.

[0086] In further examples, the interlinked convolution nodes may be 3D interlinked convolution nodes. An alternative example, comprising 3D interlinked convolution nodes, is shown in FIG. 2, which may be termed a 3D(T+) UNet++.

[0087] FIG. 2 illustrates a model 200 for identifying and segmenting structural features of a vessel tree from a series of image frames in accordance with examples of the present disclosure. The model 200 of FIG. 2 broadly corresponds to the model 100 of FIG. 1, and the same reference numerals are used therein to refer to the same representative features. That is, the description relevant to FIG. 1 may be similarly referenced, except in view of the difference indicated below.

[0088] As referring to FIG. 1, the model 200 comprises an encoder-decoder structure with connections therebetween, in a layered pyramidal network architecture. The model 200 may be described in relation to a deep-learning architecture, and specifically, may be described in relation to UNet++, and may be herein termed as 3D(T+) UNet++, and reference herein to this terminology in the examples discussed may be considered in view of model 200.

[0089] The model 200 differs from model 100 in that the plurality of interlinked convolution nodes 206 are 3D convolution nodes, that is, are arranged to receive and perform convolution on received 3D data, and the arrangement of the temporal extraction nodes 108 in relation to the network. That is, as shown in FIG. 2 the temporal extraction nodes 108 are arranged before the decoder 104, such that the temporal feature extraction happens before (e.g. directly before) the decoder 104. That is, whereas in FIG. 1, the temporal extraction nodes 108 are positioned, or arranged after (e.g. directly after) the encoder 102, such that the interlinked convolution nodes 106 process 2D information, in FIG. 2, the temporal extraction nodes are positioned, or arranged, before the decoder 104, and the interlinked convolution nodes 206 process 3D information.

[0090] In detail, FIG. 2, the model 200 is configured, or arranged, to receive a data, X, which may be a plurality of sequential, or series data, as an input 110 to the 3D encoder 102. In an example, these may be a series of images (or data frames) obtained during ICA, that is, a series of sequential x-ray contrast image frames, that represent 3D information, as an input 110.

[0091] Encoder 102 generates an output. The output (3D output) of the encoder 102 (e.g. the output of nodes comprising the encoder 102) may be provided to at least one temporal extraction node 108 of the plurality of temporal extraction nodes 108 and at least one 3D interlinked convolution node 206. For example, a node of the encoder 102 may provide an input to one or more interlinked convolution nodes 206, and / or a temporal extraction node 108 arranged between the 3D encoder 102 and 2D decoder 104.

[0092] Each of 3D interlinked convolution nodes 206 is configured, or arranged, to generate a 3D output. That is, as the output of the 3D interlinked nodes 206 remains in 3D. The 3D output of each of the plurality of 3D interlinked convolution nodes is provided as an input to at least one temporal extraction node 108. An output of each respective 3D interlinked convolution node 206 may also provide an output to another one or more interlinked 3D convolution nodes (e.g. a 3D interlinked node of the same layer, or higher layer, of the network). In other words, the temporal extraction nodes 108 are provided before (e.g. directly before) the decoder 104, and any 3D output (e.g. from the 3D encoder, or 3D interlinked convolution nodes 206) is input to a temporal extraction node 108 prior to the decoder 104. Each temporal extraction node 108 receives a (e.g. one) respective 3D input.

[0093] Each of the plurality of temporal extraction nodes 108 may generate a 2D temporal output based on extracted temporal information from the received 3D input. In particular, each of the plurality of temporal extraction nodes 108 may produce, or generate, a 2D feature map from the received 3D input by squeezing (dimensionally compressing) a feature map in a time axis, so as to retain temporal information. Herein, this 2D output from temporal extraction nodes and described in the examples herein may be termed as a “temporal”, “2D temporal output”, temporally compressed 2D feature map, or 2D temporally fused output, by way of describing the retained or extracted temporal information. Further aspects of the temporal extraction node are illustrated in FIG. 3C.

[0094] An output of each of the plurality of the temporal extraction nodes 108 is provided to the decoder 104. At the 2D decoder 104, a combined temporal output may be generated based on an output from at least one temporal extraction node 108, wherein the combined temporal output represents a predicted segmentation of the vessel tree. That is, in the example of Fig. 2, the decoder 104, which may comprise the plurality of 2D convolution nodes arranged in layers of the network, may receive a 2D temporal output from the plurality of temporal extraction nodes. Each node of decoder 104 arranged in the network may receive input (e.g. a 2D temporal input) from one or more temporal extraction nodes. Each node of decoder 104 may up sample the received inputs (e.g. combined inputs) through a next layer node of the decoder 104 arranged in the network to the final node U0 4to generate a combined temporal output. Herein, the predicted segmentation, is represented in FIG. 2 by fs(.). That is, the combined temporal output is an output representative of the segmented structural features of the vessel tree, based on predicted segmentation applied by the model 200. In some examples, the output may be presented (or output) as an image representative of the segmented structural features of the vessel tree.

[0095] That is, as the plurality of interlinked convolution nodes 206 are 3D convolution nodes, 3D convolution occurs through the network, until the up-sampling of the feature maps generated by the output of the temporal extraction nodes 108.

[0096] Similarly to the notation with reference to FIG. 1, this may be expressed as: j = 0 = . v ' +i <4’7 >0 [v S([ / ^-9]) i +j = 4,j >0

[0097] In the above-equation, the functions V(), v(), S(), F() and [] represent the 3D and 2D convolution blocks, up-sampling layer, temporal extraction node, and concatenation layer, respectively.

[0098] Herein, the features, or steps performed by each of the 3D convolution blocks, 2D convolution blocks, and the temporal extraction nodes 108, are performed as described in relation to FIG. 3Ato FIG. 3C, respectively, and in view of FIG. 1 and FIG. 2, and reference is made to this preceding discussion, which similarly apply here.

[0099] The steps and functions performed by the described 3D convolution nodes and 2D convolution nodes, in model 100 and model 200, may be further described with reference to FIG. 3A to FIG. 3B, as follows.

[0100] With reference to FIG. 3A, a 3D convolution block 302 (that is, a block or node receiving and processing 3D data) may comprise, or perform, convolution on a 3D input. For example, herein a 3D input may refer to the 3D data comprising three dimensions, in height, h, width, w, and time, t (that is, a temporal aspect linked to the sequence, or series, of data). With reference to FIG. 1, or FIG. 2, the blocks, or nodes, performing 3D convolution are shown in a dark shading (e.g. encoder 102, in FIG. 1, the encoder 102 and the interlinked convolution nodes 206, in FIG. 2). Any of these blocks, or nodes, may be referred to as a 3D convolution node 302.

[0101] The 3D convolution node 302 is configured, or arranged, to receive the 3D input 310. At 312, the block performs, or applies, 3D convolution ("Conv3D"), to the received input (e.g. by way of convolution layer). Herein the kernel size is set at (3, 3, 3), and a stride is set at (1, 1, 1). However, it may be appreciated that in some examples, a different kernel size and or stride may be set, in consideration of dimension, resolution or computer load, for example.

[0102] At 314, the 3D convolution mode performs batch normalisation, “BatchNorm3D”, and at step 316, configured to use a rectified linear unit (ReLu) on a respectively received input from the previous step. Such features are well known in the field of neural networks, and machine learning, and are not explained herein in detail.

[0103] At steps 318-322, the output of the ReLu is used, and the steps of convolution 318, batch normalisation 320, and rectified linear unit 322, are performed again, to obtain a 3D (convoluted) output from the 3D block.

[0104] With reference to FIG. 3B, a 2D convolution block 304 (that is, a block receiving and processing 2D data) may comprise, or perform, convolution on a 2D input. For example, with reference to FIG. 3B, the 2D data input 322 may have dimensions in height, h and width, w. With reference to FIG. 1, or FIG. 2, the blocks, or nodes, performing 2D convolution, or comprising a convolutional layer receiving 2D input, are shown in light shading (e.g. decoder 104, interlinked convolution nodes 106 in FIG. 1, the decoder 104 in FIG. 2). Said blocks, or nodes, may be referred to as a 2D convolution node 304.

[0105] The 2D convolution node 304 is configured, or arranged, to receive the 2D input 322. At 312, the block performs, or applies, 2D convolution ("Conv2D") to the received input (e.g. by way of convolution layer). Herein the kernel size is set at (3, 3), and a stride is set at (1, 1). However, it may be appreciated that in some examples, a different kernel size and or stride may be set, in consideration of dimension, resolution or computer load, for example.

[0106] At 326, the 2D convolution mode performs batch normalisation (“BatchNorm2D”) and at 328, is configured to use a rectified linear unit (ReLu) on a respectively received input from the previous step. At steps 330-334, the output of the ReLu is used, and the steps of convolution 330, batch normalisation 332, and rectified linear unit 334, are performed, to obtain a 2D (convoluted) output from the 2D block.

[0107] Fig. 3C illustrates steps performed by any of the plurality of temporal extraction nodes 108, within either of model 100 or model 200. As described above, temporal information (e.g. the temporal information from the sequentially obtained data or images), is treated, or modelled, as a third dimension. This information is extracted, and thereby retained, by the 3D-2D temporal extraction nodes 108.

[0108] As illustrated in FIG. 3C, a 3D-2D temporal block 306, which may represent, or be equivalent to, any of temporal extraction nodes 108 in FIG. 1 or FIG. 2, receives a 3D input 340. That is the data received (e.g. representative of image frame data, with reference to Fig. 1 or Fig. 2) may have three dimensions, in height, h, width, w, and time, t.

[0109] At 342, the 3D-2D convolution block 306 performs convolution on the 3D input, that is, by a convolution layer. In the example shown, kernel size (T=4, 1, 1) and stride = (1, 1, 1). Herein, the obtained data is squeezed, or dimensionally compressed in the time, t, axis, so as to obtain, or produce a 2D feature map (2D temporally compressed 2D feature map). For example, all dimensions of the input having size 1 may be removed to generate the temporally compressed 2D feature map 344. In examples, the generated output (2D feature map) may be used, or described, as a 2D input, for provision to further nodes of the network (e.g. 2D input 322, as above).

[0110] Mathematically, this can be expressed as: F = SqueezeT(Ul,J ® K')

[0111] where K is the learnable kernel with size (4, 1, 1).

[0112] Referring back conventional use of segmentation for ICA, when initial sequence of images is received, e.g. the 3D image sequence, information relating to the “temporal” information of the segmentation is often lost when this initial sequential “3D” image sequence is converted to 2D information. This loss of information may result in a loss of connectivity, or disconnected vessels, in the output of the segmentation. This information may be retained in the manual gold standard, for example, by manually retaining a link between images. In this case, by manually annotating and / or including this information, less information is “lost”, and connectivity can be retained. However, this methodology is time consuming, and still may be prone to manual error.

[0113] Consequently, as described with reference to the models above, the use of temporal extraction nodes allows the extraction, or fusion, of temporal information within the acquired 2D feature map, so as to retain said information - as linked to the connectivity of vessels - in the segmentation output.

[0114] Further to the above, to improve the identification and segmentation of vessels within the models (e.g. model 100 and model 200), aspects of the present disclosure may comprise minimising an associated energy loss associated with the system, so as to improve delineation of vessel boundaries and improve the accuracy of the vessel connectivity.

[0115] Aspects of the minimisation of the energy loss will now be described relevant to FIG. 1 and FIG. 2, as above, with reference to FIG. 4. Whilst reference is herein made to FIG. 1, by means of like and corresponding reference numerals, paths and functions, reference may herein also be made to FIG. 2.

[0116] At model 100 of FIG. 1, an output, e.g. a final output, from the decoder 104 (herein, U0 4), may represent the predicted segmentation 122, or segmentation output, of the model 100, and represented by fs(.) as shown. The model (100, 200) further comprises, or fuses, a loss function based on elastic interaction to preserve connectivity of the vessels. Specifically, the loss function, L, 124, is based on an interaction between the predicted segmentation 122, and a labelled input 120. Herein, the labelled input 120, y, may represent the aforementioned gold standard. However, it will be appreciated that any labelled, or comparative, input may be used. By means of comparison, the interaction is determined based on a determined difference between the predicted segmentation and the labelled input 120, wherein this determined difference is associated with the energy loss function that represents an elastic energy of the system of the predicted segmentation and the labelled input.

[0117] More specifically, the energy loss (or energy loss function) 124, is minimised in view of a difference, or divergence, in a boundary location of a moving boundary (that is, representative of the predicted segmentation 122), and a stationary boundary, of the labelled input 120. This model is shown in respect of FIG. 4. Herein, it may be understood that, in view of the predicted segmentation being updated or refined, as a fluid prediction, there is an associated “movement” with this prediction (and therefore vessel boundary location), whereas the labelled input 120 may be seen as fixed, or stationary. That is, as the difference, or divergence in the boundary location changes, an associated velocity (e.g. 406) corresponds to, or relates to, this difference. That is the divergence in the boundary locations comprise an associated velocity 406, which is related to the associated elastic energy of the system. In other words, a minimisation in the difference of the boundary location (and related associated velocity) corresponds to a minimised energy loss value, or minimised function.

[0118] Referring back to FIG. 1 and FIG. 2, each of models 100 and 200 are arranged, or configured, to perform deep supervision in the framework, so as to provide additional regularisation. That is, an output of each of a plurality of interlinked convolution nodes 106 (FIG. 1) or a plurality of interlinked convolution nodes 206 (FIG. 2) in a top layer 112 of the network are used to provide the supervision, as shown at 118 in FIG. 1 and FIG. 2. The output of the interlinked convolution nodes 106, or 206, providing the supervision is combined with a final node output (e.g. 122) to obtain a refined predicted output. That is, in some examples, the refined predicted output is used to calculate the energy loss value, or minimise the energy loss function, as described above.

[0119] In some examples, when performing the deep supervision, the loss may be calculated after applying a sigmoid on convolution nodes of the top layer (e.g. U°1, U° 2, U°3, and U° 4) to obtain four loss values. An average is applied to the four loss values to obtain the deep supervision (i.e. the averaged) loss. That is, the four losses, providing the deep supervision loss, act as an additional regularisation method, and may reduce test error and provide faster loss convergence.

[0120] Herein, in further detail, the determination and minimisation of the energy loss function, as detailed in any of the above references, will be further described.

[0121] The energy loss to be minimised is a function transferred from the elastic energy of dislocations in crystals. The energy system of any single curve y(x(s),y(s),z(s)) in 3D space (x.y.z) consists of two key functions: —>, x 1 r fKcLl . _ 1 r r dl-dl' w(x,y,z) =--f —v and E = — f f -7 7 47rJy |r|3 8nJYJY |r| Equations (1) and (2)

[0122] Herein, <S2(y) is a delta function of the curve y which is always perpendicular to y, dl = T82(y)dxdydz is an infinitesimal part of the curve (t is the unit tangent vector of y), w = (0,0,w3(x,y,z)) is a function of the xyz plane, and r = (x - x(s),y -y(s),z - z(s)) is a vector between a point in the space to a point on the curve. More specifically, w(x,y,z) represents the dynamics of the system under given constrains, while E represents the total elastic energy of the system. These equations are obtained by solving the Lagrange equations formed by the simplified version of the dislocation elastic energy.

[0123] The above system can be utilised for vessel segmentation, relating to the examples of the present disclosure, as it is also applicable to a collection of curves. That is, the stationary vessel boundary of a labelled input 120, e.g. gold standard, can be defined as yn and the moving boundary of the predicted segmentation as y2, the moving boundary of the predicted segmentation 406 associated with the moving location during the training process, and y can be substituted with a collection of curves y, uy2. The velocity vector of a point on y2 at an arbitrary time t is always pointing towards a gold standard point on y,. To adapt the curves to vessel boundaries in gold standard images and generated segmentation, y^ can be defined as GS] which is obtained by convolving the gold standard yn with a 2D Gaussian function. Herein, y2 is defined as / / (0) which is obtained by applying a regularised Heaviside function / / (•) on level set representation 0 of the predicted vessels fs(-). By substituting fandw, the dynamic equation which minimises energy may be expressed as: 1 f r-V(Gs+a / / (0)) v(x, y) = - — ----------dxdy 4nj^2 |rp Equation (3)

[0124] a, in the above, is a hyper-parameter. By defining Gs + aH^~) = T(x,y), the energy of the system can be derived from equation (2) as: 1 r [ VT(x,y)-VT(x',y') , E = — dxdy I ------------dx dy |r| Equation (4)

[0125] That is, the above equation, is the loss function L of the energy system, and equation (3) represents the corresponding gradient of this loss function.

[0126] In further detail, the energy loss function may be derived in detail, as below.

[0127] Without the simplification of w = (0,0,w3(x,y,z)), a single curve is considered (e.g. the solid curve boundary 404, of FIG. 4). The energy system of this curve, as before, may be y(x(s),y(s),z(s)), and consistent of three key functions (equations (1), and (2) reproduced from above): V x w = (y)T Equation (5) r X dl w Equation (1) Equation (2)

[0128] Herein S2(y) is a delta function in 2D, £ is the unit tangent vector of y, and dl = fo2 (y)dxdydz is an infinitely small part of the curve, w = (w1(x,y,z),w2(x,y,z), w3(x,y, z)) is a vector in the xyz plane, and r = (x - x(s), y - y(s), z - z(s)) is a vector between points.

[0129] More specifically, equation (5) is the constrain function of the energy system, which holds no matter how the curve changes. Equation (1) represents the dynamics of the system under given constrain, and equation (2) represents the total elastic energy of the system when an element of the curve is expressed as dl. Given the property of r and dl, w can geometrically be seen as a vector which is always perpendicular to the image plane 400.

[0130] The system is also valid on a collection of curves. Referring to FIG. 4, the stationary boundary of the ground truth (i.e. the labelled input, 404, or gold standard) is yr and the moving boundary of the prediction 402 as yz. Therefore, y can be substituted with a collection of curves y1 u y2 for applying this system on image segmentation.

[0131] By introduction of a moving curve y, the velocity to describe its movement at time t is defined as: , dy • • v(x, y,z) = — = w X t Equation (6)

[0132] This is the cross product of these two vectors, and as y2 is the only curve that is moving -f can be changed to T£ which is the unit tangent vector of predicted boundary.

[0133] The level set method can be used to describe the state of the moving boundary. Any curve that is evolving can be imagined as a cross section of a curved surface or level set $ and the curve is correspondingly changing as the plane of cross section moves. The velocity of the moving curve is defined implicitly in the evolution equation of the level set ¢: dd> — = v(x,y)|V0| Equation (7)

[0134] This corresponds to the magnitude of pointer in FIG. 4 and the moving curve is normally represented by the zero level set function which is < / >(x,y, 0) in this case. This level set function is linked to prediction from the model by <p(x,y,0) = 0.5 - Softmax(fs(-')'). Gs is used as stationary boundary which can be obtained by convolving the ground truth yn with a two-dimensional Gaussian function.

[0135] The unit tangent vectors, Tf and and the 2D delta function of the stationary boundary yt and the moving boundary y2 can be separately obtained as: = k and 62(Y1) = IVGs|8(z) Equation (8) = x k and S2 Cyz) = 82(4>)|V4>|8(z) Equation (9)

[0136] Here, 6(z) is the delta function of z, and k is a unit vector along the z-axis. As y = {Yi u Yz} Equations (8) and (9) can be linearly combined for 62(y)t in Equation (5): Vx w = V(GS + aH(4>)) x k8(z) = 62(Y)t Equation (10)

[0137] Where H($) = 62(4>)V(4>) as HQ) is a regularised Heaviside function and a is a hyperparameter to control the strength of the dynamics in the moving boundaries. In detail, the regularised Heaviside function is accompanied by a factor p: fl 0<-Z? sin 0 > Equation (11)

[0138] For calculation efficiency, a HardTanh function is used with (min, max) = (0,1) to approximate the normally used regularised function for image smoothing. Due to the property of w and the position of the image plane 400 (z=0), it can be proved that = w2 = 0. Therefore, Equation (6) can be simplified with Equation (9) as: _ - Vc|> x k V0 Equation (12)

[0139] Substituting equation (10) into dl and Equation (1) into Equation (12), the velocity 406 of the moving boundary 402 in Equation (7) can be calculated to provide Equation (3): If r-V(Gs+aH^)) v(x,y) = - — ----------dxdy |r|3 Equation (3)

[0140] The total energy can be derived from Equation (3), and, taking Gs + aH($) = TQx.y) derives Equation (4): 1 C f VT(x,y) ■ VT(x',y') --------Pi-------- Equation (4)

[0141] Herein, it is noted that Equation (4) represents the supervised loss function (Lsup), that is, energy loss 124, L, shown in in FIG. 1 or FIG. 2, with deep supervision, as described above. Equation (3) as above, is the corresponding gradient of the loss function.

[0142] In some examples, the gradient of the loss function (e.g. Equation (3) above) may be backpropogated at each of the nodes of the network to update an assigned weighting of each respective node. That is, in a machine learning networks, weights or parameters are assigned to nodes (neurons) during the initialisation phase. Herein, in the described examples (e.g. with reference to models 100 of Fig. 1 or model 200 of Fig. 2) these weights, or parameters, may be updated every time after a new loss value is calculated. The weights are assigned to convolutional filters or kernels. In examples of the present disclosure, the backpropagation and weight update are performed at each node of the model.

[0143] In some examples, the model may be performed (that is, the steps or processes of the model performed, or trained, in either model 100 or model 200) in mixed precision assigned memory, where memory size may be differently allocated. In particular, 3D(T+) UNet++ (model 200) is a very dense, quasi-3D network, and requires high computational resource. Performing, or training, the model with single precision in 32-bit floating point (FP32) may require large memory usage. Therefore, in some examples, steps of the image segmentation model (e.g. model 200) may be performed in half tensor, by assignation of a memory size of 16 floating points. Minimisation of the energy loss function may be performed in full tensor, by assignation of a memory size of 32 floating points. That is, only the loss calculation is kept in FP32, whereas all other components operate in 16-bit (FP16). In this example, gold standard segmentation masks, yn, are kept in FP 32 and the predicted output fs () is converted back to FP32 from FP16 during loss calculation.

[0144] Referring back to FIG. 1 and FIG. 2, aspects of the present disclosure further relate to methods of training a machine learning image segmentation model (e.g. model 100, and / or model 200) for (identifying and) segmenting structural features of a vessel from image frames (e.g. a sequence, or series, of contrast x-ray image frames).

[0145] Referring to model 100, or model 200, the method of training the model comprises receiving a labelled training set for the machine learning segmentation model. For example, input 110 (X) in either FIG. 1, or FIG. 2 can relate to a labelled training set. In particular, the labelled training set may comprise a plurality of images (sequential contrast x-ray image frames), such as obtained during ICA. In examples, the labelled training set, or input 110, may be termed a gold standard, that is, relate to one or more manual annotations and / or identification of vessels, or vessel structural features, within the plurality of images.

[0146] Herein, the steps performed fortraining the segmentation model in accordance with FIG 5A and FIG. 5B, correspond to the disclosure of the model as set out in respect of FIG. 1 and / or FIG. 2, respectively, as above, wherein the labelled input (gold standard) is used as input 110. For completeness, reference to like terms and steps performed in accordance with FIG. 1 to FIG. 4 are not repeated herein, but will be appreciated to similarly apply. That is, the architecture, blocks, steps and processes performed are referenced herein in respect of model 100 and 200.

[0147] FIG. 5A illustrates a method 500a of training a machine learning image segmentation model (e.g., model 100). Herein, the model 100 may comprise a 3D encoder and 2D decoder, and wherein the 3D encoder and 2D decoder are connected by at least one interlinked 2D convolution node and a plurality of temporal extraction nodes therebetween.

[0148] Referring to FIG. 5A, at step 502, training the image segmentation model comprises using the received labelled input 110 (e.g. a labelled training set), as an input to the encoder (e.g. encoder 102).

[0149] At step 504, an output of the encoder 102 may be provided to a plurality of temporal extraction nodes 108 to extract temporal information (e.g., in accordance with FIG. 3C, and as described with reference to Fig. 1 and Fig. 2).

[0150] At step 506, the plurality of temporal extraction nodes may generate a 2D temporal output (that is, a temporal output, or 2D feature map, squeezed or compressed in the temporal axis) based on the extracted temporal information, and at step 508, provide the generated 2D temporal output to the at least 2D convolution node (e.g. interlinked convolution node 106) and 2D decoder.

[0151] At step 510, the method comprises generating, at the decoder 104, a combined temporal output based on an output of the at least one interlinked 2D convolution node 106 and at least one temporal extraction node, wherein the combined temporal output represents a predicted segmentation of the vessel tree.

[0152] At step 512, the method may output the trained image segmentation model usable for identifying and segmenting structural features of a vessel tree from image frames.

[0153] FIG. 5B illustrates a method 501a of training a machine learning image segmentation model (e.g, model 200). Herein, model 200 comprises a 3D encoder and 2D decoder, and wherein the 3D encoder and 2D decoder are connected by at least one interlinked 3D convolution node and a plurality of temporal extraction nodes therebetween.

[0154] At step 520, training the image segmentation model comprises using the received labelled input 110 (e.g. a labelled training set), as an input to the encoder (e.g. encoder 102).

[0155] At step 522, an output of the encoder 102 may be provided as an input to at least one interlinked 3D convolution nodes 206 and at least one temporal extraction nodes 108. Herein, an output of the at least one interlinked 3D convolution node is provided as an input to at least one temporal extraction node 108.

[0156] At step 524, the plurality of temporal extraction nodes 108 may extract, by each of the plurality of temporal extraction nodes, temporal information of the received input from the 3D encoder or at least one interlinked 3D convolution node. At step 526, the plurality of temporal extraction nodes 108 may generate a 2D temporal output (that is, a temporal output, or 2D feature map, squeezed or compressed in the temporal axis) based on the extracted temporal information from the 3D encoder and at least one interlinked 3D convolution node.

[0157] At step 528, the method comprises generating, at the decoder 104, a combined temporal output based on an output (e.g. the 2D temporal output) of the plurality of temporal extraction nodes 108, wherein the combined temporal output represents a predicted segmentation of the vessel tree.

[0158] At step 530, the method may output the trained image segmentation model usable for identifying and segmenting structural features of a vessel tree from image frames.

[0159] FIG. 6 represents a method 600 of automatically identifying and segmenting structural features in a vessel tree in a plurality of image frames (e.g. sequential contrast x-ray image frames), in accordance with examples of the present disclosure.

[0160] At step 602, the method comprises receiving a series of image frames.

[0161] At step 604, the received image frames are provided to a trained machine learning image segmentation model (e.g., trained in accordance with the steps of FIG. 5A or FIG. 5B) for automatically identifying and segmenting structural features of a vessel tree from the image frames. That is, the model may be trained in accordance with the above described examples, and may represent either of model 100 or model 200, as described in relation to FIG. 1 and FIG. 2.

[0162] At step 606, the trained model provides an output representative of the segmented structural features of the vessel tree from the plurality of images. In examples, this output may comprise one or more images representing the segmented structural features of the vessel tree.

[0163] FIG. 7 illustrates a computer readable medium having stored thereon computer-readable code representative of a machine learning image segmentation model in accordance with examples of the present disclosure.

[0164] That is, a computer readable medium 702 may comprise, or store, code representative of the model 706 (e.g. model 100 and / or model 200), and / or code to perform the method described above with reference to FIG. 6. The stored model 706 may have been trained in accordance with the examples detailed above, and with reference to FIG. 5A and FIG. 5B. The model 706 may be executed by means of a processor 704 in connection with the computer readable medium 702.

[0165] The following examples hereby describe experimental settings and results associated with the examples and embodiments discussed above.

[0166] To evaluate the results (e.g. on the predicted segmentation output of model 100 or model 200), the following post-processing steps may be performed:

[0167] Skeletonisation: The gold standard segmentation yn and predicted segmentation fs() may be skeletonised for comparison. A skeletonisation method o() is applied which includes a clockwise pixel assessment of a 3x3 mask through the image.

[0168] Denoisinq: The output fs( ) may undergo a denoising operation conditioned by the size of the pixel set which is classified as vessels. The number of pixels in every blob of vessel area is counted, and then the blob with a number of pixels less than a certain threshold is removed.

[0169] The following evaluation methods may be performed:

[0170] Conventional Metrics: To evaluate performance, in the examples shown, conventional metrics are used (Recall, Precision, and Dice) which use the same 4-frame ICA sequence data.

[0171] Skeletonisation Metrics: The evaluation of skeletonised results is done on two metrics, referred to as Completeness, Cr, and correctness, Cp, and broadly correspond to buffered versions of Recall and Precision on the skeleton.

[0172] FIG. 8 represents skeletisation metrics for vessel centreline with 1 pixel width. True positives (TP), and False Positives (FP), and False Negatives (FN), are indicated. The definitions of True / False Positives / Negatives are changed, as illustrated in FIG. 8. As the vessel skeleton is only 1-pixel width, the buffered width is set on both sides to one pixel, so that the buffered skeleton is 3 pixels wide. To obtain FP and TP, only a(yn) is buffered to compare with o(fs()). The part that sits within the buffered o(yn) is considered as TP and vice versa for FP. FN is equivalently obtained, buffering a(fs()) only. The buffering of skeletons is performed by dilation of the single pixel structure.

[0173] Experimental Results

[0174] Training of the network (e.g. model 100 or model 200 in accordance with FIG. 1 or FIG. 2) may be performed on a suitable training apparatus (e.g. processing unit). Frames may be augmented with flipping, saturation and contrast changes. Random rotation (e.g. 90 degrees) for 0 or more times may be applied (e.g. by applying probability 0.7). Trainings in full tensor may be optimised with an optimisation algorithm (e.g. Adam), for example, using learning rate =10 5 Pi=0.85, weight decay = 10 5 and all other hyper-parameters set as default in the framework (e.g. Pytorch).

[0175] In an example, training in half tensor may be optimised with Stochastic gradient descent SGD) in view of dynamic range of FP16, and a cosine annealing schedular applied to gradually decrease the learning rate from 10 6 to minimum value 10 7 with respect to the number of epochs. The weight decay may be decreased to 10-6 and momentum set to 0.9 for optimisation with SGD. For all training, the value of a in the loss function, E (equation 5) is 0.35. The training may last for 2000 epochs to assure convergence of the loss function.

[0176] 2D vs 3D vs 3D+

[0177] To validate the effect of temporal information for multi-frame ICA segmentation, three different network architectures, 2D, 3D(T) and 3D(T+) UNet++ are trained and compared. Herein, 3D(T) and 3D(T+) are represented by FIG. 1 and FIG. 2, models 100 and 200, and reference is made to the description of such. As 3D(T+) has a large network size, and batch size in the results is 4. 3O(T) UNm* I 13D(W) LW** 8O5AO3 | 8X43x04 | Ail tables present mean ± AodsO deviation. Table 1

[0178] Table 1 illustrates how introduction of temporal information can result in the performance improvement of 1% on Dice and maximum of 2% on Recall and Precision with generally lower standard deviations on all three metrics even at low batch size. 3D(T) UNet++ produces the best overall results with highest Dice and Precision obtained. 2D UNet++ has one very low outlier in Recall, about 0.49, and a much larger minimum-maximum range. 3D(T) UNet++ produced no outliers, suggesting that the introduction of the temporal component (e.g. temporal extraction nodes 108) improves results and generalisation ability of the network.

[0179] FIG. 9 shows a qualitative evaluation of segmentation output sample using 2D, 3D(T) and 3D(T+) UNet++ with comparable number of parameters.

[0180] Referring to FIG. 9, from the arrows shown (1st row), the 3D(T) UNet++ yields a good delineation on thick vessels, and also accurately defines small bifurcations and the distal part of the coronary vessel tree, which are key areas for preserving the topology. Considered also in terms of computing efficiency and performance, 3D(T) UNet++ shows improved architecture for temporal vessel segmentation in ICA sequences. Within the dashed boxes, it can be seen that all network generated segmentations have a similar structure, but vary from the gold standard. However, by zooming into the gold standard on the second row (e.g. see lower arrow), it can be seen that there are also limitations in this (labelled) dataset, which may affect evaluation metrics. That is, the annotations in this dataset may be considered as coarse-grained labels.

[0181] Loss function

[0182] For evaluation from the energy loss function, training was performed on 3D(T) UNet++ with simple Dice loss and with energy loss function L respectively. Reference may be made herein to model 100 of FIG. 1. In some examples, the optimal batch size for 3D(T) UNet++ was found to be 6. | Modes tunicm Wo t>S 3D(T) UNd-n- w DS | SVS-serU^ 83.2712.57 83.4112.45 84.2815.31 84.6014.45 83.4914.4t 84.3114.62 wins. 13 83.5115.65 183.5215.5417071713 ! 83.9616.04 184.5715.93 176.5517.29 s 84.92105 | N / A I N / A J Table 2: Comparison of Deep Supervision and Loss Functions.

[0183] Fig. 10 represents a qualitative evaluation of segmentation and skeleton output samples using 3D(T)UNet++ with Dice loss and energy loss function.

[0184] Referring to Table 2, first two rows, and FIG. 10, the energy loss function improves Precision by about 3% and Dice by about 1%. The Recall with Dice loss is slightly higher, but may be seen to be caused by over segmentation of vessel boundaries, e.g. in the zoomed area (shown in arrows on Fig. 10). Dice loss causes disconnections, which not only affects the geometry but also yields about 1.5% lower Cr for skeletonised vessels shown in Table 2, even though the Recall is higher. CPj as the buffer version of Precision for skeletonisation, shows lower values compared to Precision as FPs normally occur in relatively thin vessels. During skeletonisation, thicker vessels lose more pixels compared to thin vessels, thus increasing FP / TP ratio and resulting in a drop in Cp. When using the energy loss function, pixels may be identified as vessels not in the labelled input (gold standard). In some cases, these vessels may have been missed in the labelled input manual annotation (e.g. boxed area of FIG. 10), which may show the larger drop from Precision to Cp.

[0185] Deep Supervision

[0186] Referring back to FIG. 1 and FIG. 2, models 100 and 200 may include deep supervision. The effect of this inclusion is shown in the last three rows of Table 2 (DS), comparing 3D(T) UNet++ without deep supervision, 3D(T+) UNet++ with deep supervision, and a state-of-art method SVS-net. Using deep supervision, the performance improvement only differs in the mean value where Recall increases about 1% and Dice is slightly higher. A similar trend can be found for skeletonised vessels in Table 2, where the same network architecture with deep supervision yields higher Cr. Comparing to SVS-net, the models 100, 200, achieve half the standard deviation in both Dice and Recall. In consideration of coarse-grained labels, higher Recall and comparable Dice and Precision compared to the state-of-the-art method on the same dataset were achieved.

[0187] FIG. 11 illustrates vessel segmentation performance using 3D(T) UNet++ with deep and without deep supervision (with same colour code). FIG. 11 qualitatively evaluates the effects of deep supervision. 3D(T) UNet++, referring to model 100, is shown without deep supervision, 1102, and with deep supervision 1104. For completeness, although FIG. 11 refers to 3D(T) UNet++, model 100, deep supervision may similarly be used for 3D(T+) as shown in model 200, above. Herein, FIG. 11 merely represents an evaluation of an example of deep supervision.

[0188] Referring to FIG. 11, for segmentation for both vascular trees of skeletons, deep supervision fills the missed vessels (FN) with correct delineation (TP). FP represent potentially fake vessels (shown at the end of the vessel tree). Two of the missed distal vessels (e.g. without DS 1002) and one disconnection are all enhanced with the use of deep supervision with DS 1104, with potential FP only occurring at the end of some vessels but not producing thicker (inaccurate) vessel boundaries.

[0189] Ablation Study

[0190] Re-segmentation

[0191] Visual analysis of segmentation results highlighted incompleteness in available (Coarse grained) segmentation. A complete annotation of samples from a dataset (e.g. 10) was performed. To prevent bias, all test set images were ranked based on the Dice values of their 3D(T) UNet++ result (Table 1) and selected on 0, 10, 20, 30, 40, 50, 60, 75, 85, 100 percentiles. An example of the distribution is shown in FIG. 13. This was termed the “New Gold standard”, or “fine grained Gold Standard”. This new gold standard was compared with segmentation results and the previous ("coarse-grained", old) gold standard.

[0192] FIG. 12 shows 3 of the 10 re-segmented samples with minimum, median and maximum Dice values. Minimum value 1202 = 81.61%, median Dice 1204 = 84.69% and maximum Dice 1006 = 94.76%, and shown top to bottom (1202 - 1206, respectively). It is shown that the level of completeness of the initial annotations (e.g. a labelled input, or gold standard) varies significantly between samples, having an effect on the calculated, evaluative, Dice values.

[0193] FIG. 13 illustrates the cumulative distribution of the Dice coefficient scores for all tests samples from the 3D(T) UNet++ result in Table 1, which averaged to be 82.65%. These Dice values are calculated on the original manual segmentation from the external dataset. The samples at 0, 10, 20, 30, 40, 50, 60, 75, 85, and 100th percentile were selected to represent the test samples from varying segmentation quality by the 3D(T) UNet++. The 10 samples are shown in the bars of FIG. 13, with the Dice values shown on the top of each bar. These 10 samples are resegmented with the differences between them, and the original (coarse-grained) gold standard are presented in FIG. 14 in ascending order of the Dice values which are listed at the end of every image row. This Dice value is calculated using re-segmentation as gold standard and coarsegrained manual segmentation as generated output.

[0194] FIG. 14 illustrates the re-segmented samples from the test dataset, with ascending Dice values. From left to right, the ICA frame, original gold standard, new gold standard, and the difference in segmentation, are shown. In FIG. 14 the percentile data corresponds to the bars of FIG. 13, stating the original location of selected samples to be re-segmented in the cumulative distribution.

[0195] Corresponding evaluation Results

[0196] Further testing and evaluation were performed with no additional training, using the new gold standard.

[0197] FIG. 15 illustrates results of a qualitative evaluation of re-segmentation and skeleton output sample (same colour code). | Model Dice (%) Recall (%) Precision (%) Q(%) Cp (%) I SVS-net

[19] 79.84+4.25 74.38+8.49 87.08+4.29 66.38+7.89 85.74+4.39 Old gold standard 87.66+5.30 85.44+7.95 90.32+4.05 76.81 ±11.24 91.35+5.08 | 3DCD UNet++ w DS 86.19+1.94 86.26+5.24 86.44+3.27 84.00+5.25 87.55+6.17 Table 3: Comparison of segmentation quality on 10 re-segmented samples

[0198] The results are also shown in Table 3, and compared to SVS-net method (downloaded from https: / / github.com / Binjie-Qin / SVS-net). It can be seen that for vascular trees, the model surpasses SVS-net by approximately 6% in Dice and 12% in Recall. The results show the network (e.g. Model 100) generated higher recall compared to the old gold standard (which, may in some examples correspond to the labelled input of the training set for model 100), whereas the difference in Dice is only about 1.5%. The advantage is increased in skeletonised results for preserving vascular geometries, wherein the method produces the highest Cr of 84.00% and the smallest drop between Recall and Cr. The correctly segmented vessels in the currently described method have the least disconnection and shift. Qualitatively, with reference to FIG. 15, fewer FNs, more TPs, and comparable FPs are generated.

[0199] Aspects and examples of the present disclosure relate to a deep learning framework, referring to model 100 and model 200, that utilises multiple unlabelled frames of ICA sequences for generating accurate coronary artery segmentation. The architecture integrates temporal convolution blocks, or nodes, for fusing the information on the time axis. The architecture also comprises an energy loss function for enhancing topology preservation onto a dense UNet++ that has deep supervision enabled.

[0200] In evaluating the results, the framework was trained on the same dataset using the same data split. The results show the advantage of a multi-frame approach, and also associated problems with incomplete manual annotation of vessel trees.

Claims

1. A machine learning image segmentation model for automatically identifying and segmenting structural features of a vessel tree from image frames, the model comprising a 3D encoder and 2D decoder, the 3D encoder and 2D decoder connected by at least one interlinked convolution node and a plurality of temporal extraction nodes therebetween, and wherein the model is configured to identify and segment structural features of a vessel tree from the plurality of image frames, by the model being configured to:receive a plurality of image frames as an input to the 3D encoder;provide an output of the 3D encoder as an input to the plurality of temporal extraction nodes to extract temporal information;generate, by the plurality of temporal extraction nodes, a 2D temporal output based on the extracted temporal information;provide the generated 2D temporal output to the at least one interlinked 2D convolution node and 2D decoder;generate, at the 2D decoder, a combined temporal output based on an output of the at least one interlinked 2D convolution node and at least one temporal extraction node, wherein the combined temporal output represents a predicted segmentation of the vessel; andgenerate an output representative of the segmented structural features of the vessel tree based on the predicted segmentation.

2. The machine learning image segmentation model of claim 1, wherein, to extract temporal information, each of the plurality of temporal extraction nodes of the model are configured to:apply 3D convolution to the received 3D input; anddimensionally compress the 3D input so as to produce a temporally compressed 2D feature map representing the extracted temporal information.

3. The machine learning image segmentation model of claim 1 or claim 2, wherein the at least one interlinked 2D convolution node comprises a plurality of interlinked 2D convolution nodes connecting the 3D encoder and 2D decoder, andthe plurality of interlinked 2D convolution nodes connecting the 3D encoder and 2D encoder comprise one or more skip connections therebetween.

4. The machine learning image segmentation model of any preceding claim, wherein the 3D encoder, 2D decoder, and the at least one interlinked 2D convolution node are arranged as nodes in a layered pyramidal network; andwherein the plurality of temporal extraction nodes are arranged between the respective 3D and 2D nodes of network.

5. The machine learning image segmentation model of any preceding claim, wherein the model is further configured to:minimise an energy loss associated with the predicted segmentation of the vessel based on determining a difference between the predicted segmentation and the labelled input,wherein the determined divergence is associated with an energy loss function representing an elastic energy of a system comprising the predicted segmentation and the labelled input.

6. The machine learning image segmentation model of claim 5, wherein the model is further configured to:determine the difference between a velocity of a vessel boundary of the predicted segmentation and a vessel boundary of the labelled input; andminimise the energy loss function based on the determined difference.

7. The machine learning image segmentation model of claim 5 or claim 6, wherein the 3D encoder, 2D decoder, and the at least one interlinked 2D convolution node are arranged as nodes in a layered pyramidal network, and wherein, to minimise the energy loss, the model is further configured to:combine the predicted segmentation with an output of at least one interlinked 2D convolutional node arranged in a top layer of the network to output a refined predicted segmentation; anddetermine the difference between a velocity of a vessel boundary of the refined predicted segmentation and the vessel boundary of the labelled input; and minimise the energy loss function based on the determined difference.

8. The machine learning segmentation model of claims 5 to 7, wherein the 3D encoder, 2D decoder, and the at least one interlinked 2D convolution node are arranged as nodes in a layered pyramidal network, and the model is further configured to:backpropagate a determined gradient of the energy loss function at each of the nodes in the one or more layers of the network to update an assigned weighting of each respective node,wherein the gradient of the energy loss function is associated with a velocity of a vessel boundary of the predicted segmentation.

9. A machine learning image segmentation model for automatically identifying and segmenting structural features of a vessel tree from image frames, the model comprising a 3D encoder and 2D decoder, the 3D encoder and 2D decoder connected by at least one interlinked 3D convolution node and a plurality of temporal extraction nodes therebetween, and wherein the model is configured toidentify and segment structural features of a vessel tree from the plurality of image frames, by the model being configured to:receive a plurality of image frames as an input to the 3D encoder;provide an output of the 3D encoder as an input to at least one interlinked 3D convolution node and at least one temporal extraction node,wherein an output of the at least one interlinked 3D convolution node is provided as an input to at least one temporal extraction node;extract, by each of the plurality of temporal extraction nodes, temporal information of the received 3D input from the 3D encoder or at least one interlinked 3D convolution node;generating, by the plurality of temporal extraction nodes, a 2D temporal output based on the extracted temporal information;generate, at the 2D decoder, a combined 2D temporal output based on the 2D temporal output of the plurality of temporal extraction nodes, wherein the combined temporal output represents a predicted segmentation of the vessel; andgenerate an output representative of the segmented structural features of the vessel based on the predicted segmentation.

10. The machine learning image segmentation model of claim 9, wherein, to extract temporal information, each of the plurality of temporal extraction nodes of the model are configured to:apply 3D convolution to the received 3D input; anddimensionally compress the 3D input so as to produce a temporally compressed 2D feature map representing the extracted temporal information.

11. The machine learning image segmentation model of claims 9 or 10, wherein the at least one interlinked 3D convolution node comprises a plurality of interlinked 3D convolution nodes connecting the 3D encoder and 2D decoder, andthe plurality of interlinked 3D convolution nodes connecting the 3D encoder and 2D encoder comprise one or more skip connections therebetween.

12. The machine learning image segmentation model of any of claims 9 to 11, wherein the 3D encoder, 2D decoder, and the at least one interlinked 3D convolution node are arranged as nodes in a layered pyramidal network architecture; andwherein the plurality of temporal extraction nodes are arranged between the respective 3D and 2D nodes of the network.

13. The machine learning image segmentation model of any of claims 9-12, wherein the model is further configured to:minimise an energy loss associated with the predicted segmentation of the vessel based on determining a difference between the predicted segmentation and the labelled input,wherein the determined divergence is associated with an energy loss function representing an elastic energy of a system comprising the predicted segmentation and the labelled input.

14. The machine learning image segmentation model of claim 13, wherein the model is further configured to:determine the difference between a velocity of a vessel boundary of the predicted segmentation and a vessel boundary of the labelled input; andminimise the energy loss function based on the determined difference.

15. The machine learning image segmentation model of claim 13 or claim 14, wherein, to minimise the energy loss, the 3D encoder, 2D decoder, and the at least one interlinked 3D convolution node are arranged as nodes in a layered pyramidal network, and the model is further configured to:combine the predicted segmentation with an output of at least one interlinked 3D convolutional node arranged in a top layer of the network to output a refined predicted segmentation;determine the difference between a velocity of a vessel boundary of the refined predicted segmentation and the vessel boundary of the labelled input; andminimise the energy loss function based on the determined difference.

16. The machine learning image segmentation model of claims 13 to 15, wherein the model is performed in mixed precision assigned memory, wherein the steps of the image segmentation model are performed in half tensor, by assigning a memory size of 16 floating points, except for the minimisation of the energy loss function which is performed in full tensor, by assigning a memory size of 32 floating points.

17. The machine learning image segmentation model of claims 13 to 16, wherein the 3D encoder, 2D decoder, and the at least one interlinked 3D convolution node are arranged as nodes in a layered pyramidal network, and the model is further configured to:backpropagate a determined gradient of the energy loss function at each of the nodes in the one or more layers of the network to update an assigned weighting of each respective node,wherein the gradient of the energy loss function is associated with a velocity of a vessel boundary of the predicted segmentation.

18. A method of training a machine learning image segmentation model for identifying and segmenting structural features of a vessel tree from image frames, the method comprising:receiving a labelled training set for the machine learning image segmentation model, wherein the labelled training set comprises a plurality of image frames of a vessel tree;training the image segmentation model, the model comprising a 3D encoder and 2D decoder, and wherein the 3D encoder and 2D decoder are connected by at least one interlinked 2D convolution node and a plurality of temporal extraction nodes therebetween, andwherein training the image segmentation model comprises:using the received labelled training set as an input to the 3D encoder;providing an output of the 3D encoder as an input to the plurality of temporal extraction nodes to extract temporal information;generating, by the plurality of temporal extraction nodes, a 2D temporal output based on the extracted temporal information;providing the generated 2D temporal output to the at least one interlinked 2D convolution node and 2D decoder;generating, at the 2D decoder, a combined temporal output based on an output of the at least one interlinked 2D convolution node and at least one temporal extraction node, wherein the combined temporal output represents a predicted segmentation of the vessel tree; and outputting the trained image segmentation model usable for identifying and segmenting structural features of a vessel tree from image frames.

19. A method of training a machine learning image segmentation model for automatically identifying and segmenting structural features of a vessel tree from image frames, the method comprising: receiving a labelled training set for the machine learning image segmentation model, wherein the labelled training set comprises a plurality of image frames of a vessel tree;training the image segmentation model, the model comprising a 3D encoder and 2D decoder, and wherein the 3D encoder and 2D decoder are connected by at least one interlinked 3D convolution node and a plurality of temporal extraction nodes therebetween, andwherein training the image segmentation model comprises:using the received labelled training set as an input to the 3D encoder,providing an output of the 3D encoder as an input to at least one interlinked 3D convolution node and at least one temporal extraction node,wherein an output of the at least one interlinked 3D convolution node is provided as an input to at least one temporal node;extract, by each of the plurality of temporal extraction nodes, temporal information of the received input from the 3D encoder or at least one interlinked 3D convolution node; andgenerating, by the plurality of temporal extraction nodes, a 2D temporal output based on the extracted temporal information;generating, at the 2D decoder, a combined 2D temporal output based on the 2D temporal output of the plurality of temporal extraction nodes, wherein the combined temporal output represents a predicted segmentation of the vessel tree; andoutputting the trained image segmentation model usable for identifying and segmenting structural features of a vessel tree from image frames.

20. A method of automatically identifying and segmenting structural features of a vessel tree in a plurality of image frames, the method comprising:receiving a plurality of image frames;providing the received images to a trained machine learning image segmentation model for automatically identifying and segmenting structural features of a vessel tree from image frames; andreceiving, from the trained model, an output representative of the segmented structural features of the vessel tree of the plurality of image frames.

21. The method of claim 20, wherein the trained model is trained according to the method of any one of claims 18 or 19.

22. A trained image segmentation model for automatically identifying and segmenting structural features of a vessel tree from image frames, wherein the model is trained in accordance with any of claims 18 or 19.

23. A computer-readable medium having stored thereon computer-readable code representative of a machine learning machine segmentation model in accordance with any of claims 1-8, 9-17, or claim 22.

24. A computer-readable medium having stored thereon computer-readable code representative of an image segmentation model trained in accordance with any of claims 18 or 19.

25. A computer-readable medium having stored thereon computer-readable code executable so as to perform the method in accordance with any of claims 20 to 21.Application No: GB2402400.2Claims searched: 1-8, 18, 22-24Examiner:      Mr Charles DurdenDate of search: 21 June 2024Patents Act 1977: Search Report under Section 17Documents considered to be relevant:Category Relevant to claims Identity of document and passage or figure of particular relevance X 1-5, 18, 22-24 Neural Networks, Vol. 128, August 2020, Hao et al., "Sequential vessel segmentation via deep channel attention network", pages 172-187, see especially page 174 column 1, sections 2.1 &2.7, fig. 1 Available from httDs: / / www. sciencedirect.com / science / article / abs / oii / S08936080203016 A - 72 [accessed 18th June 2024] WO 2020 / 146905 Al (LIGHTLAB IMAGING, INC.) - see eg. paragraph 159 A - US 2021 / 0219850 Al (SIEMENS HEALTHCARE GMBH) - see eg. paragraphs 10-14, 33 A - CN 110443813 A (TENCENT MEDICAL HEALTH (SHENZHEN) CO., LTD.) - see eg. "Summary of the Invention"Categories:X Document indicating lack of novelty or inventive step A Document indicating technological background and / or state of the art. Y Document indicating lack of inventive step if p Document published on or after the declared priority date but combined with one or more other documents of same category. before the filing date of this invention. & Member of the same patent family E Patent document published on or after, but with priority date earlier than, the filing date of this application.Field of Search:Search of GB, EP. WO &US patent documents classified in the following areas of the UKCX :Worldwide search of patent documents classified in the following areas of the IPC____________A61B; G06N; G06T; G06V________________________________________The following online and other databases have been used in the preparation of this search report SEARCH - PATENTSInternational Classification:Subclass Subgroup Valid From G06T 0007 / 10 01 / 01 / 2017 G06N 0003 / 0464 01 / 01 / 2023 G06N 0020 / 00 01 / 01 / 2019 G06T 0007 / 00 01 / 01 / 2017 G06V 0010 / 26 01 / 01 / 2022 G06V 0010 / 70 01 / 01 / 2022

Images