Segmentation of medical images

Abstract

A method is disclosed for generating a segmentation mask for a medical image. The method comprises detecting 210 and disk encoding 220 user input clicks on the medical image and encoding the medical i

Description

Technical Field The present disclosure relates to a computer implemented method for generating a segmentation mask for a medical image. The present disclosure also relates to a segmentation node, and to a computer program product configured, when run on a computer, to carry out a method for generating a segmentation mask for a medical image. Background Interactive image segmentation strives to produce precise mask predictions with minimal human interaction, leading to various practical applications, including pixel-level annotation and image manipulation. Segmentation of medical images may involve delineation of internal organs, including for example a target of radiotherapy treatment, and neighbouring organs at risk. Various types of user interaction have been explored to achieve image segmentation, including click

[44] , scribble

[49] , polygon [1], bounding box

[51] , and their combinations

[53] , Among these approaches, click-based interactive segmentation has gained significant attention owing to its simplicity and well-established simulation and evaluation strategies. In click-based interactive segmentation, the typical paradigm involves converting both positive and negative clicks ( / .e., positional indications for foreground and background) into disk maps. These disk maps are then concatenated with the input image and fed into a segmentation model to predict the target mask. Some methods [8] incorporate the predicted mask from the previous interaction step to achieve coarse mask correction. Based on this paradigm, various works have been proposed to minimize the interaction rounds, such as local refinement [8, 27], click attention

[28] , pseudo clicks

[31] , and model adaptation [17, 21,43], These methods typically adopt a carefully designed neural network to predict a single target mask based on user clicks. While these methods have shown significant performance improvement, they still suffer from the issue of target ambiguity [23, 48], Specifically, in complex scenes, a single click may correspond to multiple target objects, making it difficult for these methods to accurately identify the intended target. Figure 1 illustrates a common challenge in target selection according to which one click in complex scenes may indicate multiple target objects. Referring to Figure 1, if a user were to click on the jacket of the person in the image, it is unclear whether the intention of the user was to select the jacket, the entire person, or the combination of the person and bicycle. Some works, such as [23, 24], have attempted to address the target ambiguity issue by introducing multiple neural networks to generate diverse segmentation masks. For instance,

[23] proposes to generate diverse segmentation masks by multiple convolutional networks, which is followed by a selection network to select one of the synthesized masks. 47 Unfortunately, these methods require multiple segmentation networks or post-processing techniques such as non-maximum suppression (NMS) to achieve decent performance, which is not efficient. Furthermore, in existing methods where multiple segmentation networks are employed, the generated masks are often independent, lacking implicit information exchange between them. As a result, these methods tend to produce similar masks, which provide limited priors for addressing the challenge of target ambiguity. Summary It is an aim of the present disclosure to provide a method, a segmentation node, and a computer program product which at least partially address one or more of the challenges mentioned above. It is a further aim of the present disclosure to provide a method, a segmentation node, and a computer program product which cooperate to generate a segmentation mask for a medical image, while minimising target ambiguity in user interaction during the course of the generation of the segmentation mask. According to a first aspect of the present disclosure, there is provided a computer implemented method for generating a segmentation mask for a medical image. The method comprises detecting user input clicks on the medical image, disk encoding the detected user clicks, and encoding the medical image by submitting the disk encoded user clicks, the medical image, and a reference segmentation mask to an image encoder Machine Learning (ML)model, wherein the image encoder ML model is operable to generate a plurality of image embeddings. The method further comprises initiating a plurality of click prompts, wherein each initiated click prompt comprises a learnable prompt token, a positional encoding of user clicks, and a content encoding of user clicks. The method further comprises updating the initiated click prompts using the plurality of image embeddings, generating, from the updated click prompts and the plurality of image embeddings, a plurality of intermediate segmentation masks, and determining a transport plan for Optimal Transport from the updated click prompts to the plurality of image embeddings. The method further comprises updating the plurality of intermediate segmentation masks using the determined transport plan, and inputting the plurality of updated intermediate segmentation masks and the plurality of image embeddings to a decoder ML model, wherein the decoder ML model is operable to generate a predicted segmentation mask for the medical image. According to another aspect of the present disclosure, there is provided a computer program product comprising a computer readable medium, the computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform a method according to any one or more aspects or examples of the present disclosure. According to another aspect of the present disclosure, there is provided a segmentation node for generating a segmentation mask for a medical image. The segmentation node comprises processing circuitry configured to cause the segmentation node to detect user input clicks on the medical image, disk encode the detected user clicks, and encode the medical image by submitting the disk encoded user clicks, the medical image, and a reference segmentation mask to an image encoder ML model, wherein the image encoder ML model is operable to generate a plurality of image embeddings. The processing circuitry is configured to cause the segmentation node to initiate a plurality of click prompts, wherein each initiated click prompt comprises a learnable prompt token, a positional encoding of user clicks, and a content encoding of user clicks. The processing circuitry is configured to cause the segmentation node to update the initiated click prompts using the plurality of image embeddings, generate, from the updated click prompts and the plurality of image embeddings, a plurality of intermediate segmentation masks, and determine a transport plan for Optimal Transport from the updated click prompts to the plurality of image embeddings. The processing circuitry is configured to cause the segmentation node to update the plurality of intermediate segmentation masks using the determined transport plan, and input the plurality of updated intermediate segmentation masks and the plurality of image embeddings to a decoder ML model, wherein the decoder ML model is operable to generate a predicted segmentation mask for the medical image. According to another aspect of the present disclosure, there is provided radiotherapy treatment apparatus comprising a segmentation node according an aspect or example of the present disclosure. Aspects and examples of the present disclosure thus provide a segmentation method, segmentation node, computer program and radiotherapy treatment apparatus which mitigate the issue of target ambiguity in interactive segmentation by introducing a plurality of click prompts to capture different latent targets. Click prompt learning with optimal transport is used to generate multiple intermediate masks, which serve as strong and diverse priors for final mask prediction. As discussed in greater detail below, extensive experiments validate the superiority of an implementation of the methods disclosed herein over state-of-the-art methods on eight benchmarks. Additionally, the tested implementation exhibits strong generalization capabilities in the challenging medical domain. Brief Description of the Drawings For a better understanding of the present disclosure, and to show more clearly how it may be carried into effect, reference will now be made, by way of example, to the following drawings in which: Figure 1 illustrates the issue of target ambiguity in image segmentation; Figure 2 is a flow chart illustrating process steps in a computer implemented method for generating a segmentation mask for a medical image; Figures 3a to 3c show a flow chart illustrating process steps in another example of a computer implemented method for generating a segmentation mask for a medical image; Figure 4 is a block diagram illustrating functional modules in an example segmentation node; Figure 5 is a block diagram illustrating an overview of an example implementation of methods disclosed herein; Figure 6 shows Table 1, which is a Number of Clicks comparison of the example implementation with existing methods; Figure 7 shows Table 2, which is an out-of-domain evaluation of the example implementation on three medical benchmarks; Figure 8 shows Tables 3, 4, 5 and 6, which provide further experimental results for the example implementation; Figures 9a to 9d illustrate convergence analysis of the example implementation and existing methods with respect to different benchmarks; and Figures 10a to 10c illustrate segmentation results on challenging and normal cases. Detailed Description As discussed above, click-based interactive segmentation aims to segment target objects with user clicks as guidance. Existing methods mainly adopt general semantic segmentation pipelines and achieve promising performance. However, the issue of target ambiguity, i.e., the possibility that a single click on a complex scene may correspond to different target objects, is still challenging for interactive segmentation, and particularly in the field of segmentation of medical images. In order to address target ambiguity, the present disclosure proposes a new method: click-prompt learning with optimal transport, an example implementation of which is discussed in detail below, and referred to in the present disclosure as CPIot. Examples of the new method proposed herein first encode input clicks as multiple click prompts, before matching these click prompts with image embeddings through optimal transport (OT). This matching process using OT allows each click prompt to efficiently focus on a specific latent target. Additionally, examples of the method disclosed herein achieve prompt-pixel alignment: aligning click prompts and image embeddings to facilitate the learning ofclick prompts. Optimized click prompts together with image embeddings may then be used to generate multiple intermediate masks, which serve as strong and diverse priors for guiding final mask prediction. Extensive experiments have been conducted using the example method implementation CPIot and eight interactive segmentation benchmarks, and improvement has been demonstrated over the benchmarks. According to examples of the method disclosed herein, multiple click prompts may thus be used to generate multiple intermediate masks that correspond to latent targets. These intermediate masks serve as strong and diverse priors for final mask prediction, thus addressing the issue of target ambiguity in an effective manner. As discussed in greater detail below with reference to the accompanying Figures, example methods according to the present disclosure first initialize click prompts in a click encoder by conditioning on both position and content information of positive clicks. Subsequently, in order to learn multiple click prompts focusing on different latent targets, two consecutive strategies are proposed: prompt-pixel alignment and pixel-to-prompt optimal transport. In prompt-pixel alignment, click prompts interact with image embedding, for example through a transformer network, to mine diverse semantic clues. Each click prompt starts from click indications to gradually learn overall information about a corresponding latent target. In the pixel-to-prompt optimal transport, click prompts are further refined by employing optimal transport to align them with the image embeddings. This may be achieved in some examples by defining a cost function that measures the similarity between the click prompts and the image embeddings. The optimal transport has the effect of pushing each click prompt to focus on a specific target. Finally, optimized click prompts, along with image embedding may be used to generate multiple intermediate masks, which serve as strong and diverse priors to guide the final mask predictions effectively, and so mitigate the issue of target ambiguity. In order to provide additional context for the example methods disclosed herein, there now follows a discussion of concepts relevant to interactive segmentation methods, prompt learning, and Optimal Transport. Interactive segmentation aims to achieve high-quality segmentation by incorporating iterative user interaction as guidance. Early works [40, 13, 14, 19] primarily adopt optimization-based graphical models built on low-level image features. However, these methods often struggle in complex environments, including for example occlusions and diverse backgrounds. Recently, deep learning-based methods [23, 25, 51] have shown considerable improvements by converting user clicks into click maps and processing them with the image in a neural network. Considering that coarse mask predicted from the last interaction contains object information, RITM

[44] and 99%AccuracyNet

[11] feed previous a mask to the network to make more accurate mask predictions. To better excavate click information, FCANet

[28] emphasizes the critical role of the first click and designs a first-click attention for better segmentation. PseudoClick

[31] proposes a clickimitation mechanism to generate pseudo next-click to reduce interactive rounds. From the perspective of model adaptation, some works, e.g. BRS

[17] , f-BRS

[43] , and CA

[21] adopt loss backpropagation or model parameter adaptation to perform mask refinement for testing images. Meanwhile, FocalClick [8] and FocusCut

[27] adopt similar local refinement strategies to correct desired local regions for high-quality segmentation. Recently, iSegFormer

[30] and SimpleClick

[29] have applied vision transformer to the interactive segmentation, and show promising performance. GPCIS

[57] introduces Gaussian Process

[42] into the model to perform effective mask refinement. It is worth noting that [23,24] proposes to synthesize a diverse set of plausible segmentation masks and select from these, aiming to tackle target ambiguity. Prompt learning has emerged as a promising method in Natural Language Processing (NLP) for efficiently adapting pre-trained large language models to downstream tasks. Rather than fine-tuning all network parameters for each downstream task with a taskspecific objective function, prompt learning formulates the downstream fine-tuning problem by training lightweight textual prompts that provide task-specific guidance to the model [37,18, 33]. The use of prompts reduces the domain shift between the pre-trained task and the downstream application, making it easier to adapt the pre-trained knowledge to downstream tasks [5], Recently, learnable textual prompts have been introduced into Vision-Language Models (VLM) [56, 39] to perform various visionlanguage tasks, such as referring image segmentation

[47] and zero-shot segmentation

[22] . Visual Prompt Tuning (VPT) is also a prompt learning method, which introduces a small number of learnable image prompts to a pretrained visual encoder for efficient adaptation to various downstream vision tasks [52, 45], In the present disclosure, the concept of prompt learning is introduced into interactive segmentation, using learnable prompts comprising click information to capture latent target objects. Optimal Transport (OT)

[35] is a mathematical framework to learn the correspondence between two probability distributions. OT theory involves finding the most efficient way to transport one distribution to another while satisfying certain constraints, such as preserving the total mass of the distributions for example. Owing to the property of distribution matching

[38] , OT has been applied in various computer vision tasks, including generative models [41, 54], graph matching

[50] , image matching [55, 32], and domain adaptation [6], Among various methods, the Sinkhorn algorithm can efficiently solve the OT problem through entropy-regularization [9], and it can be directly applied to deep learning frameworks thanks to the extension of Envelop Theorem

[38] . In the present disclosure, the concept of optimal transport is introduced into interactive segmentation to match multiple click prompts with image embedding, thus allowing each click prompt to focus efficiently on a latent target. Figure 2 is a flow chart illustrating process steps in a computer implemented method 200 for generating a segmentation mask for a medical image. The method may be performed by a segmentation node, which may comprise a physical or virtual node, and may be implemented in a computer system, treatment apparatus, such as a radiotherapy treatment apparatus, radiotherapy treatment planning system or apparatus, computing device, or server apparatus, and / or may be implemented in a virtualized environment, for example in a cloud, edge cloud, or fog deployment. Examples of a virtual node may include a piece of software or computer program, a code fragment operable to implement a computer program, a virtualised function, or any other logical entity. The segmentation node may encompass multiple logical entities, as discussed in greater detail below. Referring to Figure 2, the method 200 comprises detecting user input clicks on the medical image in step 210, disk encoding the detected user clicks in step 220, and, in step 230, encoding the medical image by submitting the disk encoded user clicks, the medical image, and a reference segmentation mask to an image encoder Machine Learning (ML) model, wherein the image encoder ML model is operable to generate a plurality of image embeddings. In step 240, the method 200 further comprises initiating a plurality of click prompts, wherein each initiated click prompt comprises a learnable prompt token, a positional encoding of user clicks, and a content encoding of user clicks. The method 200 further comprises updating the initiated click prompts using the plurality of image embeddings in step 250, generating, from the updated click prompts and the plurality of image embeddings, a plurality of intermediate segmentation masks in step 260, and, in step 270, determining a transport plan for Optimal Transport from the updated click prompts to the plurality of image embeddings. The method 200 further comprises updating the plurality of intermediate segmentation masks using the determined transport plan in step 180. Finally, in step 290, the method 200 comprises inputting the plurality of updated intermediate segmentation masks and the plurality of image embeddings to a decoder ML model, wherein the decoder ML model is operable to generate a predicted segmentation mask for the medical image. As discussed above, the method 200 seeks to address the issue of target ambiguity, which remains a problem for existing click based interactive segmentations methods. The method 200 seeks to address this issue by generating multiple intermediate segmentation masks, which, owing to the way in which they are generated and updated using image embedding and Optimal Transport, are able to serve as diverse priors to guide final mask prediction and mitigate the issue of target ambiguity. According to different examples of the method 200, the number of click prompts to be used may be predetermined, initiated and updated, or may be established for example through trial and error. Also according to examples of the present disclosure, the image encoder ML model may be implemented as a one or more of a range of feature extractors, including for example a ResNet series model, HRNet, Vistion Transformer (ViT), etc. The medical image for which the segmentation mask is generated may be obtained using any suitable imaging modality, including for example radiography such as X-Ray imaging, Computed Tomography imaging (CT and / or CBCT), Magnetic Resonance Imaging, Scintigraphy, Positron Emission Tomography, Ultrasound, Visible Light Imaging, etc. The method 200 refers to both an image encoder ML model and a decoder ML model. For the purposes of the present disclosure, the term “ML model” encompasses within its scope the following concepts: machine Learning algorithms, comprising processes or instructions through which data may be used in a training process to generate a model artefact for performing a given task, or for representing a real-world process or system; and the model artefact that is created by such a training process, and which comprises the computational architecture that performs the task. Figures 3a to 3c show flow charts illustrating a further example of a computer implemented method 300 for generating a segmentation mask for a medical image. As for the method 200 discussed above, the method 300 may be performed by a segmentation node, which may comprise a physical or virtual node, and may be implemented in a computer system, treatment apparatus, such as a radiotherapy treatment apparatus, radiotherapy treatment planning system or apparatus computing device, or server apparatus, and / or may be implemented in a virtualized environment, for example in a cloud, edge cloud, or fog deployment. Examples of a virtual node may include a piece of software or computer program, a code fragment operable to implement a computer program, a virtualised function, or any other logical entity. The segmentation node may encompass multiple logical entities, as discussed in greater detail below. The method 300 illustrates an example of how the steps of the method 200 may be implemented and supplemented to provide the above discussed and additional functionality. Referring initially to Figure 3a, in implementing the method 300 the segmentation node may first detect user input clicks on the medical image in step 310. As illustrated at 310a, the user clicks may comprise positive clicks, which are within an error margin for a target object for the segmentation mask, and negative clicks, which are outside the error margin for the target object. The error margin may comprise a bound around the target object, allowing for imprecise delineation of the target object, such that a positive click is a click that is within or very close to the target object. In step 320, the segmentation node may then disk encode the detected user clicks. Disk encoding is a known process in which user clicks are represented as disks, generally with a fixed radius, as discussed for example in [4], In step 330, the segmentation node may then encoding the medical image by submitting the disk encoded user clicks, the medical image, and a reference segmentation mask to an image encoder ML model, wherein the image encoder ML model is operable to generate a plurality of image embeddings. As illustrated at 330a, the reference segmentation mask may comprise an initiated segmentation mask or a predicted segmentation mask from a previous iteration of the method. As discussed above, the image encoder ML model may be implemented as a one or more of a range of feature extractors, including for example a ResNet series model, HRNet, Vistion Transformer (ViT), etc. Referring now to Figure 3b, the segmentation node may then, in step 340, initiate a plurality of click prompts, wherein each initiated click prompt comprises a learnable prompt token, a positional encoding of user clicks, and a content encoding of user clicks. As illustrated at 340a, the click prompts may in some examples comprise positional and content encoding of positive clicks only, in some examples of the present disclosure, as illustrated at steps 340b and 340c, the positional encoding of user clicks may comprise a function of the disk encoded user clicks and a positional encoding of the plurality of image embeddings, and the content encoding of user clicks may comprise a function of the disk encoded user clicks and the plurality of image embeddings. In some examples, the positional encoding of user clicks may comprise: In further examples, the content encoding of user clicks may comprise: '.......' ' ■ (3) In step 350, the segmentation node may update the initiated click prompts using the plurality of image embeddings. As illustrated at 350a, this may comprise inputting the click prompts and the plurality of image embeddings to a transformer network, wherein the transformer network is operable to update the learnable prompt tokens of the click prompts. As illustrated at 350a, the transformer network may be operable to increase the alignment of the click prompts to the image embeddings. It will be appreciated that the user clicks, on the basis of which the click prompts are initiated, and image embeddings, belong to different modalities. Alignment refers to the process of ensuring that data from different modalities are synchronized or aligned in some relevant dimension. In some examples, the transformer network may therefore be operable to increase the synchronization, in one or more spatial dimensions, of the representation of user clicks that is provided by the click points with the image embeddings. As illustrated at 350c, the transformer network may comprise a plurality of transformer blocks, and each transformer block may comprise a cross-attention layer, a self-attention layer, and a feed-forward network. In some examples, each transformer block may implement the following: / £                > F) * F.F In step 360, the segmentation node may generate, from the updated click prompts and the plurality of image embeddings, a plurality of intermediate segmentation masks. As illustrated at 360a, each of the plurality of intermediate segmentation masks may comprise a function of one of the updated click prompts and the plurality of image embeddings. In some examples, each mask may comprise a function of a different click prompt and the image embeddings, so there will be a different intermediate mask for each different click prompt. In some examples, generating, from the updated click prompts and the plurality of image embeddings, a plurality of intermediate segmentation masks may comprise calculating: Referring now to Figure 3c, in step 370, the segmentation node may determine a transport plan for Optimal Transport from the updated click prompts to the plurality of image embeddings. As illustrated at 370a, this may comprise optimizing a cost matrix for transformation of the updated click prompts to the plurality of image embeddings. In some examples, the Sinkhorn algorithm may be sued to determine the transport plan, and the segmentation node may use the following optimization solution: T" ™ (°) In step 380, the segmentation node may update the plurality of intermediate segmentation masks using the determined transport plan. As illustrated at 380a, this may comprise performing an element wise multiplication of the transport plan and plurality of intermediate segmentation masks. Finally, in step 390, the segmentation node may input the plurality of updated intermediate segmentation masks, and the plurality of image embeddings, to a decoder ML model, wherein the decoder ML model is operable to generate a predicted segmentation mask for the medical image. As illustrated at 390a, the decoder ML model may comprise a plurality of convolutional layers and a classifier head. In some examples, the decoder ML model may be operable to implement: (10) In some examples, the decoder ML model may be operable to optimise the objective function: Example methods according to the present disclosure achieve interactive segmentation that demonstrates improved performance when compared to existing methods. Specifically, the methods disclosed herein can achieve target values for segmentation quality with a smaller average number of clicks than benchmark methods. In addition, with a smaller fixed number of clicks, the methods disclosed herein achieve improved segmentation quality when compared to benchmark methods, demonstrating that the methods disclosed herein effectively alleviate the target ambiguity problem that can be particularly challenging with fewer clicks. The methods described herein thus offer improved segmentation quality and speed. Segmentation of both target volumes and organs at risk (OARs) is a critical and time-consuming component of radiotherapy planning, and a significant portion of non-patient clinician time is dedicated to contouring structures prior to dosimetric treatment planning. Example methods according to the present disclosure can therefore support both the planning and delivery of radiotherapy treatment, including in the form of online Adaptive Radiotherapy (ART). The speed and quality afforded by methods of the present disclosure can in some examples support the provision of online ART, in which CBCT is used to capture patient imaging at the start of each visit of the treatment fraction. This up-to-date imaging data, if available with sufficient quality, can enable clinicians to track changes in patient anatomy, including for example tumour shrinkage over the course of the radiotherapy treatment, allowing for online target localisation and plan adaptation without the constraints of diagnostic CT imaging. Particular time constraints are present in online ART, owing to the need to complete image acquisition, image analysis including segmentation, treatment plan adaptation, and delivery of radiotherapy treatment, all within the time of a single patient visit. The improved image segmentation quality offered by methods according to the present disclosure, particularly at lower numbers of clicks, can support completion of the above actions within the time constraints imposed by ART, and may result in many additional medical treatment benefits (including improved accuracy of radiotherapy treatment, reduced exposure to unintended radiation, reduced treatment duration, etc.). The methods presented herein may be applicable to a variety of medical treatment and diagnostic settings or radiotherapy treatment equipment and devices. In one particular use case for methods of the present disclosure, a dose from a previous treatment session can be deformed or modified in light of the current patient anatomy as represented by a medical image of a patient that is contoured using a segmentation mask generated according to examples of the present disclosure. The output of the methods disclosed herein may thus be used in the creation or adaptation of a radiotherapy treatment plan. As discussed above, the methods 200 and 300 may be performed by a segmentation node, and the present disclosure provides a segmentation node that is adapted to perform any or all of the steps of the above discussed methods. The segmentation node may comprise a physical or virtual node, and may be implemented in a computer system, treatment apparatus, such as a radiotherapy treatment apparatus, radiotherapy treatment planning system or apparatus, computing device, or server apparatus, and / or may be implemented in a virtualized environment, for example in a cloud, edge cloud, or fog deployment. Examples of a virtual node may include a piece of software or computer program, a code fragment operable to implement a computer program, a virtualised function, or any other logical entity. The segmentation node may encompass multiple logical entities, as discussed in greater detail below. Figure 4 is a block diagram illustrating an example segmentation node 400 which may implement the method 200 and / or 300, as illustrated in Figures 2 to 3c, according to examples of the present disclosure, for example on receipt of suitable instructions from a computer program 450. Referring to Figure 4 the segmentation node 400 comprises a processor or processing circuitry 402, and may comprise a memory 404 and interfaces 406. The processing circuitry 402 is operable to perform some or all of the steps of the method 200 and / or 300 as discussed above with reference to Figures 2 to 3c. The memory 404 may contain instructions executable by the processing circuitry 402 such that the segmentation node 400 is operable to perform some or all of the steps of the method 200 and / or 300, as illustrated in Figures 2 to 3c. The instructions may also include instructions for executing one or more telecommunications and / or data communications protocols. The instructions may be stored in the form of the computer program 450. In some examples, the processor or processing circuitry 402 may include one or more microprocessors or microcontrollers, as well as other digital hardware, which may include digital signal processors (DSPs), special-purpose digital logic, etc. The processor or processing circuitry 402 may be implemented by any type of integrated circuit, such as an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) etc. The memory 404 may include one or several types of memory suitable for the processor, such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, solid state disk, hard disk drive, etc. In some examples as discussed above, the segmentation node may be incorporated into treatment apparatus, and examples of the present disclosure also provide a treatment apparatus and a treatment planning apparatus comprising either or both of a segmentation node as discussed above and / or a planning node operable to implement a method for adapting a radiotherapy treatment plan. Figures 2 to 3c discussed above provide an overview of methods which may be performed according to different examples of the present disclosure. These methods may be performed by a segmentation node as illustrated in Figure 4. There now follows a detailed discussion of how different process steps illustrated in Figures 2 to 3c and discussed above may be implemented, as well as presentation of experimental results for the example implementation. The functionality and implementation detail described below is discussed with reference to the modules of Figure 4 performing examples of the methods 200 and / or 300, substantially as described above. As discussed above, the goal of interactive segmentation is to develop a model that can accurately segment objects or regions of interest in an image based on user-provided clicks. In a first interaction round, users place clicks on the input image - where h and w denote the height and width of the image, respectively. In subsequent rounds, additional clicks are provided to improve segmentation performance based on the input image I and previous segmentation feedback 5 "* * , where ‘ is an empty mask for the first interaction round. The annotated clicks over all interactions can be grouped into positive and negative click sets, which are encoded as positive map A and negative map using disk encoding [4], The objective of the methods disclosed herein is to produce high-quality segmentation mask predictions using as few clicks as possible. Figure 5 is a block diagram illustrating an overview of an example implementation of the methods disclosed herein. The implementation is referred to as CPIot. As illustrated in Figure 5, CPIot comprises an extracts image embeddings by taking input image, click disk maps, and previous mask. CPIot also comprises a Click Encoder to initialize click prompts with multiple learnable prompts and click coordinates. To match click prompts with image embeddings efficiently, CPIot includes a Prompt-Pixel Alignment module and a Pixel-to-Prompt Optimal Transport (OT) module. With these modules, each click prompt learns to focus on a specific latent target. The optimized click prompts along with image embeddings are used to generate multiple intermediate masks, referred to a score maps, which capture different latent targets and serve as strong and diverse priors for guiding final mask prediction. A lightweight image decoder is then used to implicitly analyse optimized score maps with image embeddings and make final mask prediction. Each of these steps is discussed in greater detail below. Image Encoding (Steps 230 of method 200, 330 and 330a of method 300) The image encoder takes the input image I, the previous mask , and the positive and negative disk maps Dpos and Dneg to produce image embeddings s --- , where d, H, and W / represent the channel, width, and height of the image embedding, respectively. Commonly used network architectures for image segmentation, such as ResNet series, HRNet, and Vision Transformer (ViT), can be employed as the image encoder. In CPIot, ViT-B is used as the model backbone to extract image features, as in SimpleClick

[29] , To capture multi-scale information, the feature pyramid network is also employed, as in

[29] , Finally, a convolutional layer is utilized to merge the multi-scale features into one image embedding. In practice, examples of the method disclosed herein can use any existing feature extractor, and experiments are presented below with different image encoders. Click Encoder (Steps 240 of method 200, and 340, 340a, 340b, 340c of method 300) The click encoder encodes input clicks as multiple click prompts, which are used to capture semantic clues of different latent targets. As positive clicks are generally more informative in providing positional and semantic clues about a target object, CPIot only adopts positive clicks to initialize click prompts. Given positive click setV^&h^r , n being the number of positive clicks, the click encoder first encodes positive clicks as disk maps Dpos. Then, click prompts are initialized as follows: (1) Where * '' denotes N click prompts, P, Pp.e„ Pc represent learnable prompt tokens, position encoding of clicks, and content encoding of clicks, respectively. The learnable prompt tokens P are vectors to be optimized during training. The click position encoding Ppe. should contain information about the location of the target object, thus Where 777 represents the non-linear transformation function, e.g., an MLP. MAP is the masked average pooling operation, 7 ' is the positional encoding of image embeddings. Sinusoidal encodings

[46] are used for positional information, although there is potential for developing more effective encodings, considering that Dpos represents a small circular region around clicks. Lastly, for the content encodings of clicks, the aim is to capture the local semantic and appearance information around the click corresponding to the target objects. To achieve this, the image embedding F is combined with the positive disk map Dpos.: (3) By introducing the click encoder, positional and content information about the target object is integrated into the click prompts, leading to enhanced click initialization. Following encoding, CPIot employs two strategies: prompt-pixel alignment and pixel-to-prompt optimal transport, to effectively optimize the learning of click prompts, allowing each prompt to focus on a specific latent target. Prompt-Pixel Alignment Module (steps 250, 260 of method 200, 350; 350a-c, 360, 360a-c of method 300) The goal of having / V click prompts is to represent multiple latent targets effectively. As clicks and image embeddings belong to different modalities, an objective for the promptpixel alignment module is to align these two modalities. To leverage the ability to capture long-range dependencies in the data, CPIot employs vision transformers for the alignment of click prompts and image embeddings. Specifically, the click prompts and image embeddings are input to a transformer network, which consists of L transformer blocks (step 350a). Each block comprises a cross-attention layer, a self-attention layer, and a feed-forward network (step 350b). Taking the / -th transformer block as an example: 4- ' — Saif ■ ,., : ■ .,- (4) Where CrossAtten, SelfAtten, and FFN represent a cross attention layer, a self attention layer, and a feed forward network, respectively. and represent the click prompts at the / -th and { / - 1}-th transformer block, respectively. Interacting with image embeddings in the transformer network, click prompts learn to focus on various latent target objects. Finally, the updated click prompts are obtained. Using click prompts * and the image embedding F, one score matrix is generated per transformed click •p prompt x that measures prompt-pixel alignment, that is: (5) Where M...... , 6 denotes the flatten operation in the spatial dimension, and the superscript T refers to the transpose operation. In such way, the updated click prompts 3* and the prompt-pixel score maps S are obtained. The prompt-pixel score maps S can be envisaged as intermediate segmentation masks. Ideally, the different prompt pixel score maps for different would directly capture latent object targets. However, in the optimization these maps tend to be similar to one another in the absence of any explicit constraint that ensures the score maps are indeed different. To this end, to increase target diversity and achieve better matching between click prompts and image embedding, the pixel-to-prompt optimal transport module is introduced. Pixel-to-Prompt Optimal Transport Module (steps 270, 280 of method 200, 370; 370a, 380, 380a of method 300) Optimal Transport (OT) is widely used to minimize the transport distance between two probability distributions. In the context of CPIot, the focus is on optimal transport with discrete distributions. Considering two discrete distributions p and v defined in probability space F and G: ’ (6) Where Srand 59are Dirac Delta function located at support points f and g, respectively. „1 V^:V .-1 pm and qnare discrete probability vectors that satisfyXn^i / ¾. - 1 and - >. Then the discrete optimal transport can be formulated as: T* - y y sx Tl* ™ = Here, C is the cost matrix which measures the distance between fm and gn, e.g., cosine i -.......... distance Cmn HZ™ ihHibdh. T is transport plan that learns to minimize the distance between two distributions. To speed up optimization, most works with optimal transport adopt the Sinkhorn algorithm [9], which is a strictly convex optimization problem. This results in a fast optimization solution with few iterations as: '1-- a* ""£? / A s k1m^( ) ... . . ; , .. (8) Where t is the iteration, A is a hyper-parameter, af = iJ / exp(-C / A)b'\ and b* = v / expt-C / A^, with the initialization b°= 1. Click Prompt OT Solver: OT is incorporated into the click prompt learning process, which enables multiple click prompts to capture different underlying targets for segmentation. Specifically, given the flatten image embedding and updated click prompts 0 <■" p * , CPIot learns the transport plan T with F and ” by optimizing the cost matrix C to push :to F. Here, the cost matrix in Equation (7) is defined with prompt-pixel score map as C := 1 - S, where ' ' . With the fast optimization solution in Equation (8) provided by the Sinkhorn algorithm, .... the optimal transport plan 4 "can be obtained. As the goal is to obtain the prompt-pixel score map that minimizes the total distances between image embedding and click prompts, the final score map, or updated intermediate segmentation masks, are reformulated as: Where ' "is the refined score map, namely refined intermediate segmentation masks for latent targets. The refined score map provides strong and diverse priors for the final mask prediction, thus mitigating the issue of target ambiguity. Image Decoder and Objective (step 290 of method 200, 390, 390a of method 300) The score matrix (updated intermediate segmentation masks) in Equation (9), is regarded as a prior mask indicating various latent targets. This score matrix, together with the image embedding, is input to a decoder network to obtain a final mask prediction: Where is the predicted mask, Cat is the concatenation operation (the reshape operation is omitted for simplicity). is a light-weighted encoder network which consists of multiple convolutional layers followed by a classifier head. To supervise the click prompt learning and score map optimization, the following objective is adopted: , Is xv. XV. J X >' x -x X-v . . x.x xVxx . X. \ v ■: % y'Xs $ X - s xxX _> . '------- • Ass x v. <$ ' ? * (11) Where denotes the normalized focal loss, is the single score map indicating underlying target, a and jB are two hyper-parameters. The second and third loss terms are adopted to ensure that N score maps contain characteristics of shared targets while exhibiting variance of different underlying targets. Experiments Datasets. Model training is conducted on SBD

[16] and COCO

[26] +LVIS

[15] separately. The model is then evaluated on five natural image datasets, GrabCut

[40] , Berkeley

[34] , SBD

[16] , 237 DAVIS

[36] , Pascal VOC

[10] , and three medical datasets, ssTEM

[12] , BraTS [3], OAIZIB [2]. More details about datasets can be found in

[29] . The training sets are as follows: SBD

[16] : 8,498 training images with 20,172 instances and 2,857 validation images with 6,671 240 instances. The training set and validation set are adopted for model training and validation respectively. COCO

[26] +LVIS

[15] : 118K training images (1.2M instances) and various scene variations. This dataset is only used for model training. Evaluation Metric. The strategy used in previous methods such as [29, 44, 27, 8, 57] is followed to simulate clicks, where the next click is placed at the centre of the largest error region obtained by comparing the prediction and ground truth. The Number of Clicks (NOC) is used as the evaluation metric, which computes the average number of clicks required to achieve a fixed Intersection over Union (loU). Specifically, the target loU is set as 85% and 90%, resulting in the metrics NoC85 and NoC90, respectively. The maximum number of clicks allowed for each instance is set to 20 by default. The average loll given a fixed number of clicks (mloU@k) is also used as an evaluation metric to measure the segmentation quality at different stages of the interactive process. Implementation Details. In the experiments, the training strategies in

[29] are followed: training for 55 epochs on both SBD and COCO+LVIS datasets. ViT-Bis used as the backbone, initialized by MAE pretraining on ImageNet. Images are resized and cropped to 448x448, and the same data augmentation is used during training as in

[29] , CPIot is trained with the Adam 264 optimizer with parameters set as / 3i = 0.9, / 32 = 0.999, the initial learning rate is set to 5 x 1CT5 and decreased to 5 x 1O~6 after 50 epochs. The number of click prompts is set as N = 5, and the number of transformer block layers in the promptpixel alignment module is set as L = 2. The hyper-parameters in Equation (11) are set as a = 1 and / 3 = 1. All experiments are conducted on four NVIDIA Tesla A100 GPUs. Figure 6 shows Table 1, which is a NoC comparison of CPIot with existing methods. Results are reported with respect to five benchmarks: GrabCut

[40] , Berkeley

[34] , SBD

[16] , DAVIS

[36] , and Pascal VOC

[10] , + denotes generalist model, while * denotes results provided by SSEM

[58] , The CPIot implementation of the methods disclosed herein is a consistent top-performer on all benchmarks. Figure 7 shows Table 2, which is an out-of-domain evaluation on three medical benchmarks: ssTEM

[12] , BraTS[3], and OAIZIB[2]. mloU after kclicks is reported. CPIot can be seen to generalize well to the medical domain even without finetuning. Ablation study Extensive ablations have been conducted on five natural image datasets to demonstrate the effectiveness of CPIot. SimpleClick

[29] was taken as a baseline model. The training was performed on the SBD dataset and NoC90 is reported as evaluation metric. Analysis for Prompt Initialization. To analyze the effectiveness of the proposed prompt initialization method (Equation (1)), this method is compared with three initialization variants: (1) randomly initialized as learnable prompts P, (2) initialized via position encoding of clicks P + Ppe., and (3) initialized via content encoding of clicks P +PC. The experimental results reported in Table 3 (Figure 8) show that click prompts with the proposed initialization method achieve the best performance across all five benchmarks. This is reasonable as such initialization injects both position and content priors about the target object into click prompts, leading to faster and stable optimization of click prompts for capturing latent targets. Figure 8 shows Tables 3, 4, 5 and 6. Table 3 is an analysis of prompt initialization, and demonstrates that the proposed initialization in Equation (1) performs better. Table 4 shows the benefits of different values of / V click prompts. CPIot with N = 5 is the topperformer. Table 5 shows the benefits of model components PPA (Prompt-Pixel Alignment) and OT. With PPA and OT, CPIot achieves consistent best performance on all five benchmarks. Table 6 shows the benefits of different values of L transformer blocks. CPIot with L = 2 performs best on four of five datasets. Benefits of Prompt-Pixel Alignment. The prompt-pixel alignment (PPA) module (steps 250, 260, 350, 360) aims to align prompt and image embedding, thus allowing click prompts to focus on different target regions. The output of the PPA module, which is the prompt-pixel aligned score matrices S, is directly fed into the decoder to obtain the final prediction. Experimental results, as presented in Table 5, demonstrate that baseline model with the PPA module consistently improves the performance across all benchmarks. Benefits of Pixel-to-Prompt Optimal Transport. With the pixel-to-optimal transport module, a refined score map is obtained as set out in Equation (9). The effectiveness of this updated score map (intermediate segmentation masks) is illustrated in Table 5 by removing the PPA module from CpIot. Optimal transport also consistently improves model capacity, for instance, CPIot with OT outperforms baseline model by 0.14 clicks. With both prompt-pixel alignment module and pixel-to-prompt optimal transport module, CPIot achieves improved performance. Effect of Click Prompts number N. Each click prompt refers to a latent target, so the number of click prompts N is an important hyper-parameter in the model. N is varied to be [1,3,5,7] and, the results of this variation are reported in Table 4. It can be concluded that CPIot with five click prompts achieves best performance, which indicates that five latent targets are enough to model target variance in tackling target ambiguity. Effect of Transformer Block number L. In the prompt-pixel alignment module discussed above, L transformer blocks are used to align click prompts and image embeddings. Experiments with CPIot using different numbers of transformer blocks, and the experimental results are shown in Table 6. It can be concluded that CPIot achieves the best performance when L = 2, indicating the model is both effective and efficient. Performance Evaluation Comparison with State-of-the-art. CPIot is compared with existing click-based interactive segmentation methods as well as two generalist segmentation models, SAM

[20] and 316 SEEM

[58] , in Table 1. The results show that CPIot achieves state-of-the-art performance on all five natural image benchmarks. Specifically, when trained on the SBD training set, CPIot achieves 3.05 NOC@85 and 4.95 NOC@90 on the validation set, outperforming the previous SOTA method by 8%. When trained on COCO+LVIS (C+L), CPIot also achieves competitive results. The generalist models SAM and SEEM exhibit poor performance in the click-based interactive segmentation task. For exmple, when evaluated on the SBD validation set, they the generalist models achieve 9.76 and 322 9.99 NoC@90, respectively. Considering that these generalist models are optimized to generalize to multi-modal inputs, such as text and bounding box, it is not surprising that they perform poorly in specialized tasks like click-based interactive segmentation. Convergence analysis on Berkeley

[34] , DAVIS

[36] , 297 OAIZIB [2], and BraTS [3] is also reported in Figure 9a to 9d, which further highlights the effectiveness and superiority of CPIot. Out-of-domain Evaluation. The out-of-domain generalization of CPIot is evaluated on three medical datasets: ssTEM

[12] , BraTS[3], and OAIZIB[2], The evaluation results on these datasets are reported in Table 2. Overall, CPIot exhibits good generalization to the medical domain, especially with fewer clicks. For example, CPIot achieves 78.61 @10 on the OAIZIB dataset, which is 2.61 higher than that of the SimpleClick. This is convincing evidence that CPIot can alleviate the target ambiguity problem that usually arises with fewer clicks. Figures 10a-c illustrate segmentation results on DAVIS

[36] and BraTS

[36] , The Figures show (a) challenge case on natural image, (b) challenge case on medical image, and (c) five normal cases. The segmentation probability maps are shown on the right; the segmentation maps are overlaid on original images, on the left. Positive and negative clicks are marked with pale and dark dots on the image, respectively. Although CPIot benefits from click prompt learning, it does have certain limitations, as illustrated in Figures 10a-c. One of these limitations is the difficulty in accurately segmenting thin structures such as ropes and bicycles. To address this issue, incorporating local refinement techniques, such as those proposed in Chen et al.[8] and Lin et al.

[27] , could be beneficial. Additionally, in the case of particularly complex structures, may require a higher number of clicks. This challenge can be mitigated by fine-tuning the model on appropriate medical datasets to improve its performance on images containing particularly complex structures. Examples methods according to the present disclosure thus provide a new and effective method for generating a segmentation mask for medical images, in which click prompt learning and optimal transport are used to address the target ambiguity issue in interactive segmentation. Example methods disclosed herein leverage multiple click prompts to generate diverse mask candidates, which serve as strong priors for final mask prediction. Prompt-pixel alignment and click prompt optimal transport techniques are used learn optimal click prompts. Experimental results on natural image and medical datasets demonstrate the superiority of the disclosed methods over state-of-the-art approaches. Strong generalization capabilities in the medical domain. Example methods according to the present disclosure thus provide an effective and promising solution for accurate interactive segmentation. The methods of the present disclosure may be implemented in hardware, or as software modules running on one or more processors. The methods may also be carried out according to the instructions of a computer program, and the present disclosure also provides a computer readable medium having stored thereon a program for carrying out any of the methods described herein. A computer program embodying the disclosure may be stored on a computer readable medium, or it could, for example, be in the form of a signal such as a downloadable data signal provided from an Internet website, or it could be in any other form. It should be noted that the above-mentioned examples illustrate rather than limit the disclosure, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims or numbered embodiments. The word “comprising” does not exclude the presence of elements or steps other than those listed in a claim or embodiment, “a” or “an” does not exclude a plurality, and a single processor or other unit may fulfil the functions of several units recited in the claims or numbered embodiments. Any reference signs in the claims or numbered embodiments shall not be construed so as to limit their scope. References [1] David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler. Efficient interactive annotation of segmentation datasets with Polygon-RNN++. In Proceedings of the IEEE / CVF International 352 Conference on Computer Vision, pages 859-868, 2018. [2] Felix Ambellan, Alexander Tack, Moritz Ehlke, and Stefan Zachow. Automated segmentation of knee bone and cartilage combining statistical shape knowledge and convolutional neural networks: Data from the osteoarthritis initiative. Medical image analysis, 52:109-118, 2019. [3] lljjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Kalpathy-Cramer, Felipe C Kitamura, Sarthak Pati, et al. The rsnaasnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314, 2021. [4] Rodrigo Benenson, Stefan Popov, and Vittorio Ferrari. Large-scale interactive object segmentation with human annotators. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 11700-11709, 2019. [5] Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Prompt learning with optimal transport for vision-language models. arXiv preprint arXiv:2210.01253, 2022. [6] Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, and Jingjing Liu. Graph optimal transport for cross-domain alignment. In International Conference on Machine Learning, pages 1542-1553. PMLR, 2020. [7] Xi Chen, Zhiyan Zhao, Feiwu Yu, Yilei Zhang, and Manni Duan. Conditional diffusion for interactive segmentation. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 7345-7354, 2021. [8] Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, and Hengshuang Zhao. FocalClick: Towards practical interactive image segmentation. In Proceedings of the IEEE / CVF 374 International Conference on Computer Vision, pages 1300-1309, 2022. [9] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.

[10] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2):303-338, 2010.

[11] Marco Forte, Brian Price, Scott Cohen, Ning Xu, and Francois Pitie. Getting to 99% accuracy in interactive segmentation. arXivpreprint arXiv:2003.07932, 2020.

[12] Stephan Gerhard, Jan Funke, Julien Martel, Albert Cardona, and Richard Fetter. Segmented anisotropic sstem dataset of neural tissue, figshare, pages 0-0, 2013.

[13] Leo Grady. Random walks for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(11):1768-1783, 2006.

[14] Varun Gulshan, Carsten Rother, Antonio Criminisi, Andrew Blake, and Andrew Zisserman. Geodesic star convexity for interactive image segmentation. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3129-3136, 2010.

[15] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 5356-5364, 2019.

[16] Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik.Semantic contours from inverse detectors. In Proceedings of IEEE International Conference on Computer Vision, pages 991-998, 2011.

[17] Won-Dong Jang and Chang-Su Kim. Interactive image segmentation via backpropagating refinement scheme. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 5297-5306, 2019.

[18] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423—438, 2020.

[19] Tae Hoon Kim, Kyoung Mu Lee, and Sang Uk Lee. Nonparametric higher-order learning for interactive segmentation. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3201-3208, 2010.

[20] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, 405 Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023.

[21] Theodora Kontogianni, Michael Gygli, Jasper Uijlings, and Vittorio Ferrari. Continuous adaptation for interactive object segmentation by learning from corrections. In Proceedings of the European Conference on Computer Vision, pages 579-596, 2020.

[22] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language driven semantic segmentation. arXiv preprint arXiv:2201.03546, 2022.

[23] Zhuwen Li, Qifeng Chen, and Vladlen Koltun. Interactive image segmentation with latent diversity. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 414 577-585, 2018.

[24] Jun Hao Liew, Scott Cohen, Brian Price, Long Mai, Sim-Heng Ong, and Jiashi Feng. Multiseg: Semantically meaningful, scale-diverse segmentations from minimal user input. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 662-670, 2019.

[25] JunHao Liew, Yunchao Wei, Wei Xiong, Sim-Heng Ong, and Jiashi Feng. Regional interactive image segmentation networks. In Proceedings of IEEE International Conference on Computer Vision, pages 2746-2754, 2017.

[26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.

[27] Zheng Lin, Zheng-Peng Duan, Zhao Zhang, Chun-Le Guo, and Ming-Ming Cheng. Focus Cut: Diving into a focus view in interactive segmentation. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 2637-2646, 2022.

[28] Zheng Lin, Zhao Zhang, Lin-Zhuo Chen, Ming-Ming Cheng, and Shao-Ping Lu. Interactive image segmentation with first click attention. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 13339-13348, 2020.

[29] Qin Liu, Zhenlin Xu, Gedas Bertasius, and Marc Niethammer. Simpleclick: Interactive image segmentation with simple vision transformers. arXivpreprint arXiv:2210.11006, 2022.

[30] Qin Liu, Zhenlin Xu, Yining Jiao, and Marc Niethammer, isegformer: Interactive segmentation via transformers with application to 3d knee mr images. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2022:25th International Conference, Singapore, September 18-22, 2022, Proceedings, Part V, pages 464-474. Springer, 2022.

[31] Qin Liu, Meng Zheng, Benjamin Planche, Srikrishna Karanam, Terrence Chen, Marc Niethammer, and Ziyan Wu. Pseudoclick: Interactive image segmentation with click imitation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI, pages 728-745. Springer, 2022.

[32] Weide Liu, Chi Zhang, Henghui Ding, Tzu-Yi Hung, and Guosheng Lin. Few-shot segmentation with optimal transport matching and message flow. arXivpreprint arXiv:2108.08518, 2021.

[33] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103.10385, 2021.

[34] Kevin McGuinness and Noel E O’connor. A comparative evaluation of interactive segmentation algorithms. Pattern Recognition, 43(2):434-444, 2010.

[35] Gaspard Monge. Memoire sur la theorie des deblais et des remblais. Mem. Math. Phys. Acad. Royale Sci., pages 666-704, 1781.

[36] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 724-732, 2016.

[37] Fabio Petroni, Tim Rocktaschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXivpreprint arXiv: 1909.01066, 2019.

[38] Gabriel Peyre, Marco Cuturi, et al. Computational optimal transport. Center for Research in Economics and Statistics Working Papers, (2017-86), 2017.

[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.

[40] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. GrabCut: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics, 23(3):309-314, 2004.

[41] Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving gans using optimal transport. arXivpreprint arXiv: 1803.05573, 2018.

[42] Matthias Seeger. Gaussian processes for machine learning. International journal of neural systems, 14(02):69-106, 2004.

[43] Konstantin Sofiiuk, Ilia Petrov, Olga Barinova, and Anton Konushin. f-BRS: Rethinking backpropagating refinement for interactive segmentation. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 8623-8632, 2020.

[44] Konstantin Sofiiuk, Ilia A Petrov, and Anton Konushin. Reviving iterative training with mask guidance for interactive segmentation. arXivpreprint arXiv:2102.06583, 2021.

[45] Kihyuk Sohn, Yuan Hao, Jose Lezama, Luisa Polania, Huiwen Chang, Han Zhang, Irfan Essa, and Lu Jiang. Visual prompt tuning for generative transfer learning. arXiv preprint arXiv:2210.00990, 2022.

[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and lllia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.

[47] Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 11686-11695, 2022.

[48] Qiaoqiao Wei, Hui Zhang, and Jun-Hai Yong. Focused and collaborative feedback integration for interactive image segmentation. arXiv preprint arXiv:2303.11880, 2023.

[49] Jiajun Wu, Yibiao Zhao, Jun-Yan Zhu, Siwei Luo, and Zhuowen Tu. MILCut: A sweeping line multiple instance learning paradigm for interactive image segmentation. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 256-263, 2014.

[50] Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin Duke. Gromov-wasserstein learning for graph matching and node embedding. In International conference on machine learning, pages 6932-6941. PMLR, 2019.

[51] Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas Huang. Deep interactive object selection. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 373-381, 2016.

[52] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXiv:2210.07225, 2022.

[53] Shiyin Zhang, Jun Hao Liew, Yunchao Wei, Shikui Wei, and Yao Zhao. Interactive object segmentation with inside-outside guidance. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 12234-12244, 2020.

[54] He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. Neural topic model via optimal transport. arXiv preprint arXiv:2008.13537, 2020.

[55] Wenliang Zhao, Yongming Rao, Ziyi Wang, Jiwen Lu, and Jie Zhou. Towards interpretable deep metric learning with structural matching. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 9887-9896, 2021.

[56] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.

[57] Minghao Zhou, Hong Wang, Qian Zhao, Yuexiang Li, Yawen Huang, Deyu Meng, and Yefeng Zheng. Interactive segmentation as gaussian process classification. arXiv preprint arXiv:2302.14578, 2023.

[58] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong 5 Jae Lee. Segment everything everywhere all at once, 2023.

Claims

1. A computer implemented method for generating a segmentation mask for a medical image, the method comprising:detecting, on the medical image, user input clicks;disk encoding the detected user clicks;encoding the medical image by submitting the disk encoded user clicks, the medical image, and a reference segmentation mask to an image encoder Machine Learning, ML model, wherein the image encoder ML model is operable to generate a plurality of image embeddings;initiating a plurality of click prompts, wherein each initiated click prompt comprises:a learnable prompt token;a positional encoding of user clicks; anda content encoding of user clicks;updating the initiated click prompts using the plurality of image embeddings;generating, from the updated click prompts and the plurality of image embeddings, a plurality of intermediate segmentation masks;determining a transport plan for Optimal Transport from the updated click prompts to the plurality of image embeddings;updating the plurality of intermediate segmentation masks using the determined transport plan; andinputting the plurality of updated intermediate segmentation masks and the plurality of image embeddings to a decoder ML model, wherein the decoder ML model is operable to generate a predicted segmentation mask for the medical image.

2. A method as claimed in claim 1, wherein the reference segmentation mask comprises an initiated segmentation mask or a predicted segmentation mask from a previous iteration of the method.

3. A method as claimed in claim 1 or 2, wherein the user clicks comprise positive clicks, which are within an error margin for a target object for the segmentation mask, and negative clicks, which are outside the error margin for the target object, and wherein the click prompts comprise positional and content encoding of positive clicks only.

4. A method as claimed in any one of the preceding claims, wherein the positional encoding of user clicks comprises a function of the disk encoded user clicks and a positional encoding of the plurality of image embeddings.

5. A method as claimed in claim 4, wherein the positional encoding of user clicks comprises:

6. A method as claimed in any one of the preceding claims, wherein the content encoding of user clicks comprises a function of the disk encoded user clicks and the plurality of image embeddings.

7. A method as claimed in claim 6, wherein the content encoding of user clicks comprises:

8. A method as claimed in any one of the preceding claims, wherein updating the initiated click prompts using the plurality of image embeddings comprises inputting the click prompts and the plurality of image embeddings to a transformer network, wherein the transformer network is operable to update the learnable prompt tokens of the click prompts.

9. A method as claimed in claim 8, wherein the transformer network is operable to increase the alignment of the click prompts to the image embeddings.

10. A method as claimed in claim 8 or 9, wherein the transformer network comprises a plurality of transformer blocks, and wherein each transformer block comprises a cross-attention layer, a self-attention layer, and a feed-forward network.

11. A method as claimed in claim 10, wherein each transformer block implements the following:

12. A method as claimed in any one of the preceding claims, wherein each of the plurality of intermediate segmentation masks comprises a function of one of the updated click prompts and the plurality of image embeddings.

13. A method as claimed in any one of the preceding claims, wherein generating, from the updated click prompts and the plurality of image embeddings, a plurality of intermediate segmentation masks comprises calculating:

14. A method as claimed in any one of the preceding claims, wherein determining a transport plan for Optimal Transport from the updated click prompts to the plurality of image embeddings comprises optimizing a cost matrix for transformation of the updated click prompts to the plurality of image embeddings.

15. A method as claimed in any one of the preceding claims, wherein updating the plurality of intermediate segmentation masks using the determined transport plan comprises performing an elementwise multiplication of the transport plan and plurality of intermediate segmentation masks.

16. A method as claimed in any one of the preceding claims, wherein the decoder ML model comprises a plurality of convolutional layers and a classifier head.

17. A method as claimed in any one of the preceding claims, wherein the decoder ML model is operable to implement:

18. A method as claimed in any one of the preceding claims, wherein the decoder ML model is operable to optimise the objective function:. <S V .V i . . , v£. ™ Ai,. A / (S A19. A computer program product comprising a computer readable medium, the computer readable medium having computer readable code embodied therein, thecomputer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform a method as claimed in any one of the preceding claims.

20. A segmentation node for generating a segmentation mask for a medical image, the segmentation node comprising processing circuitry configured to cause the segmentation node to:detect, on the medical image, user input clicks;disk encode the detected user clicks;encode the medical image by submitting the disk encoded user clicks, the medical image, and a reference segmentation mask to an image encoder Machine Learning, ML model, wherein the image encoder ML model is operable to generate a plurality of image embeddings;initiate a plurality of click prompts, wherein each initiated click prompt comprises: a learnable prompt token;a positional encoding of user clicks; anda content encoding of user clicks;update the initiated click prompts using the plurality of image embeddings;generate, from the updated click prompts and the plurality of image embeddings, a plurality of intermediate segmentation masks;determine a transport plan for Optimal Transport from the updated click prompts to the plurality of image embeddings;update the plurality of intermediate segmentation masks using the determined transport plan; andinput the plurality of updated intermediate segmentation masks and the plurality of image embeddings to a decoder ML model, wherein the decoder ML model is operable to generate a predicted segmentation mask for the medical image.

21. A segmentation node as claimed in claim 20, wherein the processing circuitry is further configured to cause the segmentation node to carry out a method as claimed in any one of claims 2 to 18.

22. A radiotherapy treatment apparatus comprising a segmentation node as claimed in claim 20 or 21.

Images