Method and apparatus

Abstract

Broadly speaking, the present techniques generally relate to a method for processing images of an organ (or part thereof) using a trained machine learning, ML, model in order to more efficiently and accurately provide information on different tissue regions of the organ. Advantageously, the ML model generates an augmented image of the organ showing the different tissue regions, such as diseased tissue regions and healthy tissue regions. This may help a surgeon to decide where to perform a surgical intervention in a way that only diseased tissue is excised.

Description

[0001] METHOD AND APPARATUS

[0002] Field

[0003] The present invention relates to a method of processing images of an organ using a trained machine learning, ML, model to predict a boundary between unhealthy and healthy regions of the organ, and generating an augmented image of the organ depicting the predicted boundary.

[0004] Background

[0005] Typically, surgical intervention is performed with surgeon judgement of tissue quality or vitality including for reconstruction after disease excision. However, surgeon judgement may be subjective and / or may be based on general principles rather than personalised to an individual patient and / or the tissue quality or character being observed may be difficult to ascertain visually.

[0006] Quantification of tissue perfusion is of interest to clinicians. Existing methods for quantifying tissue perfusion lack sufficient clinical applicability due to their subjective output which still requires user interpretation (e.g. perfused vs not perfused, or accepted perfusion vs rejected perfusion).

[0007] For example, correct interpretation of indocyanine green fluorescence angiography reduces anastomotic leak rates following colorectal resections. However, inherent subjectivity, steep learning curve, camera behaviours, and cognitive burden on surgeons interpreting the imagery in theatre present challenges.

[0008] Hence, there is a need to improve judgement of tissue quality generally, for example for reconstruction after disease excision or characterisation of tissue vitality, and, in particular, assist inexperienced surgeons achieve a similar level of tissue understanding as expert surgeons and so achieve similar standard of care for patients. This may also apply to judgment in interpretation of surgical imaging including visual vascular perfusion assessment.

[0009] Summary

[0010] It is one aim of the present invention, amongst others, to provide a method of discriminating between a first tissue status and a second tissue status of an organ or a part thereof which at least partially obviates or mitigates at least some of the disadvantages of the prior art, whether identified herein or elsewhere. For instance, it is an aim of embodiments of the invention to provide a method of discriminating between a first tissue status and a second tissue status of an organ or a part thereof that improves judgement of tissue quality, for example for reconstruction after disease excision or judgement of vitality after field dissection. Generally, the aspects relate to tissue perfusion assessment technology and experts’ interpretation of tissue perfusion using artificial intelligence methods for resectional surgery, for example during colorectal resectional surgery and / or thyroidectomy or cancer characterization.

[0011] According to the present invention there is provided a method of discriminating between a first tissue status and a second tissue status of an organ or a part thereof, as set forth in the appended claims. Also provided are an ex vivo method for treatment of an organ by surgery or therapy or an ex vivo therapy or diagnostic method practised on an organ, a method for treatment of the human or animal body by surgery or a therapy or diagnostic method practised on the human or animal body, a computer, a computer program and a nontransient computer-readable storage medium. Other features of the invention will be apparent from the dependent claims, and the description that follows.

[0012] Method of Inferring:

[0013] In a first aspect of the present techniques, there is provided a computer-implemented method of processing images of an organ, the method comprising: receiving a sequence of images of an organ or part of an organ, each image in the sequence depicting a set of spatial regions including a first spatial region having a first tissue status and a second spatial region having a second tissue status, wherein the sequence of images depict the organ or the part of the organ over a period of time perfusion with a contrast agent; analysing the received sequence of images using a trained machine learning, ML, model, wherein the trained ML model is trained to determine an optimal boundary between the first spatial region and the second spatial region based on how the contrast agent perfuses through the first and second spatial regions over time; generating an augmented image of the organ or the part of the organ by adding an overlay to an image, wherein the overlay depicts the determined optimal boundary between the first spatial region and the second spatial region.

[0014] Advantageously, the present techniques automate, using a trained machine learning, ML, model, how to distinguish between different spatial regions of an organ (or part of an organ) so that a surgeon does not have to judge this themselves (or does not have to judge this purely by themselves). The organ has “healthy” and “unhealthy” / ”diseased” regions: these regions are generally referred to herein as spatial regions of the organ. Each spatial region has an associated tissue status, e.g. healthy or diseased. The surgeon needs to determine the location of the two spatial regions, so that a surgical intervention may be performed at the right location on / in the organ. The trained ML model analyses a sequence of images captured after a contrast agent has been provided in the organ. The perfusion through the organ occurs at different rates depending on the health of the tissue, as described in International Patent Application W02020 / 083501 which is incorporated herein by reference in its entirety. The trained ML model analyses the sequence of images, captured over a period of time immediately after commencement (i.e. after the contrast agent has been provided in the organ), to predict where the spatial regions are within the organ. The prediction is then provided in visual format, via the generation of an augmented image of the organ with an overlay showing the predicted location of the different spatial regions (or boundary therebetween). This augmented image is then displayed on a display so that the surgeon is able to see the different spatial regions as predicted by the trained ML model, before performing a surgical intervention and / or while performing a surgical intervention.

[0015] Although reference is made to analysing images after perfusion with a contrast agent, it will be understood that the present techniques may also be suitable for dye-free perfusion imagers (e.g. hyperspectral imaging). Thus, the present techniques apply equally to images obtained from other techniques to determine perfusion, and it will be understood that images obtained after one or more contrast agents is applied are just one example way to determine perfusion. Thus, in a related aspect, there is provided a computer-implemented method of processing images of an organ, the method comprising: receiving a sequence of images of an organ or part of an organ, each image in the sequence depicting a set of spatial regions including a first spatial region having a first tissue status and a second spatial region having a second tissue status, wherein the sequence of images are obtained via a hyperspectral imaging process; analysing the received sequence of images using a trained machine learning, ML, model, wherein the trained ML model is trained to determine an optimal boundary between the first spatial region and the second spatial region based on changes in hyperspectral image data from the first and second spatial regions over time; and generating an augmented image of the organ or the part of the organ by adding an overlay to an image, wherein the overlay depicts the determined optimal boundary between the first spatial region and the second spatial region. The features of the first aspect apply equally to this related aspect.

[0016] The optimal boundary may be the location at which a surgeon should perform an intervention (e.g. a transection or incision).

[0017] It will be understood that, in some cases, the augmented image does not dictate a surgeon’s final decision on where to perform an intervention. Rather, the augmented image provides the surgeon with a recommendation and assists the surgeon in making the final decision which is based on their expertise. However, the system is capable of targeting an action to a site previously judged only by a human surgeon to the same level of accuracy and could therefore be deployed to direct an autonomous surgical act based on perfusion assessment if this were to be otherwise appropriate.

[0018] In other cases, the augmented image may be used to control a robotic surgical device for fully-automated surgery or for robot-assisted surgery, to perform an intervention. In this way, determining a boundary between the two spatial regions, (also referred to herein as discriminating between the first tissue status and the second tissue status of the organ or a part thereof) is objective and individualised (for example, personalised) for the individual organ or part thereof since the discriminating is performed by the trained machine learning algorithm, on the time series of images of the perfused organ or part thereof. In this way, judgement of tissue quality is improved, for example for reconstruction after disease excision, thereby indicating to a surgeon where to transect the organ at an optimized transection point, since the optimized transection point of the organ is the same or similar to the area that would be recommended by an expert surgeon using the same signalling data, thereby improving healing after transection, for example. In other words, the method provides determination of tissue resection extent, for example ex vivo or in vivo during surgery, by imaging of perfusion over time during a procedure so that the optimum transection point can be made based on the specific perfusion profile of an individual intraoperatively. In another way, the method indicates that an expert surgeon would judge the tissue to be vital and exert normal function postoperatively and so additional measures to provide such function are unnecessary.

[0019] Generally, tissues require sufficient perfusion to function and indeed heal after surgery. Conventionally, operations build in such consideration by surgeon judgement regarding cut (transection) lines related to tissue resection. The inventors have developed a machine learning algorithm and clinical process that enables identification of a reasonable site for subsequent incision based on performing early intraoperative assessment of relevant tissues with repeated assessment after surgical preparation rather than the conventional method of assessment only when at the point of resection. The method involves the use of contrast agents (also known as dyes) and electromagnetic radiation (EMR) to excite the contrast agents and amplify the inherent perfusion patterns within tissues relevant to the disease site and at sites potentially harbouring disease and thereafter a machine learning algorithm to compare the imagery to indicate perfusion.

[0020] The inventors have also previously found that the intensity of light emitted from a target bodily tissue is different in time scale, depending on whether the tissue is benign or malignant. Specifically, the intensity to time ratio of light emitted from the background bodily tissue during an initial time period shortly after administration of a suitable contrast agent to a subject (i.e. an initial uptake phase) is delayed, if the target bodily tissue is malignant. The inventors have also found that the intensity of light emitted from the target bodily tissue is higher than the intensity of light emitted from the background bodily tissue in a later time period after administration of the contrast agent to the subject (i.e. a washout phase), if the target bodily tissue is malignant. These differences in light emitted from the target bodily tissue and the background bodily tissue may be due to the different amounts of contrast agent present in the different tissues at different times after administration or may be due to a localised increase in intensity of the contrast agent in the target bodily tissue due to some other mechanism.

[0021] These effects are believed to be due to the differing pharmacokinetic properties of malignant tissue structures to similar benign tissue structures or adjacent healthy tissue. Therefore, these tissue types have been found to absorb and excrete compounds such as contrast agents at different rates, which provide different intensities of light emission at different time points after dosing of the contrast agent.

[0022] The method of processing images may further comprise: generating, using the trained ML model, a perfusion score that indicates how well perfusion has occurred in the first spatial region and the second spatial region; and outputting the generated perfusion score. The perfusion score may be based on any one or more of: fluorescence intensity, rate of decay of the contrast agent, and tissue response to the contrast agent. It will be understood that the perfusion score may depend on the type of contrast agent. The perfusion score may be between 0 and 1 , for example, where the score is higher when perfusion occurs in a way that is expected in normal or healthy tissue, and the score is lower when perfusion occurs less well, as expected in unhealthy / diseased tissue. It will be understood that the value of the upper limit for the perfusion score may depend on other factors, such as gender, ethnicity, age, etc.

[0023] Generating a perfusion score may comprise: generating a perfusion score after analysing all images in the sequence of images. That is, a score may only be generated once the sequence of images have all been analysed. This may be because the score depends on rate of perfusion, which is only determined from the full sequence. In some cases, the perfusion score may be generated for each image as and when the image is received, and the perfusion score may be dynamically updated as new images in the sequence are received. The perfusion score may be updated not only as new images are received, but also as a result of intraoperative changes that occur while the images are being obtained, e.g. due to repositioning of the surgical field of view or changes in vascular clamping status.

[0024] Outputting the generated perfusion score may comprise: adding the generated perfusion score to the augmented image. That is, the perfusion score may be added to the augmented image itself. For example, the perfusion score for each tissue or spatial region may be added on or next to the relevant region in the image. In other cases, the perfusion score may be outputted separately from the augmented image.

[0025] The method of processing images may further comprise: inputting, into the trained ML model, sensitivity information to adjust the analysis performed by the trained ML model. The ML model may have been trained to analyse images captured fora variety of different patients, but some patient characteristics may alter how perfusion is expected to occur in healthy tissue. For example, perfusion may occur differently in men and women, or people under 50 vs people over 50, or people of different ethnicities. Thus, what is considered normal or good perfusion may vary across patient types (e.g. genders, age groups, ethnicities, etc.), which will impact determining the optimal boundary between the spatial regions I tissues. These characteristics are not evident from the images themselves, so the method may comprise inputting such characteristics so that the ML model can analyse the images with these characteristics in mind.

[0026] Inputting sensitivity information may comprise inputting information patient-specific information, wherein the patient-specific information includes any one or more of: age; health status; ethnicity; gender; and disease status. It will be understood that this is a non-exhaustive list of example patient-specific information or characteristics.

[0027] The analysing may therefore comprise analysing the received sequence of images using a trained machine learning, ML, model and the sensitivity information.

[0028] The step of receiving the sequence of images (also referred to herein as a time series of images) may comprise receiving images acquired using an imaging device. The imaging device may be, for example, a camera or a video camera such as a CCD or a CMOS device or a finger probe. The sequence of images may be, for example, photographs acquired periodically and / or frames from a video.

[0029] In one example, the step of receiving the sequence of images comprises acquiring the time series of images using a fluorescence imaging device, for example a camera or a video camera such as a CCD or a CMOS device or a finger probe, for example photographs acquired periodically and / or acquired frames from a video, wherein the contrast agent comprises and / or is a fluorescent dye. In one example, the method comprises acquiring the time series of images using fluorescence angiography. In one example, images comprise and / or are white light, near-infrared and / or fluorescence images. In one example, the method comprises stabilising the images, for example using a geometric transformation such as an affine geometric transformation. In one example, the method comprises extracting fluorescence intensities and / or time-related features from the images. In one example, the method comprises producing heatmaps to visualise fluorescence intensity and / or time-related features, for example from the extracted fluorescence intensities and / or time-related features from the images.

[0030] Generally speaking, the sequence of images may comprise a minimum of two images (e.g. a first image and a second image) captured at sufficiently different stages of the perfusion process. However, it will be understood that the prediction of the boundary between the two spatial regions is more accurate when the sequence of images comprises more than two images.

[0031] In one example, the first image is an RGB image ora greyscale image. In one example, the first time series of images includes M images, wherein M is a natural number greater than or equal to 2, for example 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10000, 20000, 50000 or more. The second image may be as described with respect to the first image. The M images may be as described with respect to the first image. In one example, a first field of view of the first time series of images of the organ is fixed or constant (i.e. the first time series of images are of the same view of the organ).

[0032] In one example, the first image and / or the second image are generated from signals acquired using an imaging device, for example a CCD or a CMOS device, or a finger probe. In one example, the imaging device is configured to acquire white light, near-infrared and / or fluorescence images (i.e. the first image and / or the second image comprise and / or are white light, near-infrared and / or fluorescence images).

[0033] In one example, the first time period (i.e. during which the first time series of images is acquired) is in a range from 1 second to 10 minutes, preferably in a range from 10 seconds to 5 minutes, more preferably in a range from 30 seconds to 2 minutes, for example 1 minute..

[0034] In one example, the first time period is immediately after (i.e. the first time duration is zero) the first perfusion of the organ with the first contrast agent. In one example, the first time period is a first time duration after the first perfusion of the organ with the first contrast agent. In one example, the first time duration (i.e. during which the first time series of images is acquired) is in a range from 1 second to 10 minutes, preferably in a range from 10 seconds to 5 minutes, more preferably in a range from 30 seconds to 2 minutes, for example 1 minute.

[0035] The images may not only show the organ (or part thereof), but may also show surgical equipment (e.g. clamps), other organs, skin, and so on. Thus, the trained ML model may perform a pre-processing step to segment the parts of the images that are to be analysed, so that only the organ (or part thereof) is analysed by the model. Any known segmentation process may be used by the trained ML model.

[0036] In some cases, the step of receiving the sequence of images may comprise receiving the sequence of images in real time or near real time after perfusion of the contrast agent. This means that the trained ML model performs the analysis on the images in real time or near real time.

[0037] Thus, in one example, the method comprises and / or is a real-time method. It should be understood that the real-time method is compatible with a timescale of the method. In this example, discriminating between the first tissue status and the second tissue status of the organ or a part thereof is provided responsive to receiving the images, for example at the same rate. In this way, the discriminating is provided sufficiently quickly to feedback into a process including the method, for example during surgery whereby a surgeon may perform the surgery based on a result of the discriminating. In other words, a surgeon may be guided by the discriminating, for example. In this way, patient outcome is improved. The sequence of images may be frames of a video. For example, the video may be a 30 frames per second video. Each frame of the video may be analysed by the trained ML model. Alternatively, every n frames of the video may be analysed for more efficient processing, where n is an integer. For example, every fifth frame of the video may be analysed.

[0038] In the cases where the images are received in real time or near real time, the step of analysing the received sequence of images may comprise analysing the received sequence of images in real time or near real time.

[0039] In one example, analysing the received sequence of images in real time or near real time may comprise: receiving a first set of images of the sequence of images; generating, using the first set of images and the trained ML model, a prediction for the optimal boundary between the first spatial region and the second spatial region; receiving a second set of images of the sequence of images; and updating the generated prediction using the second set of images and the trained ML model. In other words, the analysis may be performed as the images are being received, or in chunks or sets of images for computational efficiency. This means that the ML model generates a changing prediction of the boundary between the two spatial regions, which changes as more images are received and the perfusion process continues. That is, the prediction at the start of the perfusion process may not be very accurate or reliable because it is based on relatively few images and because the differences between the spatial regions may not yet be apparent. However, as time and the perfusion process progresses, the ML model has more images to analyse and compare and the prediction evolves and improves in accuracy. Thus, the ML model effectively updates its prediction as more images are received and processed.

[0040] This process of updating the prediction is preferably repeated whenever more images are received. Thus, the method may further comprise updating the generated prediction when a further set of images of the sequence of images is received.

[0041] Generating an augmented image is also referred to herein as generating a visualisation.

[0042] In one example, the method comprises generating a visualization (for example, an image) of the organ based on a result of discriminating between the first tissue status and the second tissue status of the organ or a part thereof. In this way, the organ and the result may be visualized, for example by displaying the generated visualization on a display, for example in a surgical theatre to a surgeon. In one example, generating the visualization comprises including an estimated transection point therein. In this way, an optimum transection point can be displayed to a surgeon, for example.

[0043] In one example, generating the visualization of the organ based on the result of discriminating between the first tissue status and the second tissue status of the organ or a part thereof comprises distinguishing respective spatial regions included in the set of spatial regions. In this way, the set of spatial regions may be mutually distinguished visually, for example based on respective relative perfusions thereof. In this way, a surgeon may be guided, for example continuously, to transect the organ at an optimum transection point, for example.

[0044] In one example, distinguishing the respective spatial regions included in the set of spatial regions comprises visually distinguishing the respective spatial regions included in the set of spatial regions, for example using colour, contour lines, reference signs, such as alphanumeric characters, and / or markers, such as graphics. In this way, a surgeon may be further guided, for example continuously, to transect the organ at an optimum transection point such as guided by the visual distinguishing, for example.

[0045] In one example, generating the visualization of the organ based on the result of discriminating between the first tissue status and the second tissue status of the organ or a part thereof comprises indicating respective boundaries between the respective spatial regions included in the set of spatial regions. In this way, a surgeon may be guided, for example continuously, to transect the organ at an optimized transection point such as a boundary, for example.

[0046] In one example, the method comprises displaying the generated visualization of the organ during a second time period after the first time period, for example in a surgical theatre to a surgeon such as in real-time.

[0047] In one example, displaying the generated visualization on a display comprises displaying the generated visualization on an augmented reality display such as displaying the generated visualization overlaying a real time image of the organ. In this way, a surgeon may be guided, for example continuously, to transect the organ at an optimized transection point such as a boundary, for example. In one example, displaying the generated visualization on a display comprises displaying the generated visualization on a virtual reality display such as displaying the generated visualization overlaying a computer generated or stored image of the organ. In this way, a surgeon during training may be guided, for example continuously, to transect the organ at an optimized transection point such as a boundary, for example.

[0048] Generally speaking, there are two points in time when the augmented image may be generated: in real time or near real time, and after the full sequence of images has been received and processed.

[0049] Thus, in one example, generating an augmented image may comprise adding an overlay to the sequence of images in real time or near real time. In this case, the generated image may be displayed as soon as it is generated, such that a surgeon is provided with a real time / near real time indication of the ML model’s prediction of the spatial regions. As noted above, since the ML model’s prediction may evolve, the generated image also evolves because the overlay will change. The overlay may be, for example, a perfusion heatmap that visually distinguishes well-perfused regions from ischemic regions. The heatmap may be colour-coded, such that different colours indicate different perfusion states. For example, green may indicate optimal perfusion (including that judged by expert surgeons as optimal), and red may indicate ischemic risk.

[0050] In another example, generating an augmented image may comprise: adding the overlay to a final image in the sequence of images. That is, the overlay is only added to (i.e. superimposed on) the final image of the sequence of images, after the full sequence of images has been received and processed by the ML model. The overlay may be, for example, a perfusion heatmap that visually distinguishes well-perfused regions from ischemic regions.

[0051] In an alternative example, generating an augmented image may comprise: receiving a further image of the organ or part of the organ; and adding the overlay to the received further image. That is, the overlay is only added after the full sequence of images has been received and processed by the ML model. Furthermore, the overlay may be added to a new image (or set of images, i.e. a video) of the organ (i.e. not an image that was processed by the ML model). Preferably, receiving a further image may comprise receiving a further real-time image captured of the organ or part of the organ, for example, after the perfusion process has completed.

[0052] The step of generating an augmented image by adding an overlay to an image may comprise: adding an overlay comprising any one or more of: a colour over one of the first and second spatial regions; a different colour over each of the first and second spatial regions; a contour line between the first and second spatial regions; a reference sign; alphanumeric characters; a graphical marker; a line or marker indicating the determined optimal boundary; and a colour-coded line or marker indicating the determined optimal boundary.

[0053] The method may further comprise: displaying the generated augmented image on a display, an augmented reality display and / or on a virtual reality display.

[0054] The step of analysing the received sequence of images using a trained machine learning, ML, model, may comprise: determining, for each pixel in each image, an intensity value corresponding to an intensity of the contrast agent; comparing the intensity value of each pixel in the sequence of images; and predicting, based on the comparing, an optimal boundary between the first spatial region and the second spatial region.

[0055] The step of comparing the intensity value of each pixel in the sequence of images may comprise comparing a rate of change of the intensity value of each pixel across the sequence of images.

[0056] The method generally comprises discriminating between the first tissue status and the second tissue status of the organ or a part thereof using the trained machine learning algorithm. In this way, healthy and malperfused tissue may be discriminated, as described above.

[0057] In one example, discriminating between the first tissue status and the second tissue status of the organ or a part thereof using the trained machine learning algorithm comprises classifying the first tissue status and / or the second tissue status of the organ or a part thereof. In this way, the first tissue status and / or the second tissue status of the organ or a part thereof may be classified according to perfusion thereof, for example as described above.

[0058] In one example, classifying the first tissue status and / or the second tissue status of the organ or a part thereof comprises classifying the first tissue status and / or the second tissue status of the organ or a part thereof according to perfusion thereof, for example as described above.

[0059] In one example, discriminating between the first tissue status and the second tissue status of the organ or a part thereof using the trained machine learning algorithm comprises discriminating between the first tissue status and the second tissue status of the organ or a part thereof using the trained machine learning algorithm based on presence or absence of the contrast agent in the images. In this way, the machine learning algorithm discriminates between the first tissue status and the second tissue status of the organs or a parts thereof based on presence or absence of the contrast agent in the images.

[0060] In one example, discriminating between the first tissue status and the second tissue status of the organ or a part thereof using the trained machine learning algorithm based on presence or absence of the contrast agent in the images comprises discriminating between the first tissue status and the second tissue status of the organ or a part thereof using the trained machine learning algorithm based on intensities (for example, pixel intensities of the first image and / or the second image) due to the first contrast agent in the images. In this way, the machine learning algorithm discriminates between the first tissue status and the second tissue status of the organ or a part thereof based on the intensities (for example, pixel intensities) due to the contrast agent in the images.

[0061] In one example, discriminating between the first tissue status and the second tissue status of the organ or a part thereof using the trained machine learning algorithm comprises discriminating between the first tissue status and the second tissue status of the organ or a part thereof using the trained machine learning algorithm along a line in the images. In this way, the machine learning algorithm discriminates between the first tissue status and the second tissue status of the organ or a part thereof along the line.

[0062] In one example, the line spans (partly or completely) the set of spatial regions, for example the first spatial region and the second spatial region. In one example, the line is predetermined. In one example, the method comprises drawing the line on one or more images. In one example, discriminating between the first tissue status and the second tissue status of the organ or a part thereof using the trained machine learning algorithm along the line comprises discriminating between the first tissue status and the second tissue status of the organ or a part thereof using the trained machine learning algorithm along a series of points, including a first point and a second point, in one or more images.

[0063] In one example, the method comprises including a line, as described above, in one or more images. In one example, the method comprises creating a rectangle based on the line, for example a rectangular region centred around the line, having regions above and below the line. In one example, the method comprises calculating a median value at every (x or y) coordinate to condense the information back into a 1 dimensional single line.

[0064] In one example, the method comprises estimating a transection point, based on a result of the discriminating. In this way, an optimum transection point can be made based on the specific perfusion profile of an individual intraoperatively.

[0065] Generally, a trained machine learning algorithm is also known as a machine learning model.

[0066] In one example, the machine learning algorithm comprises and / or is a neural network. In one example, the machine learning algorithm comprises and / or is a K-Nearest Neighbour (KNN) model. Logistic regression, naive Bayes, decision tree, random forest, support vector machines (SVMs) and / or gradient boosting models may also be suitable.

[0067] In one example, the machine learning algorithm comprises and / or is a recurrent neural network (RNN). In one example, the machine learning algorithm comprises and / or is a long short-term memory (LSTM) model. Generally, a LSTM model is an advanced version of RNN architecture, designed to model chronological sequences (i.e. time series) and their long- range dependencies more precisely than conventional RNNs. Surprisingly, the inventors have identified that LSTMs are suitable and may be preferable for the discriminating described herein. In one example, the machine learning algorithm comprises and / or is a bidirectional long short-term memory (bi-LSTM) model. In one example, training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises deep learning, DL, for example using a LSTM or a bi-LSTM model. Convolutional Neural Networks (CNNs), Long Short Term Memory Networks (LSTMs), Recurrent Neural Networks (RNNs), Generative Adversarial Networks (GANs), Radial Basis Function Networks (RBFNs), Multilayer Perceptrons (MLPs), Self Organizing Maps (SOMs), Deep Belief Networks (DBNs), Restricted Boltzmann Machines( RBMs) and / or Autoencoders may also be suitable.

[0068] In a specific embodiment, the ML model may comprise a convolutional neural network, CNN, which has been trained using supervised learning and a dataset of expert- 1 a be I led images. The images may be ICG fluorescence images. The labels may be applied by experts, i.e. surgeons and other expert medical professionals. The labels may indicate safe and unsafe transection margins. As noted below, the labels may be based on an aggregation of expert opinion.

[0069] In a specific embodiment, the ML model may also comprise a recurrent neural network, RNN, or a transformer-based architecture. RNNs and transformers are useful for analysing changes in fluorescence intensity over time, i.e. determining how the fluorescence intensity changes over a series of frames captured for the same organ. This can improve accuracy in perfusion assessment, because the rate of change may help determine the boundaries between tissue types more accurately compared to looking at a single pre-perfusion and a single post-perfusion image.

[0070] More details on the ML model are explained below, particularly with reference to training of the model.

[0071] The step of receiving the sequence of images comprises receiving the sequence of images after a single perfusion of the organ (or part thereof) with the contrast agent. The contrast agent may be, or may comprise, a fluorescent dye. For example, the fluorescent dye may be indocyanine green or methylene blue. Thus, the step of receiving the sequence of images may comprise receiving images obtained via fluorescence angiography.

[0072] Preferably, only a single perfusion of the organ or part thereof is performed with the contrast agent. That is, the perfusion of the organ or part thereof with the contrast agent (or any other contrast agent) is not repeated. In this way, the ML model makes a prediction of the optimal boundary between the first and second regions using a singly perfused organ or part thereof. By limiting to a single perfusion, speed is increased while complexity is reduced, thereby providing a method that is more compatible with conventional surgical practice.

[0073] The contrast agent comprises and / or is a fluorescent dye, for example indocyanine green or methylene blue, and wherein obtaining the first time series of images, including the first image and the second image, of the organ or part thereof comprises fluorescence angiography.

[0074] Indocyanine green (ICG) is a fluorescent dye, which emits fluorescence on excitation by a NIR light source at a wavelength of approximately 785 nm. The emitted fluorescence (approximate wavelength band of 800-850 nm) can be captured (imaged) and processed. Indocyanine green (ICG) is a sterile, water-soluble but relatively hydrophobic tricarbocyanine molecule. Following intravenous injection, ICG is rapidly bound to plasma proteins with minimal leakage into the interstitium and is excreted by the liver in bile about 8 min after injection. This emission intensity signal can then be used to accurately classify perfusion, through the use of biophysical modelling and image analysis techniques. Similarly, methylene blue (MB) can be excited from 550-700 nm, with an emission centered around 690 nm. Other fluorescent dyes are known. Fluorophore molecules may be either utilized alone, or serve as a fluorescent motif of a functional system. Based on molecular complexity and synthetic methods, fluorophore molecules may be generally classified into four categories: proteins and peptides, small organic compounds, synthetic oligomers and polymers, and multicomponent systems. See, for example, https: / / en.wikipedia.org / wiki / Fluorophore.

[0075] As noted above, the first spatial region may comprise and / or be healthy tissue, for example tissue having normal or relatively better perfusion. Similarly, the second spatial region may comprise and / or be malperfused tissue, for example having subnormal or relatively poorer perfusion (malperfusion) such as malperfused, diseased or malignant tissue. In this way, healthy and malperfused tissue may be discriminated.

[0076] In one example, the organ comprises and / or is the digestive tract, for example the colorectal and / or internal gastrointestinal tract, or a urinary organ or is an endocrine organ. In one example, the organ is in vivo i.e. in a patient. In one example, the organ is ex vivo, for example a transplant organ received from a donor before transplanting into a patient. In one example, the organ comprises and / or is a human or an animal organ i.e. originating from a human or an animal. In one example, the organ comprises and / or is an engineered organ, for example a lab-grown organ. For example, a donor section of an organ, such as from a human or an animal donor or from an engineered organ, may be introduced into a patient, for example to replace a diseased section of the patient’s organ. In one example, the organ comprises and / or is a model organ, for example for surgery training purposes.

[0077] In one example, the second spatial region has a second tissue status. In one example, the first tissue status comprises and / or is healthy tissue for example having normal or relatively better perfusion. In one example, the second tissue status comprises and / or is malperfused tissue, for example having subnormal or relatively poorer perfusion (malperfusion) such as diseased or malignant tissue. In this way, healthy and malperfused tissue may be discriminated.

[0078] Generally, a spatial region (also known as a region of interest) is a surface or near- surface region of the organ, having a surface area, that is affected by perfusion and that may be imaged. In one example, the first spatial region and the second spatial region comprise and / or are the same region. In one example, the first spatial region and the second spatial region comprise and / or are mutually overlapping regions. In one example, the first spatial region and the second spatial region comprise and / or are mutually adjacent regions. In one example, the first spatial region and the second spatial region comprise and / or are contiguous regions. In one example, the first spatial region and the second spatial region comprise and / or are mutually non-overlapping regions. In one example, the set of spatial regions includes R spatial regions, wherein R is a natural number greater than or equal to 2, for example 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 50, 100, 200, 500, 1000 or more. In one example, the first spatial region has a size of a x b pixels (e.g. of a CCD or CMS imaging device for acquiring the images), wherein a and b are each natural numbers greater than or equal to 1 , for example 2, 3, 4, 5, 6, 7, 8, 9, 10, 16, 20, 32, 50, 64, 100, 128, 200, 256, 500, 512, 1000, 1024 or more. In one example, a = b. In one example, a = b = 1. In this way, discrimination is performed on a per pixel basis.

[0079] As noted above, the method is implemented by the computer comprising a processor and memory. That is, the method is a computer-implemented method. It should be understood that while the method uses data (i.e. the first time series of images and the first second series of images) obtained after perfusion of the organ, the computer implemented method is not practised on the human or animal body.

[0080] Thus, generally speaking, the method of the first aspect may comprise and / or be a method for use live real-time in theatre analysing indocyanine green fluorescence angiography imaging signal in colorectal resection surgery, comprising: administering fluorescence dye (indocyanine green) intravenously; drawing a line on the bowel where perfusion is deemed questionable by a surgeon; and displaying possible output classes of experts’ interpretation on the images of the bowel in a graphical user interface (GUI) in a colour coded format following the GUI.

[0081] In one example, the possible output classes comprise three possible output classes of experts’ interpretation on the images of the bowel in the colour coded format, including: (i) expert zone (green), representing where an ICGFA expert would recommend staple placement; (ii) good zone (blue), representing region of the bowel segment proximal to the recommended staple placement zone (perfused side of the bowel); and (iii) poor zone (red), representing poorly perfused region of the bowel.

[0082] In one example, the method is performed by a software system for analysing fluorescence signals using a sequence to sequence classification approach, the system configured to: stabilize video frames captured in white light, near-infrared, and fluorescence overlay imager using affine geometric transformation; extract fluorescence intensity and time- related features from grid regions; generate heatmaps to visualise fluorescence intensity and time-related features; facilitate user interaction by allowing the drawing of a line on the bowel image, which creates a rectangular region centered around the line (regions above and below the drawn line), followed by calculating a median value at every (x or y) co-ordinate to condense the information back into a 1 dimensional single line; employ sequence-based artificial intelligence methods, such as long short term memory models, for making predictions; and display and plot the predictions on an image of the bowel video within a live video feed.

[0083] In one example, the artificial intelligence models are trained on a panel of ICGFA experts’ annotations of colorectal resection surgery videos where patients did not suffer anastomotic leaks. In a second, related aspect of the present techniques, there is provided a system for processing images of an organ, the system comprising: at least one processor coupled to memory, for: receiving a sequence of images of an organ or part of an organ, each image in the sequence depicting a set of spatial regions including a first spatial region having a first tissue status and a second spatial region having a second tissue status, wherein the sequence of images depict the organ or the part of the organ over a period of time after perfusion with a contrast agent; analysing the received sequence of images using a trained machine learning, ML, model, wherein the trained ML model is trained to determine both the tissue of interest and, within it, an optimal boundary between the first spatial region and the second spatial region based on how the contrast agent perfuses through the first and second spatial regions over time; generating an augmented image of the organ or the part of the organ by adding an overlay to an image, wherein the overlay depicts the determined optimal boundary between the first spatial region and the second spatial region.

[0084] The features described above with respect to the first aspect apply equally to the second aspect and therefore, for the sake of conciseness, are not repeated.

[0085] The at least one processor may be further configured to: generate, using the trained ML model, a perfusion score that indicates how well perfusion has occurred in the first spatial region and the second spatial region; and output the generated perfusion score.

[0086] The system may further comprise storage for storing the sequence of images, the generated augmented image, and / or the generated perfusion score. This may be useful because it enables surgeons to perform post-operative reviews. It may also be useful to analyse how useful the trained model was for clinical outcomes.

[0087] The system may further comprise at least one imaging device for capturing the sequence of images.

[0088] The system may further comprise at least one user interface for receiving, from a user, an instruction to begin receiving the sequence of images. For example, the user interface may be an audio interface that is able to detect a spoken instruction (e.g. “Start analysis”), or may be a camera that is able to detect a gesture-based instruction (e.g. a wave). The instruction is given when the contrast agent has been provided in the organ (or part thereof), so that the rate of perfusion can be determined. This also enables the ML model to be used in a “hands-free” manner, i.e. without requiring the surgeon to step-away from what they are doing and indeed activate the process from within the sterile field to start the processing using the ML model.

[0089] The system as may further comprise a display device for displaying the generated augmented image. The display device may be an augmented reality overlay on a surgical display, or may be a head-up display, for example.

[0090] Method of Training: A third aspect provides a method of training a machine learning algorithm to discriminate between a first tissue status and a second tissue status of organs or a parts thereof, the method implemented by a computer comprising a processor and a memory, the method comprising: obtaining a training dataset, wherein the training dataset comprises a set of sample data, including first sample data, for a corresponding set of organs or parts thereof including a first organ or part thereof, wherein the first sample data comprises a first time series of images, including a first image and a second image, of the first organ or part thereof, having a set of spatial regions including a first spatial region, having the first tissue status, and a second spatial region, having the second tissue status, during a first time period after a first perfusion of the first organ or part thereof with a first contrast agent; and training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data.

[0091] A related aspect of the present techniques provides a computer-implemented method of training a machine learning, ML model to process images of an organ, the method comprising the following steps. Firstly, receiving a training dataset comprising a plurality of sets of images, each set of images comprise a sequence (time series) of images of an organ (or part of an organ). Each image in each sequence depicts a set of spatial regions including a first spatial region having a first tissue status and a second spatial region having a second tissue status, wherein the sequence depicts the organ (or part of the organ) over a period of time after perfusion with the contrast agent. Secondly, training the ML model, using the training dataset, to determine an optimal boundary between the first spatial region and the second spatial region based on how the contrast agent perfuses through the first and second spatial regions over time. In some cases, the method may further comprise training the ML model to generate an augmented image by adding an overlay to an image, wherein the overlay depicts the determined optimal boundary between the first and second spatial regions. In other cases, the augmented image may be generated by another ML model or other process.

[0092] Advantageously, the present training techniques exploit correlations in multi-site, sequences of time-series observations of ICGFA perfusion. The present training approach exploits subtle correlations, only observable when the sequential nature of the regions of interest, ROIs, is included in the processing, to characterise the entire sequence of ROIs. While the extraction and use of NIR data to characterise regions of a video is generally known, this is the first time that correlations implicit in the sequence of features has been used to improve the accuracy of an Al algorithm.

[0093] Furthermore, the present training approach employs multiple domain expert input, which reduces the technical problems of ground truth bias and signal noise error. In several application domains, such as medicine, forensic science and ecology, judgements are based on the expertise of domain experts, resulting in variations in the what constitutes a ‘correct’ decision. T raining the Al model using the present approach uses feedback from multiple expert users to represent a “shared expert view”. In this way variations in expert decision making can be incorporated into the processing of the Al. Techniques exist in prior art to incorporate multiple user inputs into training (e.g. reinforcement learning, multi-party voting schemes). However, the present approach is the first to apply multi-expert techniques in the domain of ICGFA imaging systems.

[0094] In this way, the machine learning algorithm is trained to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the time series of images of the perfused organs or parts thereof. In this way, judgement of tissue quality is improved, for example for reconstruction after disease excision, thereby better guiding a surgeon to transect the organ at an appropriate transection point, since the optimized transection point of the organ has the best perfusion, thereby improving healing after transection, for example. Alternatively, it may provide information regarding tissue vitality and therefore function to a similar standard as an expert practitioner in the field may interpret the fluorescence signalling of perfusion. Alternatively, distinctive microvascular blood flow patterns indicative of neoplastic and in particular malignant disease may be revealed

[0095] The step of obtaining a training dataset may comprise: obtaining a first time series of images that is annotated with at least one annotation by an expert.

[0096] Preferably, the step of obtaining a training dataset may comprises: obtaining a plurality of the first time series of images, each time series of images being annotated with at least one annotation by a different expert; and aggregating the annotations for the first time series of images to generate a single annotated first time series of images for use during the training. That is, the training dataset may comprise multiple time series of images, obtained from different patients. In addition, the time series of images for a specific patient may be annotated by multiple experts (e.g. surgeons). For example, surgeon A may annotate where they think the optimal boundary between the first spatial region and second spatial region is for a specific time series of images, and surgeon B, surgeon C and surgeon D do the same for the same time series of images. In this way, four experts provide their opinions on the optimal boundary, based on their assessment of the perfusion. The multiple annotations may be aggregated or combined in some way to generate a single annotation for that time series of image that is used to train the ML model. For example, an average of the optimal boundary may be calculated, and this average may be used as the effective “label” for this time series of images. The same occurs for all other time series data in the training dataset. In other words, the training data may be labelled not by a single expert, but by aggregating different opinions from multiple experts. This reduces bias in the training data and improves generalisability of recommendations. Generally, tissues require sufficient perfusion to function and heal after surgery. Conventionally, operations build in such consideration by surgeon judgement regarding cut (transection) lines related to tissue resection or tissue status relevant to postoperative function. The inventors have developed a computational method including machine learning algorithm and clinical process that enables indication of expert surgeon judgement relevant to the most appropriate site for subsequent incision or judgment regarding character and / or subsequent function based on performing perfusion assessment The method involves the use of contrast agents (also known as dyes) and electromagnetic radiation (EMR), for example infra red (IR), to excite the contrast agents and amplify the inherent perfusion patterns within tissues relevant to the tissue of interest including at sites potentially at risk of malperfusion or harbouring disease and thereafter a computational method / machine learning algorithm to compare the imagery to indicate perfusion.

[0097] The first perfusion of the first organ or part thereof with the first contrast agent is a single perfusion of the first organ or part thereof with the first contrast agent.

[0098] The first contrast agent comprises and / or is a fluorescent dye, for example indocyanine green or methylene blue, and wherein obtaining the first time series of images, including the first image and the second image, of the organ or part thereof comprises fluorescence angiography.

[0099] Some or all of the images in the training dataset may comprise one or more labels, which are applied by experts. The labels may indicate different spatial regions and / or a boundary between spatial regions.

[0100] Training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises classifying the first tissue status and / or the second tissue status of the organs or a parts thereof.

[0101] Training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof based on presence or absence of the first contrast agent in the first image and / or the second image and / or sequence of images.

[0102] Training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof based on presence or absence of the first contrast agent in the first image and / or the second image comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof based on intensities due to the first contrast agent in the first image and / or the second image and / or sequence of images. Training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof along a line in the first image and / or the second image and / or sequence of images.

[0103] Training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof along the line in the first image and / or the second image comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof along a series of points, including a first point and a second point, in the first image and / or the second image and / or sequence of images.

[0104] Training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises deep learning, DL.

[0105] The method is training a machine learning algorithm to discriminate (i.e. distinguish, differentiate) between the first tissue status and the second tissue status of organs or a parts thereof. In one example, the first tissue status comprises and / or is healthy tissue, for example tissue having normal or relatively better perfusion. In one example, the second tissue status comprises and / or is malperfused tissue, for example having subnormal or relatively poorer perfusion (malperfusion) such as devascularised, diseased or malignant tissue. In this way, healthy and malperfused tissue may be discriminated. In one example, the organ comprises and / or is the digestive tract, for example the colorectal and / or internal gastrointestinal tract, or a urinary organ or an endocrine organ. In one example, the organ is in vivo i.e. in a patient. In one example, the organ is ex vivo, for example a transplant organ received from a donor before transplanting into a patient. In one example, the organ comprises and / or is a human or an animal organ i.e. originating from a human or an animal. In one example, the organ comprises and / or is an engineered organ, for example a lab-grown organ. For example, a donor section of an organ, such as from a human or an animal donor or from an engineered organ, may be introduced into a patient, for example to replace a diseased section of the patient’s organ. In one example, the organ comprises and / or is a model organ, for example for surgery training purposes.

[0106] In one example, the second spatial region has a second tissue status. In one example, the first tissue status comprises and / or is healthy tissue, for example benign tissue, for example having normal or relatively better perfusion. In one example, the second tissue status comprises and / or is malperfused tissue, for example having subnormal or relatively poorer perfusion (malperfusion) such as diseased or malignant tissue. In this way, healthy and malperfused tissue may be discriminated. Generally, a spatial region (also known as a region of interest) is a surface or near- surface region of the organ, having a surface area, that is affected by perfusion and that may be imaged. In one example, the first spatial region and the second spatial region comprise and / or are the same region. In one example, the first spatial region and the second spatial region comprise and / or are mutually overlapping regions. In one example, the first spatial region and the second spatial region comprise and / or are mutually adjacent or near adjacent regions. In one example, the first spatial region and the second spatial region comprise and / or are contiguous regions. In one example, the first spatial region and the second spatial region comprise and / or are mutually non-overlapping regions. In one example, the set of spatial regions includes R spatial regions, wherein R is a natural number greater than or equal to 2, for example 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 50, 100, 200, 500, 1000 or more. In one example, the first spatial region has a size of a x b pixels (e.g. of a CCD or CMS imaging device for acquiring the images), wherein a and b are each natural numbers greater than or equal to 1 , for example 2, 3, 4, 5, 6, 7, 8, 9, 10, 16, 20, 32, 50, 64, 100, 128, 200, 256, 500, 512, 1000, 1024 or more. In one example, a = b. In one example, a = b = 1. In this way, discrimination is performed on a per pixel basis.

[0107] The method is implemented by the computer comprising the processor and the memory. That is, the method is a computer implemented method. It should be understood that while the method uses data (i.e. the first time series of images and the first second series of images) obtained after perfusion of the organ, the computer implemented method is not practised on the human or animal body.

[0108] The method comprises obtaining (for example, from storage) the training dataset, wherein the training dataset comprises the set of sample data, including the first sample data, for the corresponding set of organs or parts thereof including the first organ or part thereof, wherein the first sample data comprises the first time series of images, including the first image and the second image, of the first organ or part thereof, having the set of spatial regions including the first spatial region, having the first tissue status, and the second spatial region, having the second tissue status, during the first time period after a first perfusion of the first organ or part thereof with the first contrast agent.

[0109] In one example, the method comprises acquiring the first time series of images using an imaging device for example a camera or a video camera such as a CCD or a CMOS device or a finger probe, for example photographs acquired periodically and / or acquired frames from a video. In one example, the method comprises acquiring the first time series of images using a fluorescence imaging device for example a camera or a video camera such as a CCD or a CMOS device or a finger probe, for example photographs acquired periodically and / or acquired frames from a video, wherein the first contrast agent comprises and / or is a fluorescent dye. In one example, the method comprises acquiring the first time series of images using fluorescence angiography. In one example, the first image and / or the second image comprise and / or are white light, near-infrared and / or fluorescence images. In one example, the method comprises stabilising the first image and / or the second image, for example using a geometric transformation such as an affine geometric transformation. In one example, the method comprises extracting fluorescence intensities and / or time-related features from the first image and / or the second image. In one example, the method comprises producing heatmaps to visualise fluorescence intensity and / or time-related features, for example from the extracted fluorescence intensities and / or time-related features from the first image and / or the second image.

[0110] In one example, the first image is an RGB image ora greyscale image. In one example, the first time series of images includes M images, wherein M is a natural number greater than or equal to 2, for example 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10000, 20000, 50000 or more. The second image may be as described with respect to the first image. The M images may be as described with respect to the first image. In one example, a first field of view of the first time series of images of the organ is fixed or constant (i.e. the first time series of images are of the same view of the organ).

[0111] In one example, the first image and / or the second image are generated from signals acquired using an imaging device, for example a CCD or a CMOS device, or a finger probe. In one example, the imaging device is configured to acquire white light, near-infrared and / or fluorescence images (i.e. the first image and / or the second image comprise and / or are white light, near-infrared and / or fluorescence images).

[0112] In one example, the first time period (i.e. during which the first time series of images is acquired) is in a range from 1 second to 10 minutes, preferably in a range from 10 seconds to 5 minutes, more preferably in a range from 30 seconds to 2 minutes, for example 1 minute.

[0113] In one example, the first time period is immediately after (i.e. the first time duration is zero) the first perfusion of the organ with the first contrast agent. In one example, the first time period is a first time duration after the first perfusion of the organ with the first contrast agent. In one example, the first time duration (i.e. during which the first time series of images is acquired) is in a range from 1 second to 10 minutes, preferably in a range from 10 seconds to 5 minutes, more preferably in a range from 30 seconds to 2 minutes, for example 1 minute.

[0114] In one example, the first perfusion of the first organ or part thereof with the first contrast agent is a single perfusion of the first organ or part thereof with the first contrast agent. In other words, the first perfusion of the first organ or part thereof with the first contrast agent is the only (i.e. a single) perfusion of the first organ or part thereof with the first contrast agent. That is, the perfusion of the first organ or part thereof with the first contrast agent (or any other contrast agent) is not repeated. In this way, the machine learning algorithm is trained to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using singly perfused organs or parts thereof. By limiting to a single perfusion, speed is increased while complexity is reduced, thereby providing a method that is more compatible with conventional surgical practice.

[0115] In one example, the first contrast agent comprises and / or is a fluorescent dye, for example indocyanine green or methylene blue, and wherein obtaining the first time series of images, including the first image and the second image, of the organ or part thereof comprises fluorescence angiography.

[0116] Indocyanine green (ICG) is a fluorescent dye, which emits fluorescence on excitation by a NIR light source at a wavelength of approximately 785 nm. The emitted fluorescence (approximate wavelength band of 800-850 nm) can be captured (imaged) and processed. Indocyanine green (ICG) is a sterile, water-soluble but relatively hydrophobic tricarbocyanine molecule. Following intravenous injection, ICG is rapidly bound to plasma proteins with minimal leakage into the interstitium and is excreted by the liver in bile about 8 min after injection. This emission intensity signal can then be used to accurately classify cancerous tissue, through the use of biophysical modelling and image analysis techniques. Similarly, methylene blue (MB) can be excited from 550-700 nm, with an emission centered around 690 nm.

[0117] Other fluorescent dyes are known. Fluorophore molecules may be either utilized alone, or serve as a fluorescent motif of a functional system. Based on molecular complexity and synthetic methods, fluorophore molecules may be generally classified into four categories: proteins and peptides, small organic compounds, synthetic oligomers and polymers, and multicomponent systems.

[0118] In one example, the first image and / or the second image comprise one or more labels. In one example, the method comprises labelling the first image and / or the second image, for example automatically and / or by one or more experts, for example experienced surgeons. Particularly, labelling the first image and / or the second image by one or more experts provides expert discrimination between the first tissue status and the second tissue status of the organs or a parts. In this way, the machine learning algorithm is trained using expert discrimination, and deployed for surgeons having lesser expertise, for example, and / or to standardise methods of treatment, therapy and / or diagnosis. In other words, the training dataset includes information regarding how an expert surgeon interpreted the visual details along with relevant clinical outcome data related to the decisions made intraoperatively on which the computational method can be grounded. In one example, labelling the first image and / or the second image comprises labelling the first tissue status and / or the second tissue status of the organs or a parts thereof according to respective perfusions thereof, for example based on a level of perfusion such as high perfusion, medium (or good) perfusion or low (or poor) perfusion, such as malperfusion. Malperfusion is defined as the loss of blood supply to a vital organ caused by branch arterial obstruction secondary to the dissection.

[0119] In one example, training the computational method / machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises supervised learning, for example using labels as described above.

[0120] In one example, training the computational method / machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises classifying the first tissue status and / or the second tissue status of the organs or a parts thereof. In this way, the first tissue status and / or the second tissue status of the organs or a parts thereof may be classified according to perfusion thereof, for example as described above.

[0121] In one example, classifying the first tissue status and / or the second tissue status of the organs or a parts thereof comprises classifying the first tissue status and / or the second tissue status of the organs or a parts thereof according to respective perfusion thereof, for example as described above.

[0122] In one example, training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof based on presence or absence of the first contrast agent in the first image and / or the second image. In this way, the machine learning algorithm is trained to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof based on presence or absence of the first contrast agent in the first image and / or the second image.

[0123] In one example, training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof based on presence or absence of the first contrast agent in the first image and / or the second image comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof based on intensities (for example, pixel intensities of the first image and / or the second image) due to the first contrast agent in the first image and / or the second image including dynamic changes in intensity. In this way, the machine learning algorithm is trained to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof based on the intensities (for example, pixel intensities of the first image and / or the second image) due to the first contrast agent in the first image and / or the second image

[0124] In one example, training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof along a line in the first image and / or the second image. In this way, the machine learning algorithm is trained to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof along the line.

[0125] In one example, the line spans (partly or completely) the set of spatial regions, for example the first spatial region and the second spatial region. In one example, the line is predetermined. In one example, the method comprises drawing the line on the first image and / or the second image, for example prior to labelling the first image and / or the second image.

[0126] In one example, training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof along the line in the first image and / or the second image comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof along a series of points, including a first point and a second point, in the first image and / or the second image.

[0127] In one example, the machine learning algorithm comprises and / or is a neural network. In one example, the machine learning algorithm comprises and / or is a K-Nearest Neighbour (KNN) model. Logistic regression, naive Bayes, decision tree, random forest, support vector machines (SVMs) and / or gradient boosting models may also be suitable.

[0128] In one example, the machine learning algorithm comprises and / or is a recurrent neural network (RNN). In one example, the machine learning algorithm comprises and / or is a long short-term memory (LSTM) model. Generally, a LSTM model is an advanced version of RNN architecture, designed to model chronological sequences (i.e. time series) and their long- range dependencies more precisely than conventional RNNs. Surprisingly, the inventors have identified that LSTMs are suitable and may be preferable for the discriminating described herein. In one example, the machine learning algorithm comprises and / or is a bidirectional long short-term memory (bi-LSTM) model. In one example, training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises deep learning, DL, for example using a LSTM or a bi-LSTM model. Convolutional Neural Networks (CNNs), Long Short Term Memory Networks (LSTMs), Recurrent Neural Networks (RNNs), Generative Adversarial Networks (GANs), Radial Basis Function Networks (RBFNs), Multilayer Perceptrons (MLPs), Self Organizing Maps (SOMs), Deep Belief Networks (DBNs), Restricted Boltzmann Machines( RBMs) and / or Autoencoders may also be suitable.

[0129] A fourth aspect provides a method of discriminating between a first tissue status and a second tissue status of an organ or a part thereof, the method implemented by a computer comprising a processor and a memory, the method comprising: receiving sample data comprising a time series of images, including a first image and a second image, of the organ or part thereof, having a set of spatial regions including a first spatial region, having the first tissue status, and a second spatial region, having the second tissue status, during a first time period after a first perfusion of the organ or part thereof with a first contrast agent; and discriminating between the first tissue status and the second tissue status of the organ or a part thereof using a trained machine learning algorithm, for example trained according to the first aspect.

[0130] Ex vivo method for treatment, therapy or diagnostic method:

[0131] A fifth aspect provides an ex vivo method for treatment of an organ by surgery or therapy or an ex vivo therapy or diagnostic method practised on an organ, comprising the method according to the first aspect.

[0132] In one example, the method comprises transecting the organ based on a result of discriminating between the first tissue status and the second tissue status of the organ.

[0133] In one example, the organ comprises and / or is the digestive tract, for example the colorectal and / or internal gastrointestinal tract, or a urinary organ.

[0134] In vivo method for treatment, therapy or diagnostic method:

[0135] A sixth aspect provides a method for treatment of the human or animal body by surgery or a therapy or diagnostic method practised on the human or animal body, comprising the method according to the first aspect.

[0136] In one example, the method comprises transecting the organ based on a result of discriminating between the first tissue status and the second tissue status of the organ.

[0137] In one example, the organ comprises and / or is the digestive tract, for example the colorectal and / or internal gastrointestinal tract, or a urinary organ.

[0138] In one example, the method comprises and / or is a real-time method for treatment of the human or animal body by surgery or a therapy or diagnostic method practised on the human or animal body. It should be understood that the real-time method is compatible with a timescale of the method. In this example, discriminating between the first tissue status and the second tissue status of the organ or a part thereof is provided responsive to receiving the sample data, for example at the same rate. In this way, the discriminating is provided sufficiently guickly to feedback into a process including the method, for example during surgery whereby a surgeon may perform the surgery based on a result of the discriminating. In other words, a surgeon may be guided by the discriminating, for example. In this way, patient outcome is improved.

[0139] A seventh aspect provides a computer comprising a processor and a memory configured to implement a method according to the first aspect and / or the third aspect; a computer program comprising instructions which, when executed by a computer comprising a processor and a memory, cause the computer to perform a method according to the first aspect and / or the third aspect; and / or a non-transient computer-readable storage medium comprising instructions which, when executed by a computer comprising a processor and a memory, cause the computer to perform a method according to the first aspect and / or the third aspect.

[0140] In a related approach of the present techniques, there is provided a computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out any of the methods described herein.

[0141] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0142] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[0143] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. Code components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high- level compiled or interpreted language constructs.

[0144] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein.

[0145] The techniques further provide processor control code to implement the abovedescribed methods, for example on a general purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD- ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system.

[0146] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

[0147] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method.

[0148] Throughout this specification, the term “comprising” or “comprises” means including the component(s) specified but not to the exclusion of the presence of other components. The term “consisting essentially of” or “consists essentially of” means including the components specified but excluding other components except for materials present as impurities, unavoidable materials present as a result of processes used to provide the components, and components added for a purpose other than achieving the technical effect of the invention, such as colourants, and the like.

[0149] The term “consisting of” or “consists of” means including the components specified but excluding other components.

[0150] Whenever appropriate, depending upon the context, the use of the term “comprises” or “comprising” may also be taken to include the meaning “consists essentially of” or “consisting essentially of”, and also may also be taken to include the meaning “consists of” or “consisting of”.

[0151] The optional features set out herein may be used either individually or in combination with each other where appropriate and particularly in the combinations as set out in the accompanying claims. The optional features for each aspect or exemplary embodiment of the invention, as set out herein are also applicable to all other aspects or exemplary embodiments of the invention, where appropriate. In otherwords, the skilled person reading this specification should consider the optional features for each aspect or exemplary embodiment of the invention as interchangeable and combinable between different aspects and exemplary embodiments.

[0152] Brief description of the drawings

[0153] For a better understanding of the invention, and to show how exemplary embodiments of the same may be brought into effect, reference will be made, by way of example only, to the accompanying diagrammatic Figures, in which:

[0154] Figure 1 is a flowchart of example steps to process images of an organ;

[0155] Figures 2A, 2B and 2C show example generated augmented images of organs;

[0156] Figure 3 is a flowchart of example steps to train a machine learning model;

[0157] Figure 4 is a flowchart of example steps to use a trained machine learning model;

[0158] Figure 5 shows partial dependence plots of the weighted KNN model for maximum fluorescence intensity (top left), time to peak (top right), upslope (bottom left) and time to reach 50% of maximum intensity (bottom right). Yellow = Poor; Red = Good; Blue = Expert;

[0159] Figure 6 shows a heatmap of activations of the LSTM model for test sequence 3; and Figure 7 is a block diagram of a system for processing images of an organ.

[0160] Detailed Description of the Drawings

[0161] Broadly speaking, the present techniques generally relate to a method for processing images of an organ (or part thereof) using a trained machine learning, ML, model in order to more efficiently and accurately provide information on different tissue regions of the organ. Advantageously, the ML model generates an augmented image of the organ showing the different tissue regions, such as diseased tissue regions and healthy tissue regions. This may help a surgeon to decide where to perform a surgical intervention in a way that only diseased tissue is excised.

[0162] Figure 1 is a flowchart of example steps of a method to process images of an organ using a trained ML model. The method is a computer-implemented method of processing images of an organ. The method comprises: receiving a sequence of images of an organ or part of an organ, each image in the sequence depicting a set of spatial regions including a first spatial region having a first tissue status and a second spatial region having a second tissue status, wherein the sequence of images depict the organ or the part of the organ over a period of time after perfusion with a contrast agent (step S100).

[0163] The step of receiving a sequence of images (step S100) may comprise receiving at least two images, i.e. a first image and a second image. Preferably, the step of receiving a sequence of images comprises receiving many images. The step of receiving the sequence of images may comprise receiving the sequence of images in real time or near real time after perfusion of the contrast agent. This means that the trained ML model performs the analysis on the images in real time or near real time.

[0164] The sequence of images may be frames of a video. For example, the video may be a 30 frames per second video. Each frame of the video may be analysed by the trained ML model. Alternatively, every n frames of the video may be analysed for more efficient processing, where n is an integer. For example, every fifth frame of the video may be analysed.

[0165] The method comprises: analysing the received sequence of images using a trained machine learning, ML, model, wherein the trained ML model is trained to determine an optimal boundary between the first spatial region and the second spatial region based on how the contrast agent perfuses through the first and second spatial regions over time (step S102).

[0166] In the cases where the images are received in real time or near real time, the step of analysing the received sequence of images (step S102) may comprise analysing the received sequence of images in real time or near real time.

[0167] In one example, analysing the received sequence of images in real time or near real time may comprise: receiving a first set of images of the sequence of images; generating, using the first set of images and the trained ML model, a prediction for the optimal boundary between the first spatial region and the second spatial region; receiving a second set of images of the sequence of images; and updating the generated prediction using the second set of images and the trained ML model. In other words, the analysis may be performed as the images are being received, or in chunks or sets of images for computational efficiency. This means that the ML model generates a changing prediction of the boundary between the two spatial regions, which changes as more images are received and the perfusion process continues. That is, the prediction at the start of the perfusion process may not be very accurate or reliable because it is based on relatively few images and because the differences between the spatial regions may not yet be apparent. However, as time and the perfusion process progresses, the ML model has more images to analyse and compare and the prediction evolves and improves in accuracy. Thus, the ML model effectively updates its prediction as more images are received and processed.

[0168] This process of updating the prediction is preferably repeated whenever more images are received. Thus, the method may further comprise updating the generated prediction when a further set of images of the sequence of images is received. This is shown in Figure 1 by the arrow between step S102 and step S100.

[0169] The step (S102) of analysing the received sequence of images using a trained machine learning, ML, model, may comprise: determining, for each pixel in each image, an intensity value corresponding to an intensity of the contrast agent; comparing the intensity value of each pixel in the sequence of images; and predicting, based on the comparing, an optimal boundary between the first spatial region and the second spatial region.

[0170] The step of comparing the intensity value of each pixel in the sequence of images may comprise comparing a rate of change of the intensity value of each pixel across the sequence of images.

[0171] The method comprises: generating an augmented image of the organ or the part of the organ by adding an overlay to an image, wherein the overlay depicts the determined optimal boundary between the first spatial region and the second spatial region (step S104).

[0172] Generally speaking, there are two points in time when the augmented image may be generated: in real time or near real time, and after the full sequence of images has been received and processed.

[0173] Thus, in one example, generating an augmented image (step S104) may comprise adding an overlay to the sequence of images in real time or near real time. In this case, the generated image may be displayed as soon as it is generated, such that a surgeon is provided with a real time / near real time indication of the ML model’s prediction of the spatial regions. As noted above, since the ML model’s prediction may evolve, the generated image also evolves because the overlay will change. This is shown in Figure 1 by the arrow between step S104 and step S100.

[0174] In another example, generating an augmented image may comprise: adding the overlay to a final image in the sequence of images. That is, the overlay is only added to the final image of the sequence of images, after the full sequence of images has been received and processed by the ML model.

[0175] In an alternative example, generating an augmented image may comprise: receiving a further image of the organ or part of the organ; and adding the overlay to the received further image. That is, the overlay is only added after the full sequence of images has been received and processed by the ML model. Furthermore, the overlay may be added to a new image of the organ (i.e. not an image that was processed by the ML model). Preferably, receiving a further image may comprise receiving a further real-time image captured of the organ or part of the organ, for example, after the perfusion process has completed.

[0176] The step of generating an augmented image by adding an overlay to an image may comprise: adding an overlay comprising any one or more of: a colour over one of the first and second spatial regions; a different colour over each of the first and second spatial regions; a contour line between the first and second spatial regions; a reference sign; alphanumeric characters; a graphical marker; a line or marker indicating the determined optimal boundary; and a colour-coded line or marker indicating the determined optimal boundary. The method may further comprise: displaying the generated augmented image on a display, an augmented reality display and / or on a virtual reality display (not shown in Figure 1).

[0177] Turning briefly to Figure 7, this shows a block diagram of a system 20 for processing images of an organ. The system 20 comprises: at least one processor 102 coupled to memory 104, for: receiving a sequence of images of an organ or part of an organ, each image in the sequence depicting a set of spatial regions including a first spatial region having a first tissue status and a second spatial region having a second tissue status, wherein the sequence of images depict the organ or the part of the organ over a period of time after perfusion with a contrast agent; analysing the received sequence of images using a trained machine learning, ML, model 106, wherein the trained ML model is trained to determine an optimal boundary between the first spatial region and the second spatial region based on how the contrast agent perfuses through the first and second spatial regions over time; generating an augmented image of the organ or the part of the organ by adding an overlay to an image, wherein the overlay depicts the determined optimal boundary between the first spatial region and the second spatial region.

[0178] The system 20 may comprise an apparatus 100, such as a computing device, which comprises the at least one processor, memory, trained ML model, and so on.

[0179] The system 20 may further comprise storage 108 for storing the sequence of images, the generated augmented image, and / or the generated perfusion score. Potentially, the training datasets used to generate the augmented image and perfusion score are noted / stored. This may be useful because it enables surgeons to perform post-operative reviews. It may also be useful to analyse how useful the trained model was for clinical outcomes, and whether the training datasets need improving and the model needs re-training.

[0180] The system 20 may further comprise at least one imaging device 112 for capturing the sequence of images.

[0181] The system 20 may further comprise at least one user interface 110 for receiving, from a user, an instruction to begin receiving the sequence of images. For example, the user interface may be an audio interface that is able to detect a spoken instruction (e.g. “Start analysis”), or may be a camera that is able to detect a gesture-based instruction (e.g. a wave). The instruction is given when the contrast agent has been provided in the organ (or part thereof), so that the rate of perfusion can be determined. This also enables the ML model to be used in a “hands-free” manner, i.e. without requiring the surgeon to step-away from what they are doing to start the processing using the ML model.

[0182] The system 20 may further comprise a display device 114 for displaying the generated augmented image. The display device may be an augmented reality overlay on a surgical display, or may be a head-up display, for example. Figures 2A, 2B and 2C show example generated augmented images of organs. Specifically, each of Figures 2A to 2C comprises a pair of augmented images. The left hand side image in each pair is generated based on a prediction from a machine learning, ML, model, while the right hand side image in each pair is generated based on a prediction from a deep learning, DL, model. The images in each pair show the same organ (or part thereof), but the prediction, and hence the overlay is different.

[0183] As shown in these Figures, in one example, the overlay may be a line or marker 10 which shows different regions of the organ. The predictions of the DL model are better, because the lines / markers 10 show the boundaries between regions more clearly. The lines / markers 10 have multiple colours (i.e. they are colour-coded lines or markers), where the meaning of the colours is provided by key 12. The key shows that the optimal boundary between the two regions is in green (“expert”, i.e. where an expert surgeon would determine the boundary). In one example, the marker 10 shows three possible predictions of the ML model in colour-coded format. Here, green represents where an ICGFA expert would recommend staple placement; One spatial region is where perfusion is predicted as being “poor” (as shown in red) - this may correspond to diseased tissue. This may correspond to a region of bowel where perfusion is poor. Another spatial region is where perfusion is predicted as being “good” (as shown in blue) - this may correspond to healthy tissue. This may correspond to a region of bowel proximal to the recommended staple placement zone (perfused side of the bowel). Thus, the green part of the line / marker 10 should indicate the boundary between the two regions. This is why the predictions of the DL model are better. It will be understood that the specific colours and the shape or nature of the marker 10 are exemplary and non-limiting.

[0184] Figure 3 shows a method according to an exemplary embodiment.

[0185] The method is of training a machine learning algorithm to discriminate between a first tissue status and a second tissue status of organs or a parts thereof, the method implemented by a computer comprising a processor and a memory.

[0186] The method comprises: obtaining a training dataset, wherein the training dataset comprises a set of sample data, including first sample data, for a corresponding set of organs or parts thereof including a first organ or part thereof, wherein the first sample data comprises a first time series of images, including a first image and a second image, of the first organ or part thereof, having a set of spatial regions including a first spatial region, having the first tissue status, and a second spatial region, having the second tissue status, during a first time period after a first perfusion of the first organ or part thereof with a first contrast agent (S201); and training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data (S202).

[0187] Figure 4 shows a method according to an exemplary embodiment. The method is of discriminating between a first tissue status and a second tissue status of an organ or a part thereof, the method implemented by a computer comprising a processor and a memory.

[0188] The method comprises: receiving sample data comprising a time series of images, including a first image and a second image, of the organ or part thereof, having a set of spatial regions including a first spatial region, having the first tissue status, and a second spatial region, having the second tissue status, during a first time period after a first perfusion of the organ or part thereof with a first contrast agent (S301); and discriminating between the first tissue status and the second tissue status of the organ or a part thereof using a trained machine learning algorithm (S302).

[0189] Real-time artificial intelligence provision of expert interpretation in indocyanine green fluorescence angiography during colorectal resections:

[0190] Introduction: Indocyanine Green Fluorescence Angiography (ICGFA) use in colorectal resectional surgery is associated with significantly reduced postoperative anastomotic leak (AL) rates [1], By identifying patients with otherwise unsuspected malperfusion when selecting intestinal level for anastomosis, ICGFA indicates patients at lowest AL risk. However, although with experience ICGFA can be consistently interpreted, learning curve as well as tissue (wherein dye moves by diffusion as well as perfusion) and camera (where the centre of screen is brighter than the periphery) behaviours mean grey areas for interpretation exist between binary indication of perfusion / non-perfusion. [2-5] Here, the inventors demonstrate artificial intelligence methods that indicate in real-time to any surgical user how an expert user would interpret the ICGFA signalling regarding colorectal transection point selection.

[0191] Methods: Computational explorations and software development was performed via prospective registered study (NCT 04220242, Institutional Review Board approval reference 1 / 378 / 2092). With DECIDE-AI guidelines for reporting [6],

[0192] Patients: Consenting eligible patients undergoing elective colorectal resection within a single university teaching hospital provided clinical outcome and intraoperative ICGFA data.

[0193] Recorded ICGFA videos were obtained after injecting 0.1 mg / kg of indocyanine green dye (Verdye, Diagnostic Green, Germany) intravenously with the colonic segment for intended transection (after major vessel ligation and mesocolic preparation) under direct visualisation whether intracorporeally or extracorporeally using a PINPOINT Endoscopic Fluorescence System (Stryker Corp, Kalamazoo, Ml, USA). Recordings comprised both white light and nearinfrared synchronously displays at least 90 seconds from the start of ICG inflow up to the point of stapler placement.

[0194] Anonymized videos, where an anastomosis was formed and the patient was known not to suffer a post-operative AL, were used for training and validation, while unseen testing was performed prospectively in consecutive patients in theatre in near real-time. ICGFA Analysis: ICGFA analysis was performed using MATLAB R2023b on an 11th Gen Intel(R) Core i7-1185G7 central processing unit laptop, video frames first underwent affine geometric transformation for image registration warping the frames to keep features aligned with the initial frame, resulting in stabilized white light and near-infrared imagery. Next frames were divided by 72 x 96 grids, and five fluorescence intensity and time-related features, namely mean and maximum fluorescence intensity, upslope and times to reach maximum and 50% maximum fluorescence intensity, were computed from each grid region. As part of data augmentation, four lines of unequal length at different positions across the fluorescence-nonfluorescent boundary were sampled from each training and validation video along with two grid regions above and below and a median value was calculated (total height = 25 pixels).

[0195] Experts’ Annotations: The stapler site chosen by the operating surgeon (R.C., who has 15 years of ICGFA experience), was labelled “expert zone” (width = 10 grid regions, 50 pixels). The perfused area to the proximal side of the “expert” region was labelled “good”, while the contralateral non-perfused area was labelled as “poor”.

[0196] Point based approach (machine learning, ML): The MATLAB classification learner toolbox was used for classifying each point on the line as either good, expert, or poor with 10- fold cross-validation applied for training . with weighted K-Nearest Neighbour (KNN) modelling based on its superior validation accuracy (66.3%).

[0197] Sequence approach (deep learning, DL): Considering each line as a sequence, a bidirectional long short-term memory (bi-LSTM) model was trained to make predictions for each point on the line using the adaptive moment estimation (Adam) algorithm as the solver. The bi-LSTM network comprised 200 hidden units, 60 maximum number of epochs, and gradient threshold of 2. Validation patience was employed to stop the training early to avoid overfitting.

[0198] Prediction Overlay and Evaluation Metrics: Predictions were overlaid on the white light image using x with green, red and blue colouring representation of “Expert”, “Good”, and “poor” zones respectively. Accuracy, precision, recall (sensitivity), specificity, F1 score, Jaccard index, and DICE coefficient were calculated for “Expert” class.

[0199] Algorithm testing: Algorithm testing was performed in theatre in real-time with surgeons were blinded to the predictions.

[0200] Results: Training dataset (18 patient videos, 72 sequences) contained 11 anterior resections, 4 right hemicolectomies, and 3 sigmoid colectomies. Validation dataset (4 patient videos, 16 sequences) contained 3 anterior resections and 1 right hemicolectomy. Learning cuvre analysis indicated stable algorithm performance at 18 cases, enabling four random cases to be used for validation (see supplementary data along with detail of prediction plots and DL heatmap activation). Procedures: Three patients underwent colorectal resections: 2 patients underwent anterior resections, while 1 patient underwent a right hemicolectomy. All patients were discharged home well with no post-operative leak.

[0201] Prediction Overlay: The models provided prediction overlays in under 2 minutes (see Figure 1). Overall, the recall (sensitivity) specificity and F1 scores of predicting an expert zone (i.e., where an expert ICG user would transect) by DL and ML methods were 94%, 92% and 0.86 and 34%, 80% and 0.37 w respectively (see Table 1). This means in every case the prediction match the zone of stapler placement.

[0202] Table 1 : Metrics for assessing predicted “expert” class in test sequences (Abbreviations: ML, machine learning; DL, deep learning).

[0203] Accuracy Precision Recall Specificity F1 Jaccard DICE score

[0204] Test 1 ML 0.66 0.36 0.4 0.76 0.38 0.24 0.38

[0205] DL 0.84 0.63 1 0.79 0.77 0.63 0.77

[0206] Test 2 ML 0.72 0.36 0.33 0.83 0.35 0.21 0.35

[0207] DL 0.94 0.85 0.92 0.95 0.88 0.76 0.88

[0208] Test 3 ML 0.61 0.5 0.3 0.81 0.38 0.23 0.38 DL 0.96 1 0.9 1 0.94 0.90 0.94 0.37 0.86

[0209] Discussion: Surgical learning is experiential with learners developing their “surgical eye” by witnessing other, more experienced surgeons perform. Often intraoperatively, a surgeon may consult another to view the same operative scene for advice. In this work, the Al method, grounded in expert ICGFA interpretation in non-leak cases, takes this role and indicates in real-time how the evolving dynamic ICGFA imagery would be interpreted by an experienced user. Previous work has detailed that experienced users’ interpretation of ICGFA correlates highly and that common intensity-timeseries curve milestones underpin such interpretations. [2] Furthermore, the literature now shows that along with a population effect on AL, patients in whom ICGFA correlates with the surgeon’s own view as to transection level sufficiency have the lowest AL.

[0210] Of the two models developed and tested here, the deep learning bi-LSTM model proved more applicable likely because of its inclusion of sequencing in its prediction generation in that its capture features from both well-perfused and not-well-perfused areas of the bowel segment. Neither model learns off the raw video but instead uses previously validated curve parameters extracted from stabilised video footage. This along with the ML model demonstration of some accuracy as well as the DL provision of parameter prediction correlates enables some inference regarding DL functioning. Despite being a relatively small experience, this work demonstrates encouraging discrimination in a way that enables rapid scaling including potential direct migration to the Nvidia Holoscan platform hardware-software Al interface platform that is already deployed in commercial endoscopic Al systems such as the Gl Genius [7, 8], Next steps will involve the inclusion of additional patient videos including ones with malperfusion (which will likely improve model effectiveness) and additional expert interpretations to provide better confidence and generalisability of predictions, again grounded in perfusion-related anastomotic outcomes with again output display reasonable to surgeons needing only straightforward observational clinical trials with predictions blinded to surgeons to demonstrate efficacy.

[0211] With ICG approaching the standard of care, potentially accelerating by the three additional randomised controlled trials [9-11] reporting this year, the focus necessarily needs to fall on useability and effectiveness rather than proof of efficiency. The software method demonstrated here could usefully provide such a “helping hand” regarding ICGFA judgement for new and uncommon users with low cognitive work imposition.

[0212] Supplementary Methods:

[0213] Interpretability techniques: Partial dependence plots were generated to illustrate the relationship between predictor and classification prediction scores. These plots explain how the point-based (machine learning) classifier model makes predictions for the entire testing data set.

[0214] Visualization of activations of the LSTM model (deep learning), a type of interpretability technique, was employed to explain network predictions. A heatmap was created by extracting the activations of the network’s hidden layers corresponding to test sequence 1 , where higher values indicate stronger activation.

[0215] Interpretability: Partial dependence plots are shown in Figure 5. The plotted lines represent the relationship between the feature variable and the predicted score for each output labels.

[0216] The activations heatmap of test sequence 3 is shown in Figure 6. The activations were extracted from each hidden units and shows how strongly each hidden unit activates. The inventors can observe higher activations throughout the sequence in some hidden units most likely due to the bidirectional nature of the network.

[0217] Computational Quantification of Indocyanine Green Fluorescence Angiography for Parathyroid Gland Vitality Prediction in Patients Undergoing Total Thyroidectomy

[0218] Background: Surgeons experienced in the performance of parathyroid (PG) angiography, using near-infrared (NIR) imaging and indocyanine green (ICG), can reliably predict PG vitality in situ during total thyroidectomy that correlates with the patient’s postoperative PG function and inversely to hypocalcaemia risk. To enable inexperienced surgeons to perform similarly, the inventors developed a computational quantification method that automatically and accurately detects parathyroid tissue and gland vascularity from the NIR imagery concordant with expert level surgical judgment.

[0219] Methods: NIR-PG angiography (FLUOBEAM® LX camera, Fluoptics Grenoble) video recordings of the thyroidectomy bed from 62 patients (75 glands) undergoing total thyroidectomy in a high volume endocrine surgery unit were used. Computational analysis (Matlab, Mathworks), included full field-of-view tracking after image stabilisation (by linear translation) and segmentation identification of PG autofluorescence. Resulting timefluorescence intensity (Fl) profiles of glands were extracted and interrogated against well perfused background regions with upslope and maximum PG / background Fl ratios being used to train a simple logistic regression model grounded in the corresponding surgical expert interpretations and patient outcome data that was then validated using unseen testing data.

[0220] In more detail, consenting patients undergoing thyroid or parathyroid surgery at a high- volume endocrine surgery centre without known allergy to ICG or iodine were eligible for inclusion in this study (Ethics Approval University Hospitals of Geneva 2020-01312). Operations were performed according to a standard protocol using magnifying surgical loupes, including anterior cervicotomy, identification of the PGs, identification of the laryngeal nerve, and the use of intraoperative neuromonitoring.

[0221] Videos of ICG fluorescence angiograms were created by injecting 5mg of ICG (Verdye, Diagnostic Green, Germany) intravenously after the thyroid gland had been removed, with the parathyroid gland(s) under direct visualisation with a commercially available NIR imaging system (FLUOBEAM® LX, Fluoptics, Grenoble, France). As described previously, each identified PG was classified based on the degree of ICG fluorescence as judged by the surgeons: ICG score 0 indicating a non-vascularised gland, ICG score 1 suggesting a partially vascularised gland and ICG score 2 indicating that the gland is well vascularised.

[0222] Anonymised videos with known expert interpretation and clinical outcome were used for development and training of the model. Testing was then carried out prospectively on a cohort of videos with the model blinded to expert interpretation and clinical outcome, a portion of which was carried out in theatre in real-time.

[0223] Using MATLAB 2023B on a AMD Ryzen 7 5700U central processing unit laptop, a video frame within 1 second before visible fluorescence inflow was selected. Using the in-built adapative thresholding, PGs were identified and selected. The centroid of the gland was used to locate a disc from which pixel intensity values could be extracted.

[0224] Video frames were then stabilised using the Kanade-Lucas-Tomasi feature tracking algorithm followed by a linear translation. Next, frames were divided into a 24 x 29 grid. Mean pixel intensity values were extracted from each grid and the aforementioned disc across the duration of the fluorescence. Features tracking was carried out for either 1500 frames (videos recorded at 25 frames per second) or until the pre-defined stability limit was exceeded.

[0225] One of the features of the FLUOBEAM® LX system is that it automatically adjusts its sensitivity (via exposure time and gain) to avoid signal saturation within the frame, these adjustments resulted in discontinuities in pixel values. The inventors ascertained that the transformations between intensity values in different settings obey an affine relationship. These models could then be used to produce continuous curves of arbitrary grayscale values modelling the situation where the camera maintained the same gain and exposure settings throughout the entire fluorescence.

[0226] Quadratic regression smoothing was applied to the curves generated from the 24x29 grid and the discs overlying the parathyroid gland(s). Upslope was calculated as the slope of a straight line connecting the points where the curve first reached 30% and 70% of its maximum intensity. From the 20 grids with the highest maximum intensity, an aggregated maximum intensity and upslope value were extracted. The inverse of these values was used to weight the gland-specific features giving ‘relative max’ and ‘relative upslope’ features that could be compared across videos.

[0227] All patients had standard follow-up consisting of measurement of calcium and PTH levels on post-operative day 1 , with systemic oral supplementation (1g calcium and 800 units 25-hydroxyvitamin D twice daily) for any patients with low levels. The Central Clinical Laboratory of the University Hospitals of Geneva performed all blood analyses. Calcium levels were adjusted according to serum albumin (calciumCorrected = (40-albumin (g ) 200) + calciummeasured). The normal range at the authors’ institution was 1 -1-6-8 pmol / l for PTH and 2-20-2-52 mmol / l for calcium.

[0228] Results: 37 patient videos (45 glands, 29 judged well perfused by the expert surgeons) were used to develop and train the method with full separation of feature data achieved with 100% accuracy. The remaining 25 videos were assessed for suitability, with 22 (27 glands, 15 judged well perfused) ultimately included fortesting (three exclusions due to excessive camera movement or incorrect settings). Segmentation-guided PG detection and subsequent time-FI curve extraction with functional prediction proved feasible in all cases with 100% sensitivity and 10% false positive ranking of tissue) within five minutes of PG angiography recording during unseen testing, including real-time in-theatre quantification, blinded to surgeon judgment, in four surgeries. Model PG vascularity prediction was 96.3% accurate versus expert surgeon judgment in this cohort with a sensitivity and specificity of 93.3% and 100% respectively.

[0229] Conclusion: PG perfusion can be quantified with accuracy using simple machine learning computational methods to a level consistent with current clinical best practice. Further technical optimisation and prospective validation to confirm generalisability is ongoing. In summary, the invention provides a method of real-time artificial intelligence transection recommendation. Although a preferred embodiment has been shown and described, it will be appreciated by those skilled in the art that various changes and modifications might be made without departing from the scope of the invention, as defined in the appended claims and as described above.

[0230] Further embodiments of the present techniques are set out in the following numbered clauses:

[0231] Clause 1. A method of training a machine learning algorithm to discriminate between a first tissue status and a second tissue status of organs or a parts thereof, the method implemented by a computer comprising a processor and a memory, the method comprising: obtaining a training dataset, wherein the training dataset comprises a set of sample data, including first sample data, for a corresponding set of organs or parts thereof including a first organ or part thereof, wherein the first sample data comprises a first time series of images, including a first image and a second image, of the first organ or part thereof, having a set of spatial regions including a first spatial region, having the first tissue status, and a second spatial region, having the second tissue status, during a first time period after a first perfusion of the first organ or part thereof with a first contrast agent; and training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data.

[0232] Clause 2. The method according to clause 1 , wherein the first perfusion of the first organ or part thereof with the first contrast agent is a single perfusion of the first organ or part thereof with the first contrast agent.

[0233] Clause 3. The method according to any previous clause, wherein the first contrast agent comprises and / or is a fluorescent dye, for example indocyanine green or methylene blue, and wherein obtaining the first time series of images, including the first image and the second image, of the organ or part thereof comprises fluorescence angiography.

[0234] Clause 4. The method according to any previous clause, wherein the first image and / or the second image comprise one or more labels.

[0235] Clause 5. The method according to any previous clause, wherein training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises classifying the first tissue status and / or the second tissue status of the organs or a parts thereof.

[0236] Clause 6. The method according to any previous clause, wherein training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof based on presence or absence of the first contrast agent in the first image and / or the second image.

[0237] Clause 7. The method according to clause 6, wherein training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof based on presence or absence of the first contrast agent in the first image and / or the second image comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof based on intensities due to the first contrast agent in the first image and / or the second image.

[0238] Clause 8. The method according to any previous clause, wherein training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof along a line in the first image and / or the second image.

[0239] Clause 9. The method according to clause 8, wherein training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof along the line in the first image and / or the second image comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof along a series of points, including a first point and a second point, in the first image and / or the second image

[0240] Clause 10. The method according to any previous clause, wherein training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises deep learning, DL.

[0241] Clause 11. A method of discriminating between a first tissue status and a second tissue status of an organ or a part thereof, the method implemented by a computer comprising a processor and a memory, the method comprising: receiving sample data comprising a time series of images, including a first image and a second image, of the organ or part thereof, having a set of spatial regions including a first spatial region, having the first tissue status, and a second spatial region, having the second tissue status, during a first time period after a first perfusion of the organ or part thereof with a first contrast agent; and discriminating between the first tissue status and the second tissue status of the organ or a part thereof using a trained machine learning algorithm, for example trained according to any of clauses 1 to 10.

[0242] Clause 12. The method according to clause 11 , wherein the first perfusion of the first organ or part thereof with the first contrast agent is a single perfusion of the first organ or part thereof with the first contrast agent. Clause 13. The method according to any previous clause, wherein the first contrast agent comprises and / or is a fluorescent dye, for example indocyanine green or methylene blue, and wherein obtaining the first time series of images, including the first image and the second image, of the organ or part thereof comprises fluorescence angiography.

[0243] Clause 14. The method according to any previous clause, comprising generating a visualization of the organ based on a result of discriminating between the first tissue status and the second tissue status of the organ or a part thereof.

[0244] Clause 15. The method according to clause 14, wherein generating the visualization of the organ based on the result of discriminating between the first tissue status and the second tissue status of the organ or a part thereof comprises distinguishing respective spatial regions included in the set of spatial regions.

[0245] Clause 16. The method according to clause 15, wherein distinguishing the respective spatial regions included in the set of spatial regions comprises visually distinguishing the respective spatial regions included in the set of spatial regions, for example using colour, contour lines, reference signs, such as alphanumeric characters, and / or markers, such as graphics.

[0246] Clause 17. The method according to any of clauses 14 to 16, wherein generating the visualization of the organ based on the result of discriminating between the first tissue status and the second tissue status of the organ or a part thereof comprises indicating respective boundaries between the respective spatial regions included in the set of spatial regions.

[0247] Clause 18. The method according to any of clauses 14 to 17, comprising displaying the generated visualization of the organ during a second time period after the first time period.

[0248] Clause 19. The method according to clause 18, wherein displaying the generated visualization comprises displaying the generated visualization on an augmented reality display and / or on a virtual reality display.

[0249] Clause 20. The method according to any of clauses 11 to 20, wherein the method comprises and / or is a real-time method of discriminating between the first tissue status and the second tissue status of an organ or a part thereof.

[0250] Clause 21 . An ex vivo method for treatment of an organ by surgery or therapy or an ex vivo therapy or diagnostic method practised on an organ, comprising the method according to any of clauses 1 to 20.

[0251] Clause 22. A method for treatment of the human or animal body by surgery or a therapy or diagnostic method practised on the human or animal body, comprising the method according to any of clauses 1 to 20. Clause 23. The method according to any of clauses 21 to 22, comprising transecting the organ based on a result of discriminating between the first spatial region and the second spatial region.

[0252] Clause 24. The method according to any of clauses 22 to 23, wherein the organ comprises and / or is the digestive tract, for example the colon thereof.

[0253] Clause 25. A computer comprising a processor and a memory configured to implement a method according to any of clauses 1 to 20; a computer program comprising instructions which, when executed by a computer comprising a processor and a memory, cause the computer to perform a method according to any of clauses 1 to 20; and / or a non-transient computer-readable storage medium comprising instructions which, when executed by a computer comprising a processor and a memory, cause the computer to perform a method according to any of clauses 1 to 20.

[0254] References

[0255] 1. Safiejko K, Tarkowski R, Kozlowski TP, Koselak M, Jachimiuk M, Tarasik A, Prue M, Smereka J, Szarpak L. Safety and efficacy of indocyanine green in colorectal cancer surgery: a systematic review and meta-analysis of 11 ,047 patients. Cancers. 2022 Feb 18;14(4):1036.

[0256] 2. Joosten JJ, Bloemen PR, van den Elzen RM, Dalli J, Cahill RA, van Berge Henegouwen Ml, Hompes R, de Bruin DM. Investigating and Compensating for Periphery- Center Effect among Commercial Near Infrared Imaging Systems Using an Indocyanine Green Phantom. Applied Sciences. 2023 Feb 4;13(4):2042.

[0257] 3. Dalli J, Shanahan S, Hardy NP, Chand M, Hompes R, Jayne D, Ris F, Spinelli A, Wexner S, Cahill RA. Deconstructing mastery in colorectal fluorescence angiography interpretation. Surgical Endoscopy. 2022 Dec;36(12):8764-73.

[0258] 4. Hardy NP, Dalli J, Khan MF, Andrejevic P, Neary PM, Cahill RA. Inter-user variation in the interpretation of near infrared perfusion imaging using indocyanine green in colorectal surgery. Surgical Endoscopy. 2021 Dec 1 :1-8.

[0259] 5. Dalli J, Hardy N, Mac Aonghusa PG, Epperlein JP, Cantillon-Murphy P, Cahill RA. Challenges in the interpretation of colorectal indocyanine green fluorescence angiography: Video vignette. Colorectal Disease. 2021 Feb 18.

[0260] 6. Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S et al. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat Med 2022;28:924-933.

[0261] 7. Medtronic and NVIDIA Collaborate to Build Al Platform for Medical Devices [Internet], lnvestor.nvidia.com. 2023 [cited 2024 Feb 23], Available from: https: / / investor.nvidia.com / news / press-release-details / 2023 / Medtronic-and-NVIDIA- Collaborate-to-Build-AI-Platform-for-Medical-Devices / 8. Medtronic. Gl Genius™ Intelligent Endoscopy Module [Internet], Covidien. [cited 2024 Feb 23], Available from: https: / / www.medtronic.com / covidien / en- gb / products / gastrointestinal-artificial-intelligence / gi-genius-intelligent-endoscopy.html#

[0262] 9. Meijer RPJ, Faber RA, Bijlstra OD, Braak J, Meershoek-Klein Kranenbarg E, Putter H, et al. AVOID; a phase III, randomised controlled trial using indocyanine green for the prevention of anastomotic leakage in colorectal surgery. BMJ Open. 2022;12(4):e051144.

[0263] 10. Armstrong G, Croft J, Corrigan N, Brown JM, Goh V, Quirke P, et al. IntAct: intraoperative fluorescence angiography to prevent anastomotic leak in rectal cancer surgery: a randomized controlled trial. Colorectal Dis. 2018;20(8):O226-o34.

[0264] 11. Kossi J. Indocyanine Green Fluorescence Imaging in Prevention of Colorectal Anastomotic Leakage (ICG-COLORAL) NCT03602677 [Available from: https: / / clinicaltrials.gov / study / NCT03602677],

[0265] At least some of the example embodiments described herein may be constructed, partially or wholly, using dedicated special-purpose hardware. Terms such as ‘component’, ‘module’ or ‘unit’ used herein may include, but are not limited to, a hardware device, such as circuitry in the form of discrete or integrated components, a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), which performs certain tasks or provides the associated functionality. In some embodiments, the described elements may be configured to reside on a tangible, persistent, addressable storage medium and may be configured to execute on one or more processors. These functional elements may in some embodiments include, by way of example, components, such as software components, object- oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Although the example embodiments have been described with reference to the components, modules and units discussed herein, such functional elements may be combined into fewer elements or separated into additional elements. Various combinations of optional features have been described herein, and it will be appreciated that described features may be combined in any suitable combination. In particular, the features of any one example embodiment may be combined with features of any other embodiment, as appropriate, except where such combinations are mutually exclusive. Throughout this specification, the term “comprising” or “comprises” means including the component(s) specified but not to the exclusion of the presence of others.

[0266] Attention is directed to all papers and documents which are filed concurrently with or previous to this specification in connection with this application and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference.

[0267] All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and / or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and / or steps are mutually exclusive.

[0268] Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.

[0269] The invention is not restricted to the details of the foregoing embodiment(s). The invention extends to any novel one, or any novel combination, of the features disclosed in this specification (including any accompanying claims, abstract and drawings), or to any novel one, or any novel combination, of the steps of any method or process so disclosed.

[0270] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will recognise that present techniques have a broad range of applications, and that the embodiments may take a wide range of modifications without departing from any inventive concept as defined in the appended claims.

Claims

CLAIMS1. A computer-implemented method of processing images of an organ, the method comprising: receiving a sequence of images of an organ or part of an organ, each image in the sequence depicting a set of spatial regions including a first spatial region having a first tissue status and a second spatial region having a second tissue status, wherein the sequence of images depict the organ or the part of the organ over a period of time after perfusion with a contrast agent; analysing the received sequence of images using a trained machine learning, ML, model, wherein the trained ML model is trained to determine an optimal boundary between the first spatial region and the second spatial region based on how the contrast agent perfuses through the first and second spatial regions over time; generating an augmented image of the organ or the part of the organ by adding an overlay to an image, wherein the overlay depicts the determined optimal boundary between the first spatial region and the second spatial region.

2. The method as claimed in claim 1 further comprising: generating, using the trained ML model, a perfusion score that indicates how well perfusion has occurred in the first spatial region and the second spatial region; and outputting the generated perfusion score.

3. The method as claimed in claim 2 wherein generating a perfusion score icomprises: generating a perfusion score after analysing all images in the sequence of images.

4. The method as claimed in claim 2 or 3 wherein outputting the generated perfusion score comprises: adding the generated perfusion score to the augmented image.

5. The method as claimed in any preceding claim further comprising: inputting, into the trained ML model, sensitivity information to adjust the analysis performed by the trained ML model.

6. The method as claimed in claim 5 wherein inputting sensitivity information comprises inputting information patient-specific information, wherein the patient-specific information includes any one or more of: age; health status; ethnicity; gender; and disease status.

7. The method as claimed in claim 5 or 6 wherein the analysing comprises analysing the received sequence of images using a trained machine learning, ML, model and the sensitivity information.

8. The method as claimed in any preceding claim wherein receiving the sequence of images comprises receiving the sequence of images in real time or near real time after perfusion of the contrast agent.

9. The method as claimed in any preceding claim wherein receiving the sequence of images comprises receiving a video.

10. The method as claimed in claim 8 or 9 wherein analysing the received sequence of images comprises analysing the received sequence of images in real time or near real time.

11. The method as claimed in claim 10 wherein analysing the received sequence of images in real time or near real time comprises: receiving a first set of images of the sequence of images; generating, using the first set of images and the trained ML model, a prediction for the optimal boundary between the first spatial region and the second spatial region; receiving a second set of images of the sequence of images; and updating the generated prediction using the second set of images and the trained ML model.

12. The method as claimed in claim 11 further comprising updating the generated prediction when a further set of images of the sequence of images is received.

13. The method as claimed in claim 11 or 12 wherein generating an augmented image comprises adding an overlay to the sequence of images in real time or near real time.

14. The method as claimed in any one of claims 1 to 13 wherein generating an augmented image comprises: adding the overlay to a final image in the sequence of images.

15. The method as claimed in any one of claims 1 to 13 wherein generating an augmented image comprises: receiving a further image of the organ or part of the organ; and adding the overlay to the received further image.

16. The method as claimed in claim 15 wherein receiving a further image comprises receiving a real-time image captured of the organ or part of the organ.

17. The method as claimed in any preceding claim wherein generating an augmented image by adding an overlay to an image comprises: adding an overlay comprising any one or more of: a colour over one of the first and second spatial regions; a different colour over each of the first and second spatial regions; a contour line between the first and second spatial regions; a reference sign; alphanumeric characters; a graphical marker; a line or marker indicating the determined optimal boundary; and a colour-coded line or marker indicating the determined optimal boundary.

18. The method as claimed in any preceding claim further comprising: displaying the generated augmented image on a display, an augmented reality display and / or on a virtual reality display.

19. The method as claimed in any preceding claim wherein analysing the received sequence of images using a trained machine learning, ML, model, comprises: determining, for each pixel in each image, an intensity value corresponding to an intensity of the contrast agent; comparing the intensity value of each pixel in the sequence of images; and predicting, based on the comparing, an optimal boundary between the first spatial region and the second spatial region.

20. The method as claimed in claim 19 wherein comparing the intensity value of each pixel in the sequence of images comprises comparing a rate of change of the intensity value of each pixel across the sequence of images.21 . A system for processing images of an organ, the system comprising: at least one processor coupled to memory, for: receiving a sequence of images of an organ or part of an organ, each image in the sequence depicting a set of spatial regions including a first spatial region having a first tissue status and a second spatial region having a second tissue status, wherein the sequence of images depict the organ or the part of the organ over a period of time after perfusion with a contrast agent; analysing the received sequence of images using a trained machine learning, ML, model, wherein the trained ML model is trained to determine an optimal boundarybetween the first spatial region and the second spatial region based on how the contrast agent perfuses through the first and second spatial regions over time; generating an augmented image of the organ or the part of the organ by adding an overlay to an image, wherein the overlay depicts the determined optimal boundary between the first spatial region and the second spatial region.

22. The system as claimed in claim 21 , wherein the at least one processor is further configured to: generate, using the trained ML model, a perfusion score that indicates how well perfusion has occurred in the first spatial region and the second spatial region; and output the generated perfusion score.

23. The system as claimed in claim 21 or 22 further comprising storage for storing the sequence of images, the generated augmented image, and / or the generated perfusion score.

24. The system as claimed in claim 21 , 22 or 23 further comprising at least one imaging device for capturing the sequence of images.

25. The system as claimed in any of claims 21 to 24 further comprising at least one user interface for receiving, from a user, an instruction to begin receiving the sequence of images.

26. The system as claimed in claim 25 wherein the user interface is able to detect spoken instructions and / or gesture-based instructions.

27. The system as claimed in any of claims 21 to 26 further comprising a display device for displaying the generated augmented image.

28. A computer-implemented method for training a machine learning, ML, model to process images of an organ, the method comprising: obtaining a training dataset, wherein the training dataset comprises a set of sample data, including first sample data, for a corresponding set of organs or parts thereof including a first organ or part thereof, wherein the first sample data comprises a first time series of images, including a first image and a second image, of the first organ or part thereof, having a set of spatial regions including a first spatial region, having the first tissue status, and a second spatial region, having the second tissue status, during a first time period after a first perfusion of the first organ or part thereof with a first contrast agent; andtraining the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data.

29. The method as claimed in claim 28 wherein obtaining a training dataset comprises: obtaining a first time series of images that is annotated with at least one annotation by an expert.

30. The method as claimed in claim 28 or 29 wherein obtaining a training dataset comprises: obtaining a plurality of the first time series of images, each time series of images being annotated with at least one annotation by a different expert; and aggregating the annotations for the first time series of images to generate a single annotated first time series of images for use during the training.31 . The method according to claim 28, 29 or 30 wherein the first perfusion of the first organ or part thereof with the first contrast agent is a single perfusion of the first organ or part thereof with the first contrast agent.

32. The method according to any of claims 28 to 31 , wherein the first contrast agent comprises and / or is a fluorescent dye, for example indocyanine green or methylene blue, and wherein obtaining the first time series of images, including the first image and the second image, of the organ or part thereof comprises fluorescence angiography.

33. The method according to any of claims 28 to 32, wherein the first image and / or the second image comprise one or more labels.

34. The method according to any of claims 28 to 33, wherein training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises classifying the first tissue status and / or the second tissue status of the organs or a parts thereof.

35. The method according to any of claims 28 to 34, wherein training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof based on presence or absence of the first contrast agent in the first image and / or the second image.

36. The method according to claim 35, wherein training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof based on presence or absence of the first contrast agent in the first image and / or the second image comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof based on intensities due to the first contrast agent in the first image and / or the second image.

37. The method according to any of claims 28 to 36, wherein training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof along a line in the first image and / or the second image.

38. The method according to claim 37, wherein training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof along the line in the first image and / or the second image comprises training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof along a series of points, including a first point and a second point, in the first image and / or the second image39. The method according to any of claims 28 to 38, wherein training the machine learning algorithm to discriminate between the first tissue status and the second tissue status of the organs or a parts thereof using the set of sample data comprises deep learning, DL.

40. An ex vivo method for treatment of an organ by surgery or therapy or an ex vivo therapy or diagnostic method practised on an organ, comprising the method according to any of claims 1 to 39.41 . A method for treatment of the human or animal body by surgery or a therapy or diagnostic method practised on the human or animal body, comprising the method according to any of claims 1 to 39.

42. The method according to claim 40 or 41 , comprising transecting the organ based on a result of discriminating between the first spatial region and the second spatial region.

43. The method according to claim 40, 41 or 42, wherein the organ comprises and / or is the digestive tract, for example the colon thereof.

44. A computer-implemented method of processing images of an organ, the method comprising: receiving a sequence of images of an organ or part of an organ, each image in the sequence depicting a set of spatial regions including a first spatial region having a first tissue status and a second spatial region having a second tissue status, wherein the sequence of images are obtained via a hyperspectral imaging process; analysing the received sequence of images using a trained machine learning, ML, model, wherein the trained ML model is trained to determine an optimal boundary between the first spatial region and the second spatial region based on changes in hyperspectral image data from the first and second spatial regions over time; and generating an augmented image of the organ or the part of the organ by adding an overlay to an image, wherein the overlay depicts the determined optimal boundary between the first spatial region and the second spatial region.

Images