Neural network system for endoscopic scoring of gastrointestinal disease

Abstract

The specification describes systems, methods, and apparatus for using machine-learning techniques to assess the severity of gastrointestinal diseases such as ulcerative colitis. According to a first aspect of this specification, there is described a computer implemented method for assessing the severity of a gastrointestinal disease, the method comprising: receiving a video of an endoscopy performed on a portion of a gastrointestinal tract of a subject, the video comprising a plurality of frames; determining, based on the plurality of frames, a local z-axis motion of a camera used to generate the endoscopy video; based on the local z-axis motion, processing, using one or more severity classification models, the plurality of frames to generate a plurality of severity scores, each severity score indicative of a severity of the gastrointestinal disease at a different location within the gastrointestinal tract; and generating, based on the plurality of severity scores, an overall severity score.

Description

NEURAL NETWORK SYSTEM FOR ENDOSCOPIC SCORING OFGASTROINTESTINAL DISEASECROSS-REFERENCE TO RELATED APPLICATIONS[00011 The present application claims priority to U.S. Provisional Application No. 63 / 671,608, filed on July 15. 2024, the disclosure of which is incorporated herein by reference.FIELD

[0002] The specification describes systems, methods, and apparatus for using machinelearning techniques to assess the severity of gastrointestinal diseases such as ulcerative colitis.BACKGROUND

[0003] Gastrointestinal diseases such as ulcerative colitis may be characterized by inflammation and / or chronic lesions, and can significantly impact the quality of life of a subject. Traditional assessment methods for gastrointestinal diseases such as ulcerative colitis rely heavily on visual inspections and subjective judgments by highly-qualified medical practitioners, for example by performing an endoscopic assessment of the colon of the subject (i.e. , a colonoscopy). Disease severity scores, such as a Mayo score in the case of ulcerative colitis, can provide quantitative tools that provide a systematic and reproducible way to evaluate the severity and extent of a gastrointestinal disease. However, manual severity' scoring can be time-consuming and prone to variability.SUMMARY

[11004] According to a first aspect of this specification, there is described a computer implemented method for assessing the severity of a gastrointestinal disease, the method comprising: receiving a video of an endoscopy performed on a portion of a gastrointestinal tract of a subject, the video comprising a plurality of frames; determining, based on the plurality of frames, a local z-axis motion of a camera used to generate the endoscopy video; based on the local z-axis motion, processing, using one or more severity classificationmodels, the plurality of frames to generate a plurality of severity scores, each severity score indicative of a severity of the gastrointestinal disease at a different location within the gastrointestinal tract; and generating, based on the plurality of severity scores, an overall severity score that is indicative of an overall severity of the gastrointestinal disease for the portion of the gastrointestinal tract.

[0005] Determining the local z-axis motion of the camera may comprise processing the plurality of frames using a motion model to output the local z-axis motion.

[0006] Processing the plurality' of frames using the motion model may comprise processing the plurality of frames using the motion model to determine a speed of the camera along the z-axis at each frame.[0007| Processing the plurality of frames using the motion model may comprise processing the plurality of frames using a Structure-from-Motion, SfM, Learner algorithm. Processing the plurality of frames using the motion model may comprise processing the plurality of frames using an Appearance Flow Structure-from-Motion, AF-SfM, Learner algorithm.

[0008] The local z-axis motion of the camera may be determined without determining a local x-axis motion or a local y-axis motion of the camera.

[0009] The method may further comprise mapping each frame of the plurality of frames to a respective relative location in the portion of the gastrointestinal tract based on the determined local z-axis motion.

[0010] The method may further comprise grouping the plurality of frames into a plurality of sets of frames based on the determined local z-axis motion such that each set of frames corresponds to a substantially identical distance moved by the camera along the local z-axis.

[0011] Processing the plurality of frames to generate a plurality of severity scores may comprise processing, using the one or more severity' classification models, each set of frames to generate, for each set of frames, a corresponding severity of the plurality of severity scores.

[0012] The plurality of frames may be a plurality' of filtered frames that have been determined based on processing the video using a frame filtering network comprising one or more vision transformer models, optionally, wherein the frame filtering network has been trained using active learning.

[0013] The one or more severity classification models may comprise a pre-trained classification model that has been fine-tuned on a training dataset comprising a plurality offrames from at least one endoscopic video and respective ground truth severity scores indicative of a severity of a gastrointestinal disease in each frame.[0014| The one or more severity classification models may utilise an Expandable 3D, X3D, architecture.|0015] The one or more severity classification models may comprise one or more vision transformer models.

[0016] The overall severity score may be a cumulative severity score, wherein generating the overall severity score comprises summing the plurality of severity scores to generate the cumulative severity score.[00171 According to a further aspect of this specification, there is described a system comprising: one or more processors; and a memory7, the memory7storing computer readable instructions that, when executed by the one or more processors, causes the system to perform operations comprising any one or more of the methods described herein.

[0018] According to a further aspect of this specification, there is described a computer program product comprising computer readable instructions that, when executed by data processing apparatus, cause the data processing apparatus to perform operations comprising any one or more of the methods described herein.[0019| The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.[0020| The described systems and methods provide automated, objective assessment of gastrointestinal disease severity from endoscopic video data. By leveraging machine learning models trained on annotated clinical data and applying active learning strategies, the system can significantly reduce the subjectivity and labor involved in traditional clinician-based video review and scoring. The described technology addresses limitations of manual scoring systems, which are prone to inter-observer variability and inconsistent interpretations. The ability to automatically process large volumes of video and localize disease severity along the gastrointestinal tract provides a more granular and reproducible scoring system, ultimately improving diagnostic accuracy and patient outcomes.

[0021] Furthermore, by implementing video-based frame filtering and active learning loops, the system selectively prioritizes high-value clips for annotation and analysis, thusreducing noise in the dataset and improving training data quality. This leads to more accurate classification of disease severity and better generalization of the trained models.

[0022] In some implementations, the system utilizes transformer-based architectures, such as vision transformers (ViT), or spatiotemporal convolutional networks, such as X3D, for video classification. These architectures demonstrate improved performance compared to traditional frame-based convolutional models due to their ability to model temporal context and spatial relationships across frames. This improvement enhances computer systems by improving computational efficiency.[0023 j For example, by classifying entire clips rather than individual frames, the system can reduce redundant computation across temporally adjacent frames, enabling more efficient inference and reducing the overall number of model evaluations required. Additionally, active learning reduces the volume of labeled data needed to train high-performing models, thereby decreasing annotation effort and associated computational overhead during training. The described motion estimation techniques further allow the system to focus processing on distinct segments of the gastrointestinal tract, avoiding unnecessary duplication of analysis across overlapping regions. These efficiencies result in lower memory usage, faster processing times, and scalability to large datasets in clinical or research settings.BRIEF DESCRIPTION OF THE DRAWINGS[0024| Example implementations will be described by way of reference to the accompanying drawings, in which:

[0025] FIG. 1 shows an overview of an example system / method for assessing the severity of a gastrointestinal disease using machine-learning;

[0026] FIG. 2 is a schematic showing severity scores and cumulative severity scores generated for the colons of two different subjects;

[0027] FfG. 3 shows an overview of an example system / method for assessing the severity' of a gastrointestinal disease using machine-learning;[00281 FIG. 4 is a graph showing an endoscope camera’s accumulated distance on[0029[ local coordinate systems x, y, and z over time, when withdrawn from a colon;

[0030] FIG. 5 is a graph illustrating the mapping of endoscopic video frames to respective relative locations in a portion of a gastrointestinal tract;[00311 FIG. 6 A shows an example model structure for a machine-learning frame filtering network and / or a machine-learning severity classification model;

[0032] FIG. 6B shows an example model structure for a machine-learning frame filtering network and / or a machine-learning severity classification model;1 033j FIG. 7A shows an example of a method for fine-tuning a frame filtering network for classifying frames as informative or non-informative;

[0034] FIG. 7B shows an example of a method for fine-tuning a severity classification model for generating a severity score;[0035 [ FIG. 8 shows a flow diagram of an example method for assessing the severity of a gastrointestinal disease using machine-learning;[0036[ FIG. 9 shows a flow diagram of an example method for assessing the severity of a gastrointestinal disease using machine-learning; and|0037] FIG. 10 shows a schematic overview of a computing system / apparatus.DETAILED DESCRIPTION

[0038] This specification describes a neural network system designed to predict severity scores for gastrointestinal diseases, such as a Mayo score for ulcerative colitis, a crucial metric for assessing the extent and severity of ulcerative colitis. The systems and methods described herein may allow for accurate disease severity prediction and localization, providing a robust tool for gastrointestinal assessment and treatment planning.

[0039] The systems, methods and apparatus described herein provide improvements in automated gastrointestinal diagnostics. By automating the prediction and localization of ulcerative colitis or other gastrointestinal diseases using neural networks, the systems, methods, and apparatus described herein can significantly reduce the subjectivity and labour involved in traditional methods of diagnosing and / or assessing gastrointestinal diseases. The ability to provide rapid and accurate assessments supports timely and tailored treatment decisions, ultimately improving patient outcomes. Furthermore, the scalable and adaptable nature of the neural network models described herein allows for continuous improvement and integration into diverse clinical environments, making it an invaluable tool for healthcare providers worldwide. Primary care providers can utilize the systems methods, and apparatus to obtain accurate gastrointestinal disease assessments, facilitating early intervention andtimely referrals for patients who require further evaluation or treatment by gastrointestinal specialists.|0040 j FIG. 1 shows an overview of an example method 100 for assessing the severity of a gastrointestinal disease using machine-learning. The method may be performed by one or more computing devices / systems, such as the device / system described in relation to FIG. 10.

[0041] In an optional pre-processing stage, a plurality of frames 104 of an endoscopy video 102 are input into a frame filtering network 106. The video 102 is of an endoscopy of a portion of a gastrointestinal tract of a subj ect, showing symptoms of a gastrointestinal disease. The frame filtering network 106 processes the plurality of frames 104 to classify each frame as informative or non-informative for determining a severity score for the gastrointestinal disease. A plurality of filtered frames 108 can be determined based on the classification of the frames 104 as informative or non-informative, wherein the filtered frames 108 comprise the frames from the plurality of frames 104 that have classified as informative by the frame filtering network 106. The set of filtered frames 108 does not include frames from the plurality of frames 104 that have been classified as non-informative by the frame filtering netw ork 106. The frame filtering network 106 may process the filtered frames 108 to generate a class for each filtered frame 108, the class indicative of whether the filtered frame 108 is informative or non-informative. As an example, the class may be numerical score e.g., w here a score of one indicates that the corresponding filtered frame 108 is informative and a score of zero indicates that the corresponding filtered frame 108 is non- informative. Alternatively, the class may be a categorical score from a set of categories, e.g., the set {“informative”, “non-informative”} or the like.[0042} After the pre-processing stage, and in a severity scoring stage, the plurality of informative filtered frames 108 are input into one or more severity classification models 108 that process the plurality of filtered frames 108 to generate a plurality of severity scores 112. Each severity score 112 is indicative of a severity of the gastrointestinal disease that may be present in the corresponding filtered frame 108. In examples where the optional preprocessing stage is not performed, the entire plurality of frames of the video 102 (i.e., without filtering to the plurality of frames 104 to remove non-informative frames) may be input into the one or more severity classification models 110 to generate the plurality of severity scores 112.[00431 The gastrointestinal disease may be any gastrointestinal disease with symptoms that visually present on a portion of the gastrointestinal tract, such that the symptoms can be visually captured by an endoscopic video 102. For example, one or more symptoms may be in the form of lesions, rashes, spots, blisters and / or the like. In some examples, the portion of the gastrointestinal tract may, for example, comprise at least a portion of the colon of a subject. Where the portion of the gastrointestinal tract comprises as at least a portion of the colon of a subject, examples of a gastrointestinal disease to assess for severity include, but are not limited to, ulcerative colitis and / or Crohn’s disease. In the following description, ulcerative colitis will be used as an example gastrointestinal disease, with the video 102 being of an endoscopy performed in at least a portion of the colon of a subject (i.e. , a colonoscopy). However, it will be appreciated that the systems and method described herein are applicable to other gastrointestinal diseases and / or gastrointestinal tract portions.

[0044] The plurality of frames 104 of the video 102 may each comprise a colour image (e.g., RGB image) of a portion of the gastrointestinal tract of the subject. The frames 104 may have a predefined size, e.g., a predefined height (H) and width (W) in pixels. Such an image frame may be represented / stored / processed as aH x W x C dimensional array of pixel values, where C is the number of colour channels, e.g., 3 in the RGB case. The video 102 and frames 104 have been captured by a user (e.g., a healthcare worker such as a clinical endoscopist, or some other third party) using an image capturing device, e.g., a camera of an endoscope.

[0045] In some examples, the frames 104 may comprise stereoscopic images, i.e., images captured using a stereoscopic camera. Such images may additionally comprise image depth data relating to the depth of features in the image, e.g., the image may be in an RGB-D format.[0046| A raw (i.e., unfiltered) endoscopic video 102 can include a lot of visual noise, which may include flushing water, reflections, tools, stool, out-of-focus frames, and / or out-of- subject frames. Such noise can obscure the view of inflamed areas of the gastrointestinal tract and / or contribute irrelevant information, which may lead to an inaccurate assessment of disease severity'. Furthermore, a substantial proportion of the frames 104 in a raw endoscopic video 102 may show healthy (i.e., non-diseased) tissue. Frames that include substantial visual noise and / or that show only non-diseased tissue may be classified as ‘non-informative’, in that they are not informative (i.e., not helpful) for determining a severity score of a gastrointestinal disease. Frames that do not include substantial visual noise and / or that doshow diseased tissue may be classified as ‘informative’, in that they are informative (i.e.. helpful) for determining a severity score of a gastrointestinal disease.[00471 Providing raw, unfiltered endoscopy videos 102 to the one or more classification models 110 that include all raw frames (i.e.. without pre-processing the frames to filter (i.e., remove) non-informative frames) may result in a substantial imbalance in the one or more classification models 110, which imbalance might undermine the model’s performance. The pre-processing stage aims to address one or more of these issues associated with raw endoscopic videos. The pre-processing stage, including the use of a frame filtering network 106, may enhance the quality of raw' endoscopic videos by removing non-informative frames that are of low or no use in generating a severity score. Such pre-processing may significantly increase the classification accuracy of the one or more classification models 110.|0048] The frame filtering netw ork 106 comprises a neural netw ork model that has been trained to classify frames (i.e., frames 104 and / or frames 108) of a video 102 as informative or non-informative for determining a severity of a gastrointestinal disease, for example using the training method described in relation to FIG. 7 A. The frame filtering network 106 may comprise one or more (e.g., a plurality) of neural network layers. The layers of the frame filtering netw ork 106 may comprise one or more convolutional layers, e.g., be a convolutional neural network. The frame filtering network 106 may comprise one or more skip connections (e.g., residual connections) between layers of the frame filtering network 106. Alternatively or additionally, the layers of the frame filtering netw ork 106 may comprise one or more transformer blocks. Alternatively or additionally, the layers of the frame filtering network 106 may comprise one or more fully connected layers, such as multi-layer perceptrons. Some examples of structures of the frame filtering network 106 are provided in Fig. 6A (an Expandable 3D architecture) and Fig. 6B (a Vision Transformer).[0049| The frame filtering network 106 may comprise one or more convolutional neural networks (CNNs) (i.e., neural networks with one or more convolutional layers), e g., be a CNN-based classifier. CNNs, with their capacity for hierarchical feature extraction from raw images, are widely used for image classification tasks. Examples of neural network models that may be used for the frame filtering network 106 include, but are not limited to, ResNet, DenseNet, EflficientNet, and / or a Visual Geometry Group (VGG) image classification (VIC) model.[0050| In some examples, the frame filtering network 106 comprises an Expandable 3D (X3D) architecture, for example as later described in relation to Fig. 6A.IOO5F| The frame filtering network 106 may alternatively or additionally comprise one or more transformer models, i.e.. models comprising one or more (e.g., a plurality of) transformer layers. For example, the one or more frame filtering networks may comprise a Vision Transformer (ViT) model, for example as later described in relation to Fig. 6B. ViT models represent images as sequences of patches, using self-attention mechanisms to effectively capture global dependencies. Transformer (e.g., ViT) models have an ability to capture global context and subtle patterns, which make them highly suitable for the presently disclosed methods.

[0052] The frame filtering network 106 has parameters that have been trained using machine-learning techniques on a set of training data comprising a plurality of frames from endoscopic videos of gastrointestinal tracts that have a gastrointestinal disease, for example as later described in relation to Fig. 7A and / or Fig. 7B.

[0053] The frame filtering network 106 processes the frames 104 based on a set of learned parameters of the frame filtering network 106 to classify each frame as either informative or non-informative (for determining a severity score). In some examples, the frame filtering network 106 may process each of the frames 104 on a frame-by-frame basis to perform the classification. In other examples, the frame filtering network 106 may process a plurality of the frames 104 together to perform the classification (i.e.. may perfonn video-based classification rather than frame-based classification).

[0054] Based on the classification, a set of filtered frames 108 can be determined, wherein the set of filtered frames 108 comprises the frames classified as informative by the frame filtering network 106, with the frames classified as non-informative by the frame filtering network 106 removed (i.e., not present in the set of filtered frames 108).[0055| The informative filtered frames 108 derived using the frame filtering network 106 are input into one or more (e.g., a plurality ol) severity classification models 108. The filtered frames 108 are, in some examples, scaled to a predefined size before being input into the classification model 110 using, e.g., image upsampling (e.g., superresolution) and / or image dow nsampling techniques. Alternatively, the filtered frames 108 are divided into patches of a predetermined size, and the patches input into the one or more classification models 110.[0056| The one or more classification models 110 process the filtered frames 108 based on a set of learned parameters of the classification model 110 to generate a plurality of severity scores 112 that are each indicative of a severity of the gastrointestinal disease. In some examples, the one or more classification models 110 generate a corresponding severity score for each filtered frame 108 processed by the classification model 110, wherein the severity score is indicative of a severity of the gastrointestinal disease in that filtered frame 108. In other examples, the one or more classification models 110 process a subset (e.g., plurality) of the filtered frames 108 to generate a single corresponding severity score for that subset of frames.(0057 The one or more classification models 110 may comprise one or more neural network models. The neural network models may comprise one or more (e.g., a plurality of) neural network layers. For example, the one or more classification models 110 may comprise one or more convolutional neural networks (i.e., neural networks with one or more convolutional layers), e g., be a multi-model CNN-based classifier. One or more of the neural network models may comprise one or more skip / residual connections. Examples of classification models 108 that may be used include, but are not limited to, ResNet and / or a Visual Geometry Group (VGG) image classification (VIC) model.(0058| The one or more classification models 110 may alternatively or additionally comprise one or more transformer models, i.e., models comprising one or more (e.g., a plurality of) transformer layers. For example, the one or more classification models 110 may comprise a Vision Transformer (ViT) model, for example as later described in relation to Fig. 6B.

[0059] Alternatively or additionally, the one or more classification models 110 may use an Expandable 3D (X3D) architecture, for example as later described in relation to Fig. 6A.

[0060] Use of an X3D or ViT architecture as one or more of the severity classification models 110 can allow for the capture of both spatial and temporal dynamics when processing frames to generate a severity score. A particular advantage of these models lies in their scalability across different dimensions — such as frame rate and resolution — to meet specific requirements (e.g., accuracy and budget needs), thereby providing a versatile tool for videobased classification in various scenarios. By utilizing model with X3D or ViT architecture, our system’s ability to understand and interpret complex video sequences may be enhanced.This may lead to more accurate assessments of the severity and extent of inflammation and other disease symptoms.100611 The one or more classification models 110 have parameters that have been trained using machine-learning techniques on a set of training data. The set of training data may comprise a plurality of frames from one or more videos of endoscopies of gastrointestinal tracts (e.g., colons) that have a gastrointestinal disease (e.g., ulcerative colitis). In some examples, each frame is associated with a corresponding one or more ground truth severity scores for the gastrointestinal disease. The one or more classification models 108 may be trained using an objective / loss function that compares candidate severity scores output by the one or more classification models 110 to the ground truth severity scores, e.g., an absolute difference-based loss, such as an L2 loss, or a classification loss (e.g., for categorical scores). An optimisation routine, such as stochastic gradient descent, may be applied to the loss function to determine updates to parameters of the model.[0062} The severity scores 112 generated using the one or more classification models 110 may be numerical scores in a respective predefined numerical range, e.g., a score out of five, a score out of ten, a percentage score, or the like. One or more of the severity scores 110 may be a categorical score from a set of categories, e.g., the set {“mild”, “moderate”, “severe”} or the like.100631 In the case of the video being of an endoscopy of a colon and the gastrointestinal disease being ulcerative colitis, the severity scores 112 may be Mayo scores. Mayo scores[0064} range from 0 to 3, where a Mayo score of 0 is indicative of a normal or inactive disease, a Mayo score of 1 is indicative of mild disease (e.g., erythema, decreased vascular[0065} pattern), a Mayo score of 2 is indicative of moderate disease (e.g., marked erythema,[0066} lack of vascular pattern), and a Mayo score of 3 is indicative of severe disease (e.g., spontaneous bleeding, ulceration). A lower Mayo score therefore indicates better mucosal health and is associated with improved overall health outcomes.10067} The severity scores 112 generated for each fdtered frame 108 may be processed (e.g., using an aggregator 114) to generate an overall severity7score 116 for the gastrointestinal disease. The overall severity score may be indicative of an overall (e.g., average) severity of the gastrointestinal disease throughout the portion of the gastrointestinal tract. The overall severity score may be derived from a combination of the individual severityscores 112 output by the one or more classification models 110. e.g., using a set of predefined rules and / or an averaging or smoothing process.IOO68| The overall severity score 116 may be. in some examples, output by the one or more classification models 110 directly.|O069] In examples where the severity scores 112 are Mayo scores, the overall severityscore 116 may be a Mayo score for the entire portion of the gastrointestinal tract (e.g., an average or smoothed Mayo score).

[0070] To improve the classification model’s 110 accuracy, an aggregation method may be used to smooth the predictions, as shown in Algorithm 1 below. Observing that directly applying classification models 110 to a whole video 102 does not always perform well, in some examples an aggregation method called “bubble merge” may be used to reduce variance in the generated severity scores 112 and enhance the accuracy and reliability of our results. First, the original video 102 may be split into a plurality of consecutive video clips, each video clip comprising a subset of the plurality of frames 104, 108. A severity score 112 such as a Mayo score is generated for each clip, using one of the methods disclosed herein. The severity scores 112 of all of the video clips in a fixed time window are merged based on the continuity of the severity- scores or based on majority voting. If the video 102 is longer than 10 seconds, we directly aggregate the severity scores based on the maximum prediction duration with majority voting. If the video 102 is shorter than 10 seconds, which may be the most common case, a sliding window is used to smooth the severity scores. Specifically, if several consecutive video clips with the same generated severity score are substantially shorter (e.g., shorter than 2 seconds) than their adjacent consecutive video clips, those video clips are instead assigned the generated severity score of their neighbouring video clips that have a longer duration. The purpose of this step is to ignore severity scores 112 for video clips that are too short to be informative for generating an overall severity score. Afterward, max pooling may be used to determine an a single, overall severity score 116 (e.g., a single Mayo score) for the entire video 102.Algorithm 1 Aggregation Method for Video ClassificationRequire: Endoscopic video predictions: ytvideo clips q, i = 1 , 2, 3...; Video ID, ID Output: Aggregated Mayo score, ranging from 0 to 3.For each ID:Merge clips q and obtain original videoMerge labels for frames with the same Mayo scores if video duration < 10s thenSliding window approach to classify frames Apply majority voting to determine final prediction elseApply aggregation based on the maximum prediction duration Apply majority voting to determine final prediction end ifCompute cumulative Mayo score by integrating outputs from previous steps.

[0071] One or more of the severity scores 112 for the filtered frames 108 and / or the overall severity score 116 may be output to a user via a display of a user device, e.g., a smartphone, a tablet, a personal computer or the like. The one or more severity scores and / or overall severity score may be output in an application performing at least a part of the method 100. The one or more severity- scores 112 and / or overall severity score 1 16 may be output as an overlay on corresponding frames.|OO721 While the provision of a single, overall Mayo score for the colon of a subject has been a standard assessment metric used in clinical diagnosis of ulcerative colitis, it fails to capture the varied disease severity at different colon locations. For example, Fig. 2 is a schematic representation of a colon of a first subject and a colon of a second subject. The colon of each patient has been divided into ten sections corresponding to ten colon locations along the axis 222, with each section having an associated Mayo score (112a-j for the first subject and 112k-t for the second subject). An overall Mayo score for a subject is often the highest Mayo score determined across the colon of that subject. For the first subject, the highest Mayo score throughout the colon is 2 (from Mayo scores 112i and 112j), therefore the overall Mayo score for the colon of the first subject will be 2. Similarly, for the second subject, the highest Mayo score throughout the colon of the second subject is also 2 (from Mayo scores 112m-q), therefore the overall Mayo score for the colon of the second patient will be also 2.[0073 j While both the first subject and the second subject have the same overall Mayo score of 2, it can be seen from Fig. 2 that the overall Mayo score is not helpful in indicating how the severity of the disease varies across different colon locations. In view of this, aspects of the present disclosure may also provide a Cumulative Disease Score (CDS) that can offer a more detailed quantification of disease severity. The provision of a CDS may enable finerdifferentiation of treatment effects and improve statistical power in clinical trials, for example. The CDS is determined by summing the severity scores generated for each section of the colon. With reference to Fig. 2, the Cumulative Disease Score 216a (which in this case may be a Cumulative Mayo Score) for the first patient is 6 (i.e., the summation of the severity scores 112a-j), while the Cumulative Disease Score 216b (again, which in this case may be a Cumulative Mayo Score) for the second patient is 11 (i.e., the summation of the severity scores 112k-t). The CDSs 216a, 216b better indicate the degree of disease seventy across the colon of the first subject and the colon of the second subject.[0074| FIG. 3 shows an overview of an example system / method 200 for assessing the severity of a gastrointestinal disease using machine-learning that generates a Cumulative Disease Score (CDS) 216. The method may be performed by one or more computing devices / systems, such as the device / system described in relation to FIG. 10. In some examples, one or more aspects of the system / method 200 may be similar or identical to one or more aspects of the system / method 100 described in relation to Fig. 1.[0075| In an optional pre-processing stage, a plurality of frames 104 of a video 102 are input into a frame filtering network 106. The video 102 is of an endoscopy of a portion of a gastrointestinal tract of a subject, showing symptoms of a gastrointestinal disease. The frame filtering network 106 processes the plurality of frames 104 to classify each frame as informative or non-informative for determining a severity score for the gastrointestinal disease. A plurality of filtered frames 108 can be determined, wherein the filtered frames 108 comprise the frames classified as informative by the frame filtering network 106. The set of filtered frames 108 does not include the frames classified as non-informative by the frame filtering network 106. The frame filtering network 106 may be similar or identical to the frame filtering network 106 as previously described in relation to the method 100 of Fig. 1, and / or may process the plurality of frames 104 to determine a set of filtered frames 108 in a similar or identical manner as previously described in relation to the method 100 of Fig. 1, for example.

[0076] Based on the plurality' of filtered frames 108 (or frames 102 if pre-processing does not take place), a local z-axis motion of a camera used to generate the endoscopy video is determined. To determine the local z-axis motion the plurality’ of filtered frames 108 (or else the frames 102 if the pre-processing stage was not performed) are input into a motion model 202. The motion model 202 processes the filtered frames 108 (or frames 102) to determine a local z-axis motion 204 of a camera used to generate the endoscopy video 102.[00771 In some examples, the motion model 202 may comprise a neural network. In some examples, the motion model 202 comprises a Structure-from-Motion, SIM. Learner algorithm, for example an Appearance Flow Structure-from-Motion, AF-SfM, Learner algorithm (described, for example, in “Selfsupervised monocular depth and ego-motion estimation in endoscopy: Appearance flow to the rescue”, S. Shao et. al., arXiv: 2112.08122 (15 December 2021) the contents of which are incorporated herein by reference).[0078J The plurality of frames 108 are processed using the pre-trained model of the AF- SfM Learner algorithm to determine the local z-axis motion of the camera. The local z-axis motion may correspond to forwards or backwards movement of the camera (i.e., as it passes through the gastrointestinal tract).

[0079] The z-axis motion may be a speed of the camera along the local z-axis at each frame 108, wherein the pre-trained model of the AF-SfM Learner algorithm may be used to estimate camera’s speed along its z-axis (i.e., the direction of moving forward and backward). This speed can be used to estimate how fast the camera inserts into or withdraws from the portion of the gastrointestinal tract such as the colon and can be further used to map a frame at a given time to a relative location in the colon. Because the length of colons might differ across subjects, numbers between 0 and 1 to represent the relative location in the colon, with 0 being Ascending colon and 1 being Rectum. The original video may be divided into 10 video clips such that in each clip the camera has travelled for the same distance through the colon.

[11080] The plurality of filtered frames 108 (or else the frames 102) are grouped into a plurality of sets of frames based on the local z-axis motion 204 of the camera that was determined using the motion model 202. The plurality of filtered frames 108 (or else the frames 102) are grouped such that each set of frames corresponds to a substantially identical distance moved by the camera along the local z-axis of the camera (i.e. a substantially equal forwards or backwards movement of the camera through the portion of the gastrointestinal tract).10081] Each set of frames is input into one or more severity classification models 110 to generate a plurality of severity scores 212, wherein each set of frames is processed by the one or more severity classification models 110 to generate a corresponding severity score of the plurality of severity scores 212 that is indicative of a severity of the gastrointestinal disease in that set of frames.[0082| Processing the plurality of filtered frames 108 using the motion model 202 to determine the local z-axis motion of the camera may comprise processing the plurality of filtered frames 108 using the motion model 202 to determine a z-axis speed of the camera at each frame of the filtered frames 108.[11083} Fig. 4 is a graph demonstrating an endoscope camera’s accumulated distance on f 0084] local coordinate systems x, y, and z over time, when withdrawn from a colon. The graph shows ground truth movements 303b, 302b, 301b of the camera along the local x-axis, local y-axis and local z-axis respectively, and predicted movements 303a, 302a, 301a of the camera along the local x-axis, local y-axis and local z-axis respectively, wherein the predicated movements have been predicted by processing video frames generated by the camera using the AF-SfMLeamer algorithm. Frame index of the video is used to represent time on the horizontal axis. The local x-axis represents up and dow n movement of the camera, the local y-axis represents left and right movement of the camera, and the local z- axis represents forward and backwards movement of the camera. The ground truth movements 303b, 302b, 301b show that the primary camera motion occurs along the local z- axis, as indicated by the ground truth movement 301b, with the z-distance increasing over time as the camera is withdrawn from the colon. In contrast, movements along the local x- axis and y-axis (represented by the ground truth movements 303b and 302b respectively) are minor and fluctuate around zero, likely due to vertical and horizontal swinging of the camera. These fluctuations are negligible compared to the movement along the z-axis. Thus, it has been determined that focusing on local z-axis movements of the camera is sufficient for colon localization. Therefore, the local z-axis motion of the camera may be determined in accordance with aspects of the present disclosure without determining a local x-axis motion or a local y-axis motion of the camera.[0085| It can be seen from Fig. 4 that the predicted movements 303a, 302a, 301a of the camera along the local x-axis, local y-axis and local z-axis respectively as predicted using the AF-SfMLeamer algorithm closely match the corresponding ground truth movements 303b, 302b, 301b, indicating that the AF-SfMLeamer algorithm offers a solution for colon localization.[0086} Applying a pre-trained model like AF-SfMLeamer to predict the camera's global trajectory[0087| on an unseen dataset may result in discrepancies from the ground truth. However, predicting local movement (e.g. along the local z-axis) is simpler than predicting the global camera trajectory, as it can avoid errors from inaccurate rotation predictions. This makes local movement prediction more accurate and feasible than predicting global camera trajectory, for real-world applications.

[0088] Returning to Fig. 3, based on the determined local z-axis motion, each frame of the plurality of frames 108 may be mapped to a respective relative location in the portion of the gastrointestinal tract. The plurality of frames 108 may be grouped into a plurality of sets of frames based on the determined local z-axis motion (e.g. based on the mapping) such that each set of frames corresponds to a substantially identical distance moved by the camera along the local z-axis.|0089] Fig. 5 is a graph illustrating the mapping of frames (represented on the horizontal axis by frame index times) to standardised colon locations represented by the vertical axis, wherein 0 on the vertical axis corresponds to the ascending colon and 1 corresponds to the rectum. Fig. 5 shows the colon can be divided into a predetermined number of segments of equal length, in this example ten, with the segments mapped to the corresponding video frames using the determined local z-axis motion of the camera (indicated by the sloping line on the graph).(0O90| Each set of frames is input into one or more severity classification models 110 to generate, for each set of frames, a corresponding severity score. The one or more severity classification models 110 may be similar or identical to any of the severity classification models 110 disclosed herein (e.g., in relation to Fig. 1). A plurality of severity scores 212 are therefore generated using the one or more severity classification models 110, each severityscore indicative of a severity of the gastrointestinal disease at a different location within the gastrointestinal tract.|00911 An overall severity score 216, in this case a Cumulative Disease Score such as a Cumulative Mayo Score, is generated based on processing the severity scores 212 (e.g., Mayo scores) generated for each set of frames. The Cumulative Disease Score is generated by summing (e.g., using an aggregator 214) the severity- scores 212 generated for each set of frames. In an example where the video 102 has been divided into ten sets of frames, each set of frames having a generated Mayo score ranging from 0 to 3, the Cumulative Mayo Score will range between 0 and 30. It should be understood, however, that in other examples theCumulative Mayo Score (or other type of Cumulative Disease Score) may have a different range than between 0 and 30, wherein the range will depend upon the number of sets of frames and / or the possible range of severity scores that could be generated for each set of frames using the one or more severity classification models 110.[0092} One or more of the severity scores 212 (e g., Mayo scores) and / or the Cumulative Disease Score (e.g., Cumulative Mayo Score) may be output to a user via a display of a user device, e g., a smartphone, a tablet, a personal computer or the like. The one or more of the severity scores (e.g., Mayo scores) and / or the Cumulative Disease Score (e.g., Cumulative Mayo Score) may be output in an application performing at least a part of the method. The one or more of the severity’ scores (e.g., Mayo scores) and / or the Cumulative Disease Score (e.g., Cumulative Mayo Score) may be output as an overlay on one or more of the filtered frames.(0093] FIG. 6A shows an example model architecture 300 for a machine-learning frame filtering network (such as the frame filtering network 106 discussed in relation to Fig. 1 and / or Fig. 2) and / or a severity' classification model (such as the severity classification model 110 discussed in relation to Fig. 1 and / or Fig. 2). This example model architecture is based on an Expandable 3D (X3D) architecture (described, for example, in “X3D: Expanding Architectures for Efficient Video Recognition”, C. Feichtenhofer. arXiv:2004.04730 (9 April 2020) the contents of which are incorporated herein by reference).(0094] X3D networks progressively expand a 2D network across the following axes indicated in Fig. 6A: temporal duration yt, frame rate yT, spatial resolution ys, width yw, bottleneck width yb, and depth ya. The X3D network may comprise a plurality of layers 606, 608, 610. A plurality of input frames 604 (such as a plurality of the filtered frames 108) forming a video clip are input into the X3D network and are processed to provide a prediction 612 (e.g., classification) as an output.(0095 FIG. 6B shows an example transformer-based structure for a machine-learning frame filtering network (such as the frame filtering network 106 discussed in relation to Fig.1 and / or Fig. 2) and / or a severity' classification model (such as the severity classification model 110 discussed in relation to Fig. 1 and / or Fig. 2). This example is based Vision Transformer (ViT) model (described, for example, in "‘AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE”, A. Dosovitskiy etal., arXiv: 2010.11929v2 (3 June 2021), the contents of which are incorporated herein by reference).

[0096] As indicated in Fig. 6B, the ViT 350 receives as input a ID sequence of token embeddings. To handle 2D images such as frames 104 or frames 108. an input frame is reshaped into a sequence of flattened 2D patches 352, which may be transformed into embeddings and processed by the transformer encoder 354. A classification head is implemented by a MLP with one hidden layer at pre-training time and by a single linear layer at fine-tuning time. The transformer encoder 354 consists of alternating layers of multiheaded selfattention (MSA) and MLP blocks. Layemorm (LN) is applied before every block, and residual connections after every block.

[0097] The use of transformer models can improve the classification performance over CNN-based approaches such as X3D. Table I shows a comparison of the segmentation accuracy on a validation dataset for X3D (a CNN-based model) and a ViT (a Transformerbased model), showing the increased test accuracy when using a ViT model.Model Test AccuracyX3D 49.81ViT 75.45Table I

[0098] FIG. 7 A shows a schematic example of a system 400 for training a frame filtering network (e.g., the frame filtering network 106) to classify input frames as infomiative or non- informative, wherein the training utilises active learning. The method may be performed by one or more computing devices / sy stems, such as the device / system described in relation to FIG. 10.[009 [ At data ingestion 402, a plurality of training frames of endoscopic videos are received. At data annotation 404, annotations are provided by one or more human annotators 406 for a subset of the training frames, the annotations classifying the training frames as informative or non-informative. A foundation model 408. which may comprise any frame filtering network 106 disclosed herein, is implemented for classifying the training frames as informative or non-informative, to filter out “not useful” clips. The foundation model 408 (i.e., frame filtering network 106) may implement frame-based classification or video-based classification. The frame-based method processes input training frames by frame, while thevideo-based model processes whole video clips (e.g., a plurality of sequential training frames). Experimentally, it has been found that video-based models generally achieve better performance than frame-based models. At model finetuning 410, the foundation model 408 is fine-tuned with an active learning loop to keep the informative clips only. Active learning is a semi-supervised technique where an active learning algorithm selectively queries the most informative and representative data points from an unlabeled dataset to be labelled (annotated) by the human annotator(s) 406. In some examples, a margin sampling strategy is used to rank candidate clips from the unlabeled pool based on model uncertainty. The system can select the clips with the smallest differences between the top two predicted class probabilities. This approach can target the clips closest to the decision boundary, which are considered most informative for improving model performance. The active learning loop can be performed iteratively. In each iteration, the system can retrain the foundation model using the updated annotated dataset, evaluate candidate clips using the margin sampling heuristic, and present newly selected clips to the human annotators 406 for labeling. Such active learning allows for the dynamic identification of frames that are most infomiative for analysis, discarding (e.g., up to 30% of) incorrect or irrelevant frames while preserving frames that are crucial for accurate diagnosis. The foundation model 408 may be evaluated 412 over iterations of the training system 400 until one or more threshold conditions are satisfied, e.g., a threshold number of training epochs or a threshold performance on a test / evaluation dataset, at which stage the foundation model 408 may be deployed 414 for inference.[01001 One or more of the severity classification models 110 described herein may be trained in a similar manner to the frame filtering network 106, with the training dataset for a severity' classification model 110 comprising a plurality of training examples, each comprising a frame of an endoscopic video and a corresponding set of one or more ground truth severity scores.

[0101] FIG. 4B shows an additional and / or alternative example of a method 402 for training one or more severity’ classification models 404 for generating a severity score. The method 402 may be performed by one or more computing devices / systems, such as the device / system described in relation to FIG. 10. The one or more severity classification models 404 may include any severity classification model disclosed herein, such as severity classification model 110.[0.102| The method 401 may be used to fine-tune a pretrained classification model to specialise in classifying frames showing a gastrointestinal disease, such as ulcerative colitis. Pre-trained models have demonstrated exceptional effectiveness across various domains. The utilization of these pre-existing models accelerates the development process and enhances model robustness by providing a solid foundation of learned features. Alternatively, the method 402 may be used to train a classification model 404 from scratch. The severity classification model 404 may have any of the structures described herein, for example those described in respect of Figs. 6A (an X3D architecture) and 6B (a ViT).[0.1031 The method 402 uses a training dataset 450 comprising a plurality of training examples 452. Each training example 452 comprises at least one training frame 454 of an endoscopic video and corresponding ground truth data 456 indicating a severity' score for a gastrointestinal disease indicted in the video. For example, where the training frame 454 is from a video of an endoscopy of a colon and the gastrointestinal disease is ulcerative colitis, the corresponding ground truth data 456 may indicate a Mayo score.[0104| The training dataset 450 may comprise a plurality of “real” examples, e.g., frames of an endoscopic video captured from a subject during an endoscopy. The training dataset 450 may further comprise a plurality of synthetic examples generated from one or more (e.g., a plurality) of the real training examples.|0105| The training / fine- tuning method 402 comprises obtaining a training example 452 from the training dataset 450 and inputting the training frame 454 of the training example 452 into a classification model 404. The classification model 404 processes the training frame 454 based on current values of parameters (e.g., weights and / or biases) of the classification model 404 to generate a candidate severity score 458 for the training frame. The candidate severity score 458 is indicative of a severity of a gastrointestinal disease that may be indicated in the training frame 454. For example, where the training frame 454 is from a video of an endoscopy of a colon and the gastrointestinal disease is ulcerative colitis, the candidate severity score 458 may be a candidate Mayo score.

[0106] The candidate severity score 458 is compared to the corresponding ground truth data 456 from the training example 452 using a loss / objective function 460. The loss / objective function 460 may be any appropriate classification loss / objective function known in the art. For example, the loss / objective function 460 may comprise a Softmax CrossEntropy Loss function, however a different suitable loss / objective function 460 may be used instead.[01071 Based on the value of the loss / objective function 460, updates to the parameters of the model 404 are determined. An optimisation procedure may be applied to the loss / objective function 460 to determine the parameter updates, e.g., stochastic gradient descent / ascent. The method 400 may be iterated over the training dataset 450 until one or more threshold conditions are satisfied, e.g., a threshold number of training epochs or a threshold performance on a test / eval nation dataset.[0108| FIG. 8 shows a flow diagram of an example method 800 for assessing the severity of a gastrointestinal disease using machine-learning. The method 800 may be performed by one or more computers operating in one or more locations. For example, the method may be performed by the system / apparatus described in relation to FIG. 10.

[0109] At operation 802, a video of an endoscopy is received, the endoscopy having been performed on at least a portion of a gastrointestinal tract of a subject. The video comprises a plurality of frames. The frames may be colour images, such as RGB images, captured by a camera of an endoscope, or the like.[01101 At optional operation 804, a frame filtering network (e.g., any frame filtering network 106 disclosed herein) is used to generate a plurality of filtered frames from the plurality of frames of the video. The filtered frames are a subset of the plurality of frames of the video, wherein each frame of the filtered frames has been classified as informative by the frame filtering network. The plurality of frames of the video are input into the frame filtering network, which processes them based on a set of learned parameters to classify each frame as informative or non-informative.[011 11 The frame filtering network may comprise one or more of: one or more convolutional layers; one or more skip / residual connections; one or more fully connected layers; one or more transfonner layers; and / or one or more multi-layer perceptrons. The frame filtering network may comprise a vision transformer or an X3D architecture, for example.

[0112] At operation 806, the filtered frames (or the frames of the video if operation 804 is not performed) are processed using one or more severity classification models (e.g., any severity classification model 110 disclosed herein) to generate a severity score of thegastrointestinal disease for each of the filtered frames (or frames of the video if operation 804 is not performed).

[0113] At operation 808, an overall severity score is generated based on the severity scores generated for the filtered frames (or frames of the video if operation 804 is not performed).|0114] One or more of the severity scores and / or the overall severity score may be output to a user (e.g., the subject and / or a healthcare professional) via a user interface, e.g., via an application on a user device, a program on a personal computer, or the like.

[0115] In some examples, the video is of a colon of a subject (i.e., captured during a colonoscopy) and the gastrointestinal disease is ulcerative colitis. In such examples, the severity' score for each frame and / or the overall severity score may, for example, comprise an Mayo score. Other severity’ scores may alternatively or additionally be used.10116] FIG. 9 shows a flow diagram of an example method 900 for assessing the severity of a gastrointestinal disease using machine-learning. The method 900 may be performed by one or more computers operating in one or more locations. For example, the method may be performed by the system / apparatus described in relation to FIG. 10.

[0117] At operation 902, a video of an endoscopy' is received, the endoscopy' having been performed on at least a portion of a gastrointestinal tract of a subject. The video comprises a plurality of frames. The frames may be colour images, such as RGB images, captured by a camera of an endoscope, or the like.

[0118] At optional operation 904, a frame filtering network (e.g., any frame filtering network 106 disclosed herein) is used to generate a plurality of filtered frames from the plurality of frames of the video. The filtered frames are a subset of the plurality' of frames of the video, wherein each frame of the filtered frames has been classified as informative by the frame filtering network. The plurality of frames of the video are input into the frame filtering network, which processes them based on a set of learned parameters to classify each frame as informative or non-informative.

[0119] The frame filtering network may comprise one or more of: one or more convolutional layers; one or more skip / residual connections; one or more fully connected layers; one or more transformer layers; and / or one or more multi-layer perceptrons. The frame filtering network may comprise a vision transformer or an X3D architecture, for example.[0120| At operation 906, a local z-axis motion of a camera used to generate the endoscopy video is determined based on the plurality of frames. Determining the local z-axis motion of the camera may comprise processing the plurality of frames using a motion model to output the local z-axis motion. The local z-axis motion of the camera may be determined without determining a local x-axis motion or a local y-axis motion of the camera. Processing the plurality of frames using the motion model may comprise processing the plurality of frames using the motion model to determine a speed of the camera along the z-axis at each frame. Processing the plurality of frames using the motion model may comprise processing the plurality of frames using a Structure-from-Motion, SfM, Learner algorithm. Each frame of the plurality of frames may be mapped to a respective relative location in the portion of the gastrointestinal tract based on the determined local z-axis motion.

[0121] At operation 910, and based on the detennined local z-axis motion, the filtered frames (or the frames of the video if operation 904 is not performed) are processed using one or more severity classification models (e.g., any severity classification model 110 disclosed herein) to generate a plurality of severity scores, each severity score indicative of a severity of the gastrointestinal disease at a different location within the gastrointestinal tract.

[0122] The plurality of frames may have been grouped into a plurality of sets of frames based on the determined local z-axis motion (e.g. based on the mapping of each frame of the plurality of frames to a respective relative location in the portion of the gastrointestinal tract) such that each set of frames corresponds to a substantially identical distance moved by the camera along the local z-axis. Processing the plurality of frames to generate a plurality of severity scores may comprise processing, using the one or more severity classification models, each set of frames to generate, for each set of frames, a corresponding severity score of the plurality of severity scores.

[0123] At operation 912, an overall severity score is generated based on the plurality of severity scores, the overall severity score indicative of an overall severity' of the gastrointestinal disease for the portion of the gastrointestinal tract. The overall severity score may be a cumulative severity score (such as a Cumulative Mayo Score), and generating the overall severity’ score may comprise summing the plurality of severity scores to generate the cumulative severity score.[0.124] One or more of the seventy scores and / or the overall severity score may be output to a user (e.g., the subject and / or a healthcare professional) via a user interface, e.g., via an application on a user device, a program on a personal computer, or the like.|0125] In some examples, the video is of a colon of a subject (i.e., captured during a colonoscopy) and the gastrointestinal disease is ulcerative colitis. In such examples, the severity score for each frame may comprise an Mayo score and / or the overall severity score may comprise a Cumulative Mayo Score, for example. Other severity scores may alternatively or additionally be used.[0.1261 One or more of the severity scores and / or the Cumulative Disease Score may be output to a user (e.g., the subject and / or a healthcare professional) via a user interface, e.g., via an application on a user device, a program on a personal computer, or the like.|0127] FIG. 10 shows a schematic example of a system / apparatus 700 for performing any of the methods described herein. The system / apparatus shown is an example of a computing device. The apparatus 700 may be in the form of a personal computer, mobile computer device (such as a smart phone, tablet and / or laptop), or distributed computing system.

[0128] The apparatus (or system) 700 comprises one or more processors 702. The one or more processors control operation of other components of the system / apparatus 700. The one or more processors 702 may, for example, comprise a general-purpose processor. The one or more processors 702 may be a single core device or a multiple core device. The one or more processors 702 may comprise a Central Processing Unit (CPU) or a graphical processing unit (GPU). Alternatively, the one or more processors 702 may comprise specialised processing hardware, for instance a RISC processor or programmable hardware with embedded firmware. Multiple processors may be included.

[0129] The system / apparatus comprises a working or volatile memory 704. The one or more processors may access the volatile memory 704 in order to process data and may control the storage of data in memory7. The volatile memory7704 may comprise RAM of any type, for example, Static RAM (SRAM) or Dynamic RAM (DRAM), or it may comprise Flash memory, such as an SD-Card.

[0130] The system / apparatus comprises a non-volatile memory 706. The non-volatile memory 706 stores a set of operation instructions 708 for controlling the operation of the processors 702 in the form of computer readable instructions. The non-volatile memory 706may be a memory' of any kind such as a Read Only Memory' (ROM), a Flash memory' or a magnetic drive memory.

[0131] The one or more processors 702 are configured to execute operating instructions 708 to cause the system / apparatus to perform any of the methods described herein. The operating instructions 708 may comprise code (i.e. drivers) relating to the hardware components of the system / apparatus 700, as well as code relating to the basic operation of the system / apparatus 700. Generally speaking, the one or more processors 702 execute one or more instructions of the operating instructions 708, which are stored permanently or semipermanently in the non-volatile memory 706, using the volatile memory 704 to store temporarily data generated during execution of said operating instructions 708.

[0132] Any mentioned apparatus and / or other features of particular mentioned apparatus may be provided by apparatus arranged such that they become configured to carry out the desired operations only when enabled, e.g. switched on, or the like. In such cases, they may not necessarily have the appropriate software loaded into the active memory in the nonenabled (e.g. switched off state) and only load the appropriate software in the enabled (e.g. on state). The apparatus may comprise hardware circuitry and / or firmware. The apparatus may comprise software loaded onto memory. Such software / computer programs may be recorded on the same memory / processor / functional units and / or on one or more memories / processors / functional units.

[11133] Any mentioned apparatus / circuitry / elements / processor may have other functions in addition to the mentioned functions, and that these functions may be performed by the same apparatus / circuitry / elements / processor. One or more disclosed aspects may encompass the electronic distribution of associated computer programs and computer programs (which maybe source / transport encoded) recorded on an appropriate carrier (e.g. memory’, signal).

[0134] Any “computer” described herein can comprise a collection of one or more individual processors / processing elements that may or may not be located on the same circuit board, or the same region / position of a circuit board or even the same device. In some examples one or more of any mentioned processors may be distributed over a plurality of devices. The same or different processor / processing elements may perform one or more functions described herein.

[0015] The term “signalling” may refer to one or more signals transmitted as a series of transmitted and / or received electrical / optical signals. The series of signals may comprise one,two, three, four or even more individual signal components or distinct signals to make up said signalling. Some or all of these individual signals may be transmitted / received by wireless or wired communication simultaneously, in sequence, and / or such that they temporally overlap one another.[0136} With reference to any discussion of any mentioned computer and / or processor and memory' (e.g. including ROM, CD-ROM etc.), these may comprise a computer processor, Application Specific Integrated Circuit (ASIC), field-programmable gate array (FPGA), and / or other hardware components that have been programmed in such a way to carry out the inventive function.[01 71 The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole, in the light of the common general knowledge of a person skilled in the art. irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that the disclosed aspects / examples may consist of any such individual feature or combination of features. In view of the foregoing description, it will be evident to a person skilled in the art that various modifications may be made within the scope of the disclosure.101 8[ While there have been shown and described and pointed out fundamental novel features as applied to examples thereof, it will be understood that various omissions and substitutions and changes in the form and details of the devices and methods described may be made by those skilled in the art without departing from the scope of the disclosure. For example, it is expressly intended that all combinations of those elements and / or method steps which perform substantially the same function in substantially the same way to achieve the same results are within the scope of the disclosure. Moreover, it should be recognized that structures and / or elements and / or method steps shown and / or described in connection with any disclosed form or examples may be incorporated in any other disclosed or described or suggested form or example as a general matter of design choice. Furthermore, in the claims means-plus-function clauses are intended to cover the structures described herein as performing the recited function and not only structural equivalents, but also equivalent structures.[0.139| Implementations of the methods described herein may be realised as in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These may include computer program products (such as software stored on e.g. magnetic discs, optical disks, memory, Programmable Logic Devices) comprising computer readable instructions that, when executed by a computer, such as that described in relation to Figure 10, cause the computer to perform one or more of the methods described herein.[0140| Any system feature as described herein may also be provided as a method feature, and vice versa. As used herein, means plus function features may be expressed alternatively in terms of their corresponding structure. In particular, method aspects may be applied to system aspects, and vice versa.|0141] Furthermore, any, some and / or all features in one aspect can be applied to any, some and / or all features in any other aspect, in any appropriate combination. It should also be appreciated that particular combinations of the various features described and defined in any aspects of the invention can be implemented and / or supplied and / or used independently.[0142| Although several embodiments have been shown and described, it would be appreciated by those skilled in the art that changes may be made in these embodiments without departing from the principles of this disclosure, the scope of which is defined in the claims and their equivalents.

Claims

CLAIMS1 . A computer implemented method for assessing the severity of a gastrointestinal disease, the method comprising: receiving a video of an endoscopy performed on a portion of a gastrointestinal tract of a subject, the video comprising a plurality of frames; determining, based on the plurality of frames, a local z-axis motion of a camera used to generate the endoscopy video; based on the local z-axis motion, processing, using one or more severity' classification models, the plurality of frames to generate a plurality of severity scores, each severity score indicative of a severity of the gastrointestinal disease at a different location within the gastrointestinal tract; and generating, based on the plurality of severity scores, an overall severity score that is indicative of an overall severity of the gastrointestinal disease for the portion of the gastrointestinal tract.

2. The method of claim 1, wherein determining the local z-axis motion of the camera comprises processing the plurality of frames using a motion model to output the local z-axis motion.

3. The method of claim 2, wherein processing the plurality' of frames using the motion model comprises processing the plurality of frames using the motion model to determine a speed of the camera along the z-axis at each frame.

4. The method of claim 2 or 3, wherein processing the plurality of frames using the motion model comprises processing the plurality of frames using a Structure-from-Motion, SfM, Learner algorithm.5 The method of any preceding claim, wherein the local z-axis motion of the camera is determined without determining a local x-axis motion or a local y-axis motion of the camera.

6. The method of any preceding claim, further comprising mapping each frame of the plurality of frames to a respective relative location in the portion of the gastrointestinal tract based on the determined local z-axis motion.

7. The method of any preceding claim, further comprising grouping the plurality of frames into a plurality of sets of frames based on the determined local z-axis motion such that each set of frames corresponds to a substantially identical distance moved by the camera along the local z-axis.

8. The method of claim 7, wherein processing the plurality of frames to generate a plurality of severity scores comprises processing, using the one or more severity classification models, each set of frames to generate, for each set of frames, a corresponding severity score of the plurality of severity scores.

9. The method of any preceding claim, wherein the plurality of frames are a plurality of filtered frames that have been determined based on processing the video using a frame filtering network comprising one or more vision transformer models, optionally, wherein the frame filtering network has been trained using active learning.

10. The method of any preceding claim, wherein the one or more severity classification models comprise a pre-trained classification model that has been fine-tuned on a training dataset comprising a plurality of frames from at least one endoscopic video and respective ground truth severity scores indicative of a severity of a gastrointestinal disease in each frame.

11. The method of any preceding claim, wherein the one or more severity' classification models utilise an Expandable 3D, X3D, architecture.

12. The method of any preceding claim, wherein the one or more severity classification models comprise one or more vision transformer models.

13. The method of any preceding claim, wherein the overall severity score is a cumulative severity score, and wherein generating the overall severity score comprises summing the plurality of severity scores to generate the cumulative severity score.

14. A system comprising: one or more processors: and a memory, the memory storing computer readable instructions that, when executed by the one or more processors, causes the system to perform the method of any preceding claim.

15. A computer program product comprising computer readable instructions that, when executed by data processing apparatus, cause the data processing apparatus to perform operations comprising a method according to any of claims 1 to 13.

Images