Cell shape analysis

Abstract

A method of predicting for a cell subject to a perturbation, for example the application of a new drug, a related perturbation, for example a previously studied drug, is disclosed. The method comprises obtaining a sequence over a period of time of cell shapes of a cell subject to the perturbation, generating a sequence of shape features by generating a shape feature for each cell shape in the sequence and applying the sequence of shape features as an input to a sequence to sequence neural network to generate a sequence of shape sequence features. The sequence of shape sequence feature is input to a classifier neural network jointly trained with the sequence to sequence neural network. A related perturbation is predicted based on the output of the classifier network and relevancy scores may be provided indicating which shape in the sequence of shapes is relevant to the prediction.

Description

CELL SHAPE ANALYSIS TECHNICAL FIELD

[0001] The present disclosure relates to the analysis of cell shape captured from images of cellsand, more specifically, to the prediction from cell shape of a perturbation, for example, the applicationof a drug or a genetic modification, to which the cells are subject.BACKGROUND

[0002] Known techniques use neural networks to process cell shapes captured from images of cellsin histology slides or in culture in order to predict a drug that has been applied to the cells

[0044] . Theneural network is trained using pairs of images of the cells and a known drug that was applied. Thetrained neural network can be used to predict an applied drug for cells to which an unknown drug hasbeen applied to identify the drug. The trained neural network can also be used if a new, known drugbut, say, with an unknown mode of action, is applied to predict a related drug. While in this case, thereis no need to identify the known drug per se, predicting a related drug enables inferences to be madeabout the mode of action of the applied drug, for example, may suggest that the applied drug affectsthe same or similar biological pathways as the predicted related drug, or simply that the applied and related drug may have similar on or off target effects. Recent advances in 3D imaging microscopy have enabled this analysis to be carried out using neural network techniques applied to the 3D shape of cells. SUMMARY

[0003] Aspects of an invention are set out in the appended independent claims. Optional features ofembodiments are set out in the dependent claims.

[0004] The inventors have realised that the dynamic behaviour of cells contains valuable additionalinformation over static images of cells and that the prediction of perturbations to which cells are subject to can be treated as a multi-variate time-series classification problem. Consequently, the inventors have developed neural network techniques to analyse this behaviour to predict perturbationsto which cells are subject to from a time series of shapes obtained from a time series of video framesof cells.

[0005] Aspects of the disclosure provide a computer-implemented method of predicting a relatedperturbation for a cell subject to a perturbation. The perturbation, and / or related perturbation, may, forexample, comprise one or more of: an agent, for example, a drug molecule, antibody, virus orbacteria, applied to the cell; a genetic modification of the cell, such as a gene knock-out or knock-down; and an environmental condition of the cell. In some cases, the perturbation may comprise theapplication of an agent, for example a drug, and the predicted related perturbation may comprise agenetic modification such as a gene knock-out or knock down, thereby enabling the drug to be linkedto the knocked-out / down pathway or gene. In some cases, the perturbation was applied to the cell at apoint in time and the time period is a time period subsequent to the point in time, when a response ofthe cell to the perturbation is at steady state. For example, the perturbation may relate to a geneticmanipulation, or the perturbation may comprise the application of drug and a sufficient amount of time 1P233173GBis waited after the application. In other cases, the sequence is obtained for an earlier time period and captures a transient response to the perturbation, for example the application of a drug.

[0006] A sequence of cell shapes of a cell subject to the perturbation is obtained over a period oftime and a sequence of shape features is generated by generating a shape feature for each cell shape in the sequence, each shape feature characterising a respective cell shape. The shapes may bethree-dimensional shapes and may be represented as, for example, point clouds, polygon meshes orbinary voxel masks. The term cell shape will be understood to refer to the shape of the cell, that is its outline or outer shape as obtained by a suitable shape segmentation and tracking algorithm. Forexample, the cell may have been imaged to generate a set of stacked slice images at each time pointof the sequence. Each cell shape may then have been generated using a suitable segmentation / tracking / shape generation technique from the respective set of stacked slice images. Slice imagesmay be obtained with various techniques, for example Light Sheet Fluorescent Microscopy

[0022] , confocal microscopy. Other techniques include two-photon microscopy, selective plane illumination microscopy (SPIM), Structured Illumination Microscopy (SIM), Super-Resolution Microscopy Techniques (e.g., STED, PALM, STORM), Spinning Disk Confocal Microscopy, Confocal Laser Scanning Microscopy (CLSM) and so forth.

[0007] Many ways of generating features for use with neural networks are known and includemanually designed features

[0011] , for example including features like Protrusivity or Spreading

[0011] orfeatures extracted from 3D convolutional networks. Other ways of generating shape-related featuresare also known and may be used in the disclosed method, for example spherical harmonics asdiscussed in

[0034] . In fact, a point cloud or other 3D representation itself can be used as the shapefeature in subsequent processing

[0040] . Shape features may be generated as the output of an encoderof a trained autoencoder neural network, for example, applied to a point cloud representation of theshape, such as FoldingNet

[0031] , or other PointNet

[0035] variants. Where the shapes are represented asmeshes, mesh-based autoencoders [36-39] could be used. A specific example of a point cloud-basedhybrid autoencoder is described below. In some implementations, however, shape features aregenerated from two-dimensional (2D) shapes obtained from images of cells.

[0008] In some implementations, the shape feature is additionally characteristic of a shape of anucleus of the respective cell. For example, a feature embedding may be obtained separately for the cell shape and the nucleus, using the same or a different embedding and the feature embeddingscombined, for example, concatenated. Typically, features and feature embeddings will be representedas numeric feature vectors of a suitable dimension to represent the shape in question. Other additionalinformation can be included in the shape feature in some implementations, for example, by combininga corresponding additional embedding with the feature embeddings for the cell shape and / or nucleus,for example, by concatenating the embeddings. Additional information may, in some implementations,relate to a cell cycle of the cell, more specifically, a cell cycle stage and a corresponding embeddingmay include an indicator indicating a corresponding stage of the cell cycle the cell is in for each frame or an embedding of a corresponding signal, for example (Proliferating cell nuclear antigen (PCNA)

[0049] . Other additional information that may be included in combined or separate additional embeddingsinclude fluorescent colour (in a suitable colour space, for example), fluorescent intensity integrated 2P233173GBover the cell, a ratio of intensities of constituent colours of a fluorescent colour of the cell. Theadditional information may include a ratio or an embedding of a ratio of fluorescent intensities indifferent regions of the cell, for example intensity from the nucleus and a region, for example a ringaround the nucleus, for example as an indication of cell cycle stage. For the avoidance of doubt, thefeature embeddings of the cell shape (and in some instances the nucleus shape)may be generated asdescribed above.

[0009] More generally, the term “embedding” of a feature is used herein in the sense of anyrepresentation of the feature that can be processed by computational means, for example a number, avector, matrix or tensor of numbers, a string, and the like. For example, an autoencoder produces an embedding of its input at its bottleneck layer, that is as the output of its encoder part. An embedding of a feature can therefore be said to contain information on, be associated with or be characteristic of the feature of which it is an embedding.

[0010] The sequence of shape features is applied as an input to a sequence to sequence neuralnetwork to generate a sequence of shape sequence features, so that each sequence shape feature isconditioned on a respective shape feature and one or more other shape features in the sequence, forexample two or more other shape features in the sequence. The respective and one or two or moreother shape features may be adjacent each other in the sequence. For example, a first layer of thesequence to sequence neural network may process the respective and one adjacent shape feature togenerate a hidden sequence feature of a sequence of hidden sequence features and a second layerof the sequence to sequence neural network may process the hidden sequence feature and anadjacent hidden sequence feature to generate the corresponding shape sequence feature. Manysuitable sequence to sequence neural networks that generate an embedding of the whole sequence for each frame (rather than independent embedding of each frame, independent of the other frames inthe sequence) are known and can be used in this method, for example Long-Short-Term-Memory(LSTM) recurrent neural networks, transformer neural networks or graph transformer neural networks. Naturally, any suitable sequence to sequence neural network model can be used, in particular those which can handle variable length sequences, such as LSTM, transformers or graph transformers.

[0011] The sequence of shape sequence features is applied to a classifier neural network togenerate a classification score for each of one or more classes. Each class corresponds to one ormore related perturbations. The classifier and sequence to sequence neural networks have beenjointly trained to predict respective training perturbations from sequences of cell shapes of cellssubject to the respective training perturbations. Using the classification scores for the one or moreclasses one or more related perturbations are predicted.

[0012] Advantageously, the prediction of the related perturbations can use the informationcontained in the sequence, that is in the dynamic behaviour of the cell. This can be contrasted with information obtained from cells in a histology slide or cell culture. While it can be expected that images of multiple cells in the slide or culture, captured at the same time, will correspond to snap shots of different shapes over time (as cell shape dynamics is unlikely to be synchronised between cells), the order of the different shapes is lost. By imaging sequences of shapes of individual cells and using the 3P233173GBimaged sequence of predicting, the disclosed method can make use of additional information as compared to prior methods.

[0013] In some implementations, the classifier neural network comprises an attention head and isconfigured to apply each sequence shape feature as an input to the attention head to generate as an output of the attention head an attention score for each sequence shape feature in the sequence. In these implementations, the classifier neural network is further configured to combine, directly orindirectly, the shape sequence features weighted by their respective attention scores to generate theclassification score for each class. Advantageously, the attention scores provide an indication as towhich frames are contributing to the prediction and therefore can provide an explanation for theprediction.

[0014] In some such implementations, the classifier neural network further comprises aclassification head and is configured to apply each sequence shape feature as an input to the classification head to generate as an output of the classification head a shape classification score for each class of the one or more classes and each sequence shape feature in the sequence. The classifier neural network is further configured to combine the generated shape classification scoresweighted by the respective attention score to generate the classification scores, thereby (indirectly)combining the shape sequence feature weighted by their respective attention scores to generate theclassification score. Advantageously, by generating a shape classification score for each shape in the sequence, interpretability can be improved. This is because for each class, each shape classification score indicates how much the respective shape has contributed to positively. While a high attentionscore per se may indicate that a shape has contributed, this could be to boost or suppress a givenclass / prediction. In other words, the shape classification scores are more directly related to the ultimate prediction and hence provide an improved explanation. Additionally, since the shape classification scores are generated in parallel to the attention scores, not only is there a computational efficiency in parallelizing this computation, the individual shape classification scores are moremeaningful as the classifier computes shape classification and attention scores separately without onedirectly influencing the other.

[0015] The method may comprise computing a relevance score for each shape feature in thesequence using the respective attention score and shape classification score(s) and identifyingshapes in the sequence for which the respective relevance score meets a relevancy condition as relevant to the prediction of the one or more related perturbations. The identified shapes can help to provide an explanation of the basis for the prediction made based on the classification scores. The identified shapes can further be used to provide additional information. For example, the method may comprise matching each of the identified shapes against a database of database cell shapes associated with known biological pathways. Biological pathways associated with matched database cell shapes can then be identified as potentially relevant for the perturbation to which the cell is subject to. For example, the associated biological pathways may be affected by the perturbation on the basis of the corresponding cell shape being part of the dynamic sequence of cell shapes leading to the perturbation being predicted. 4P233173GB

[0016] In some implementations the classifier model is configured to apply each sequence shapefeature weighted by its respective attention score as an input to a classification head of the classifier model to generate as an output of the classification head a shape classification score for each class of the one or more classes and each sequence shape feature in the sequence. That is the classification head uses the attention score in generating the shape classification score. The classifier model isfurther configured to combine the generated shape classification scores to generate the classificationscore, thereby combining the shape sequence feature weighted by their respective attention scores.While this approach, the roles of the attention and shape classification scores are less separated, the shape classification score already use the attention score, so that a separate combination of shape classification scores and attention scores to determine relevancy scores is not required.

[0017] In some implementations, the sequence to sequence neural network comprises one or moregraph transformer layers, for example two graph transformer layers. The sequence of shape featuresmay be represented by a graph having the shape features of the sequence as nodes and respectiveedges between pairs of adjacent nodes adjacent in the sequence. The graph may additionallycomprise edges between pairs of nodes for which a similarity measure between the nodes of the pairexceeds a threshold, in one or more selected layers of the graph transformer network or in all layers.

[0018] Some aspects of the disclosure provide a method of jointly training the sequence tosequence and classifier neural networks used in the methods described above, the sequence tosequence and classifier networks each being defined by a corresponding set of parameters. Themethod comprises obtaining a training data set that comprises a plurality of training pairs. Eachtraining pair comprises a training sequence of shape features and a corresponding label identifying arespective training perturbation. Obtaining the training data set comprises obtaining a trainingsequence over a period of time of cell shapes of a cell subject to the respective training perturbation and generating the training sequence of shape features by generating a shape feature for each cell shape in the sequence, each shape feature characterising a respective cell shape.

[0019] The method further comprises applying each training sequence of shape features as aninput to a sequence to sequence neural network to generate a sequence of shape sequence featuresand applying the sequence of shape sequence features to the classifier neural network to generate aclassification score for each of one or more classes. Each class corresponds to one or moreperturbations that the cells can be subject to. For each training sequence of shape features, thegenerated classification scores are compared to the respective label to compute a loss function andthe parameters of the sets of parameters are adjusted to reduce the loss function, for example a cross-entropy loss function.

[0020] In some particular use cases, the neural networks are trained on training data obtained fromcells from know-down or knock-out gene(s) genetically modified animals or cell lines, that is the perturbation which the cells are the subject of are gene knock-out / down modifications and the trained neural networks are used to predict related perturbations (knock-out / down gene modification) for cells subject to an agent, such as a drug perturbation, In this way, the neural networks can be used to predict related pathway(s) or gene gene(s) for the agent in that application of the agent and the predicted gene modification result in similar dynamic behaviour of the cell. 5P233173GB

[0021] Aspects of the disclosure further extend to one or more computer-readable media, or acomputer program product, encoding coded instructions that, when executed on a processorimplement a method as described above, and to a system comprising one or more computerprocessors and a memory, the memory encoding coded instructions that, when executed on the oneor more computer processors implement a method as described above.BRIEF DESCRIPTION OF DRAWING

[0022] Specific implementations are now described by way of example and with reference to theaccompanying drawings, in which: Figure 1 illustrates a system for capturing and processing cell shapes;Figure 2 illustrates a method for predicting a related perturbation from a sequence of cellshapes of a cell subject to a perturbation; Figure 3A illustrates a sequence of imaged three dimensional volumes;Figure 3B illustrates a corresponding sequence of cell shapes;Figure 4A illustrates an autoencoder for generating a shape feature;Figure 4B illustrates fine tuning an encoder for shape feature generation;Figure 5 illustrates a joint sequence to sequence and classifier model neural networkarchitecture; Figure 6A illustrates a method for generating a prediction with a relevance explanation;Figure 6B illustrates an option for pooling shape sequence predictions and generating arelevance vector; Figure 6C illustrates another option for pooling shape sequence predictions and generating a relevance vector; Figure 7A illustrates a method of identifying potential biological pathways associated with aperturbation based on cell shapes identified to be relevant for predicting a related perturbation;Figure 7B illustrates a data base for use in the method of Figure 7A; Figure 8 illustrates a method of training a joined sequence to sequence and classifier neural network; and Figure 9 illustrates a computing device for use in implementing the disclosed methods.SPECIFIC DESCRIPTIONAcquisition and analysis system and method overview

[0023] With reference to Figure 1, a system 100 for acquiring a sequence, or time-series, of 3Dimages of cells comprises a microscope 102, for example a Light Sheet Fluorescence Microscope102, for example imaging fluorescence from within an oblique sheet of light. LSFM is a well-knowntechnique and a particular implementation is discussed in

[0022] . Any other imaging technique capableof acquiring images of spatially defined slices or volumes, such as confocal microscopy, two-photon microscopy and the like may be used instead. A sample stage 104 is configured to hold a sample, forexample a tissue sample, cells suspended in a collagen matrix or a cell culture, and move it relative tothe microscope 102 so that different slices of the sample can be imaged in order to image a sufficientvolume of the sample to acquire cell shapes. An image acquisition module 106, for example a Charge6 P233173GBCoupled Device or other image acquisition sensor is configured to acquire a digital image of the slicesimaged by the microscope 102 and store the acquired digital images in an image storage 108 as 3Dimages, each comprising a number of acquired slices. The components of the system 100 are underthe control of an acquisition controller 110, for example implemented on a suitable computing device.

[0024] A shape engine 120 is configured to access the stored 3D images and to extract a cellshape from each 3D image. To that end, a cell is segmented and tracked in each slice and the segmented cell image is converted to a suitable shape representation, such as a polygon mesh, a point cloud or a binary voxel mask. Segmentation and tracking may use any suitable known technique,for example Otsu’s thresholding and a simple particle tracking algorithm, respectively. In someimplementations, nucleus shape is also captured for each cell, for example using active contours. In some instances, shapes may initially be captured by a meshes and then converted into point clouds,or the shapes may be further processed as a meshes. The shape engine accesses the stored 3Dimages and generates and stores a corresponding sequence of cell shapes in data storage 130. Insome implementations, the system is configured to acquire 2D instead of 3D images and to generate 2D shapes.

[0025] A machine learning engine 140 implements a neural network model, accesses shapesequences stored in data storage 130 and processes the stored sequences with the neural network topredict related perturbations related to perturbations to which an imaged cell has been subject to in the sample in an inference mode. In order to train the neural network, the data storage 130 may store pairs of sequences and labels identifying the perturbation to which the cell in the respective sequencewas subjected to. Inference and training is described in more detail below. The ML engine 140 outputsits results or receives user commands and / or instructions via an interface 150, for example a user interface or a network interface.

[0026] It will be appreciated that the various system components described above represent logicalmodules that may be implemented in any combination of one or more hardware devices and that thedelineation of the modules is reflective of their function rather than necessarily their physical implementation.

[0027] A method 200, for example implemented on the system 100, comprises a step 210 ofobtaining a sequence of cell shapes of a cell subject to a perturbation. Obtaining the sequence may comprise accessing a pre-prepared sequence, or controlling the system 100 to obtain the sequence as described above. The cell from which the sequence is obtained is subject to a perturbation while the sequence is obtained. The perturbation may be inherent in the cell, for example the cell may comefrom a genetically modified cell line or organism, for example a knock-out or knock-down modificationof a given gene. The perturbation may have been applied to the cell, for example by applying an agent, such as a drug molecule or other chemical, a virus, a protein or a bacteria, to the sample from which the cell comes, prior to capturing the sequence from the cell. The images from which the shapes are derived may be captured a certain time after application of the perturbation, that is whenthe effect of the perturbation on the cell is in steady state. Alternatively, the images may be capturedimmediately after applying the perturbation or a short time after applying the perturbation to capture atransient response to the perturbation. Images of the sequence may be captured at equally spaced 7P233173GBtime intervals over a time period sufficient to capture characteristic morpho-dynamics of the cell, for example over a cell cycle. In practice this will result in time-lapse video acquisition of the cell images, for example acquiring an image (or frame) every few minutes, for example every 4 or 5 minutes. Asequence 300 of 3D images 310 and a corresponding sequence 320 or cell shapes 330 are illustrated,respectively, in Figures 3A and 3B.

[0028] Subsequent steps 220, 230 and 240, respectively, generate shape features characterisingeach captured shape, generate respective shape sequence features for shape features in thesequence, conditioned on at least part of the sequence and predict a related perturbation, related tothe perturbation to which the cell is subject, based on the shape sequence features, as discussed inmore detail below. Shape feature generation

[0029] At a step 220, shape features characterising the cell shapes captured in the images aregenerated. Any suitable way of generating informative shape features that are characteristic of cell shapes may be used. In some implementations, as illustrated in Figure 4A, the features are captured as a feature vector, for example generated by an encoder network 410 that was jointly trained with a decoder network 420 of an autoencoder 400 to recreate a cell shape input to the encoder at the output of the decoder network. For example, the autoencoder may be trained with a loss function based on a chamfer distance between the in and outputs of the autoencoder 400. The autoencoder has abottleneck layer as the last layer of the encoder network 410, providing the input for the decodernetwork 420. The activations of the units in the bottleneck layer in response to a cell shape presentedat the input of the encoder network 410, that is the output vector of the encoder network 410 is usedas the shape feature in some embodiments, referred to herein also as an embedding 430.

[0030] The encoder network 410 may in some implementations be fine tuned using a contrastiveloss function 440 to increase the similarity of pairs 450 of embeddings 430 that have been generatedby a pair of respective shapes 460, as illustrated in Figure 4B. One of the shapes of the pairs is a cellshape and the other is a synthetic shape generated from the cell shape of the pair by a transformationthat is physiologically plausible, such as rotation, jitter or shear. The embeddings 430 of each pair 450 may be passed through a further function, for example a Multi-Layer Perceptron before the lossfunction is calculated. See for example

[0033] for more details of this technique. In some implementationthe cell shape is defined in terms of a point cloud and the encoder network 410 may be any suitable network that can handle point cloud inputs, for example a Dynamic Graph Convolutional NeuralNetwork (DGCNN, see

[0028] for example. The DGCNN encoder constructs a directed k-nearestneighbours graph on the point cloud and passes this graph through a series of blocks. Each blockdefines edge features as a non-linear mapping of connected points in the input point cloud and thenapplies a channel-wise symmetric operation (such as summation or max pooling) on the edge featuresassociated with all edges from each vertex in the input graph. Other encoder networks may besuitable for other shape representations. The decoder network may be any suitable network capableof generating point clouds from the embedding 430, for example the folding-based decoder ofFoldingNet, see

[0031] for further details. The folding-based decoder takes the embedding 430 as an8 P233173GBinput and concatenates it to form a 2D grid of points, which is then fed through two three-layeredperceptrons to produce a point cloud.

[0031] The encoder network 410 is trained on a data set of cell shapes and, in someimplementations, is then fixed and used as a fixed feature generator. In one particular example, the encoder network 410 is trained as described above using a data set of 900003D images of melanomacancer cells using an ADAM optimiser

[0033] with a base learning rate of 0.0001 and a weight decay of0.0001 and a reduce on learning plateau scheduler, with a batch size of 16 and a maximum of 500 epochs with early stopping and patience of 10 epochs based on the validation loss.

[0032] The autoencoder may have any suitable composition and may for example be configured asthe original FoldingNet or as the PointNet

[0035] autoencoders. Alternatively, mesh-based autoencoders could be used [36-39]. Embeddings 430 could be calculated using autoencoders operating on the 3D images, for example in voxel format, using for example 3D convolution-based representation learning models. Other examples of convolution-based representation learning models include, for exampleResNet

[0045] . Instead of embeddings 430, features may be constructed in any other suitable way, forexample manually designed features based on shape geometry, such as volume, spreading andprotrusivity. The point-cloud or mesh representation of the shape may be used as the characteristic feature in further processing by a sequence to sequence neural network, as discussed below, usingany suitable sequence to sequence network that can handle sequences of shapes, for examplesequences of point clouds as an input. See for example

[0040] . The description below of certain neuralnetwork implementations is made for implementations using the embedding 430 but it will beappreciated that any other shape feature characteristic of cell shape may be used in variants of the described implementations. Additional feature information

[0033] In some implementations, the shape feature may comprise further information, for examplefurther embeddings concatenated with the embedding 430 generated for the cell shape. The presentdisclosure will be understood to include the use of shape features comprising such further information, for example the embedding 430 may be a concatenation of embeddings characteristic of shape and embeddings of further information. Reference to the embedding 430 should therefore be read as reference to a shape-only embedding 430 as described above and embeddings 430 representing any other shape features with or without additional information.

[0034] Additional information may, in some implementations, include an embedding of the shape ofthe nucleus of the cell. The embedding of the shape of the nucleus is in some implementations generated in the same way as the embedding for the cell shape and the nucleus shape embeddingand cell shape embedding are, in some implementations, combined, for example concatenated toform the embedding 430 or, more generally, the shape feature.

[0035] In some implementations, the cell shape embedding (and in some implementations the cellshape embedding and nucleus shape embedding) may further be combined with additional information, for example by concatenation with further embeddings. The additional information mayinclude an indicator, or an embedding of the indicator, of a stage in the cell cycle of the cell inquestion, in some implementations. Examples of additional information that may be included 9P233173GBaccording to the implementation include: fluorescent intensity of the cell; fluorescent colour of the cell;a ratio of component colours of the fluorescent colour of the cell; a ratio of fluorescent intensitiesbetween different regions of the cell; and so forth. In a specific example, a ratio of fluorescent intensityof ERK-KTR across the nucleus to fluorescent intensity across a region within a ring around thenucleus, or an embedding thereof, may be used as an indicator of the cell signalling dynamics

[0046] . In some embodiments, PCNA may be used as an indicator of the cell cycle stage.

[0036] In the present description, the term “shape feature” therefore includes a featurecharacterising cell shape, like the embedding 430 described above, as well as a feature characterising cell shape and additional information like cell nucleus shape and one or more indicators of cell cycle stage, for example in the form of a concatenation of respective embeddings. Where specific implementations are described below for the example of embedding 430 being used as the shape feature input to a neural network, it will be understood that the disclosure is not so limited and includesany shape feature characteristic of cell shape, for example differently derived shape features asdescribed above or shape features including additional information and / or concatenation of differentembeddings in place of embedding 430.Shape sequence feature generation

[0037] The sequence of shape features generated at step 220 are input to a sequence to sequenceneural network to generate shape sequence features at step 230, with each shape sequence featurebeing conditioned on at least a portion of the sequence. More specifically, each shape sequence feature is generated in dependence on the corresponding shape feature in the sequence and at leastone other shape feature in the sequence, for example an adjacent shape feature such as thepreceding or succeeding shape feature, preceding or succeeding the corresponding feature, for example coming immediately before or after in the sequence.

[0038] Various sequence to sequence models can be used in different implementations and therespective representation of the shape features. Feature vector or embedding shape features, likeembeddings 430, for example, can be handled by a variety of sequence to sequence models, forexample transformers or models including transformer layers in conjunction with position encodings,recurrent neural networks like LSTM or other types of recurrent neural networks or network layers, allof which can handle variable sequence lengths, or convolutional neural networks adapted to handlesequence information, such as InceptionTime

[0047] , all be it with a fixed sequence length.

[0039] In some implementations, the sequence to sequence model is a graph transformer inconjunction with a graph capturing the time dependence of shape features at adjacent time points. One such model is disclosed in

[0026] , where further details can be found, and is adapted here for the disclosed use case.

[0040] The shape dynamics of each cell i can be represented as a sequence ^^^ ∈ ℝ^^×^, such thatwith length ^^^ > 1, where each time point ^^^^is a ^^-dimensional vectorcorresponding to the shape feature characteristic of the shape of cell i at time step ^^ and ^^ is thenumber of variables in the shape feature (the length of the embedding 430, for example). ^^ may be256 in some implementations or may be smaller or larger than that. A typical value for ^^ may be anynumber in the range from 1 to 2048, for example. 10 P233173GB

[0041] Each time series is represented as an attribute graph, ^^^ = (^^^^, ^^^^), consisting of anadjacency matrix ^^^ ∈ ℝ^^×^^ and a node-feature matrix ^^^ ∈ ℝ^^×^. The adjacency matrix representsthe graph topology and can be characterised by the set of nodes ^^ =… , ^^^}, and edges ℰ =^^^^,^ : = ^^^^ , ^^^^ ∈ ^^ × ^^ ∣ A^,^ ≠ 0^, such that A^,^ is the (^^, ^^)-th element in ^^^. The node feature matrix^^^ contains attributes for each node, where the ^^-th row of ^^^, ^^^ ∈ ℝ^, represents the ^^-dimensional feature vector of node ^^^.

[0042] The graph comprises temporal edges ℰ that are inherently directional and designed tocapture the temporal progression from one cellular state to the next. They connect each time point ^^ tothe next one ^^ + 1, such that each edge ^^^,^^^ ∈ ℰ represents a temporal connection from time point ^^to time point ^^ + 1, for all ^^ ∈ {1,2, … , ^^^ − 1}. Thus, the set of temporal edges can be defined as ℰ ={(^^, ^^ + 1) ^^ < ^^^}.

[0043] In some implementations, the graph further comprises similarity-based edges aimed toencapsulate the relationship between different time segments based on their feature representations.For a given time series ^^^ = {^^^^ , ^^^^, … , ^^^^^}, representing different cell shapes over distinct points intime, a similarity matrix ^^ is constructed in these implementations, where each element ^^^^ denotesthe cosine similarity (or other similarity measure) between nodes ^^ and ^^ (where ^^ ≠ ^^ therebyexcluding self-loops). A threshold ^^ is applied and an edge ^^^^ is added to the graph if ^^^^ > ^^. Theweight of each edge ^^^^ added in this way is set to the similarity value ^^^^. The final adjacency matrixin these implementations, ^^^^, is constructed ^^^,^^^ = 1 for all ^^ < ^^^, and ^^^,^ = ^^^^ for all ^^, ^^ ∈

[0044] The graph constructed as discussed above is applied to a graph transformer 510 asillustrated in Figure 5, illustrating the shape features or embedding 430 as nodes connected by temporal edges 514 and some similarity-based edges 515. The graph transformer generates asequence of shape sequence features 518 (that are each dependent on at least part of the sequence),as described in detail below.

[0045] The graph transformer 510 comprises two graph transformer layers as described in

[0026] . Afirst graph transformer layer is provided with an input of a graph with embeddings 430 as nodes, andtemporal edges 514, and in some implementations also similarity edges 515). Considering a singleinstance of a cell and hence dropping the index from ^^^ for simplicity, the time series of shape features^^ = {^^^, ^^^, … , ^^^} is input to the first graph transformer layer, which calculates multi-head attention foreach edge from ^^ to ^^ as described in

[0026] :where ^^^, ^^^is the exponential dot product and ^^ is the hidden size, the size of theintermediate layer in the neural network of each attention head, as indexed by ^^. For the ^^-th attentionhead, source feature vector ^^^ and distant feature vector or vectors ^^^, ^^ ∈ ^^(^^) , the set of indicesof nodes connected by an edge (A^,^ ≠ 0), are transformed into a query matrix ^^^,^ ∈ ℝ^×^and key 11 P233173GBmatrix ^^^,^ ∈ ℝ^×^using trainable weights ^^^,^, ^^^,^and biases ^^^,^, ^^^,^. The edge features ^^^,^^=^^^,^^^^^ + ^^^,^ are encoded by ^^^,^ and ^^^,^ , with ^^^^ = A^,^, and are added to the key matrix. Thekey and query matrices are used to calculate attention weights ^^^,^^. A value matrix is computed as^^^,^ = ^^^,^^^^ + ^^^,^ and an intermediate output of the first transformer layer is computed aswhere ∥ is the concatenation operation on the outputs of the ^^ attention heads with the result theconcatenation of the head outputs. The actual output ^^^^^^ of the first transformer layer is generatedusing a gated residual connection from the input to the first graph transformer layer:^^^ = ^^^^^^ + ^^^^^^ = sigmoid^^^^[^^^^^^^; ^^^; ^^^^^^^ − ^^^]^with sigmoid being the sigmoid function and LayerNorm normalises the elements of its vectorargument, for example by determining the mean and standard deviation of the elements and subtracting the mean and scaling the result by the standard deviation for each element.

[0046] The graph transformer 510 comprises a second graph transformer layer that is the same asthe first graph transformer layer and takes as an input the output ^^^^^^ for each time step n of the timeseries ^^ as nodes of the input graph to the second layer, with edges defined by the same adjacencymatrix as the input graph and hence the same edge features ^^^^ = A^,^.The second graphtransformer layer than outputs the shape sequence features 516 as its output ^^^^^^. In this way, in thesequence of hidden outputs output by the first graph transformer layer depends on the respective embedding 430 and the adjacent embedding 430 in the input sequence and each output in thesequence of outputs of the second graph transformer layer depends on the respective hidden outputin the sequence of hidden outputs and the adjacent hidden output. Since the adjacent hidden outputdepends on its respective embedding 430 and the embedding 430 adjacent to that, each output in thesequence of outputs of the graph transformer in this implantation depends on the respectiveembedding 430 in the input sequence and the two adjacent embeddings 430 in the input sequence.

[0047] Each shape sequence feature consequently comprises information from its correspondingshape feature and, due to the two graph transformer layers, from the two shape features next in the time series connected to the corresponding shape feature by a temporal edge at, respectively, the firstand second graph transformer layers. It will be understood that, equivalently, the two shape featuresprevious in the time series can be taken into account by constructing the temporal edges as (t-1,t).Related perturbation prediction and relevance computations

[0048] At step 240, the output of the sequence to sequence model, for example the graphtransformer described above, is processed to predict a related perturbation using a classifier neuralnetwork 520. The predicted related perturbation, or a suitable identification thereof, may then bestored, presented, for example displayed, to a user or output in any other suitable way.

[0049] More specifically, in some implementations, the classifier neural network 520 generates botha prediction vector 522 for the related perturbation based on the whole sequence of shapes and a set12 P233173GBof relevancy vectors 524, each relevancy score vector 524 indicating how relevant each shapesequence feature 516 and hence each shape feature 430 and corresponding cell shape is for makingthe predictions in prediction vector 522. In some implementations he classifier neural network 520does so by combining the outputs of an attention head 526 and of a classification head 528. In some implementations, the classification head may comprise a linear layer and the attention head may comprise a perceptron layer.

[0050] With reference to Figure 6A, a method of generating both a prediction of a relatedperturbation and an indication of which shapes of the sequence are relevant or contributed to thisprediction comprises a step 610 of generating an attention score vector for each shape sequencefeature using the attention head 526 and a step 620 of generating a classification score vector foreach shape sequence feature using the classification head 528. The classifier 520 may be configuredas a binary classifier, in which case the classification score vector has a single element, a singlescalar, and a separate model may be used to obtain a prediction score for each possible relatedperturbation to be predicted. In some implementations, the classifier 520 may be configured as amulti-class classifier, in which case the classification score vector will have a number of elementscorresponding to the possible related perturbations that can be predicted, with an optional additionalclass for an unknown perturbation. The prediction and relevancy vectors 522, 524 will have acorresponding number of elements in each implementation.

[0051] At step 630, the classification score vectors are used to generate the prediction vector 522and the element of the prediction vector 522 with the largest value can be used to indicate thecorresponding related perturbation as the overall prediction, for example. At step 640, theclassification score vector and attention score are used to generate relevancy vectors 524, one foreach shape sequence feature. At step 650, the prediction and relevancy vectors are used to findshape sequence features, and hence cell shapes in the sequence that correspond, that are relevant tothe prediction and hence can explain it. The prediction vector 522 indicates for each relatedperturbation corresponding to the respective element in the prediction vector 522 a likelihood for each related perturbation being predicted as related to the perturbation the cell has been subject to. The relevancy vectors 524 indicate for their respective shape sequence feature the contribution orrelevancy of that shape sequence feature to the prediction, wherein again each element indicates therelevancy of that shape sequence feature to the prediction of the corresponding related perturbation. In the multiclass case, an overall prediction of a related perturbation can be formed by selecting the related perturbation with the highest likelihood in the prediction vector 522 and the corresponding element of the relevancy vectors 524 indicates the relevancy of the respective shape sequencefeatures to the overall prediction of a related perturbation.

[0052] With reference to Figure 6B, in some implementations, the attention score vectors andclassification score vectors are generated independently of each other, for example in parallel, atsteps 612 and 622, respectively. For example, the attention head 626 and classification head 628generate their respective outputs independently of each other, for example in parallel. The classification and attention score vectors are then combined at step 660 to generate the relevancyvectors 524 and the prediction vector 522 for the sequence.13 P233173GB

[0053] A specific implementation of the method in Figure 6b with the attention and classificationheads 526, 528 operating independently and optionally in parallel is now described. The graphtransformer’s output, the shape sequence features ^^^^^^, is directed into a conjunctive pooling modulewith independent attention and classification heads 526, 528, as described in

[0012] . This will bedescribed for an example of shape sequence feature vectors with 256 elements and a single relatedperturbation the absence or presence of which is to be predicted in a binary classification problem, sothat the attention score, classification score, relevancy and prediction vectors each have a singleelement. Shape vectors of any suitable different sizes can of course be used instead of size 256. The following description is readily generalised to multi-class problems by increasing the dimensionality ofthe vectors to match the number classes (related perturbation and possible null class), for example asdescribed above.

[0054] The classification head consists of a linear layer that generates the classification scorevector for each time step as instance logits ^^^^ = ^^^^^^^^^ + ^^^, where for the binary classification case^^^ ∈ ℝ^×^^^ is a trained weight matrix with a corresponding bias vector, ^^^ ∈ ℝ^, and ^^^^^^is the shape sequence feature at time step n, for example as output by the second graph transformer layer. If desired, the logits can be converted to probabilities for presentation using the sigmoid (for binary classification) or softmax function (for multi-class classification).

[0055] The attention head is responsible for generating an attention score for each time point of thetime series. The attention score for each time point, ^^, is computed according to the following:^^^ = sigmoid(^^^ ⋅ tanh(^^^^^^^^^ + ^^^) + b^),∈ ℝ^×^^^ and ^^^ ∈ ℝ^×^ are trained weight matrices with corresponding bias vectors, ^^^ ∈ℝ^ and b^ ∈ ℝ^, and tanh(⋅) is the hyperbolic tangent function.

[0056] The attention scores and classification score vectors are combined to generate therelevance vectors and prediction for the sequence at step 660. The final prediction output ^^^, theprediction vector 522, is produced by scaling the instance logits ^^^^ using the attention scores a^:. As required, the logits ^^^ can be converted to probabilities using the sigmoid (binary classification) orsoftmax (multi-class) function.

[0057] The relevance vectors 524 are each computed as ^^^ ⋅ ^^^^ for each time step n and againreduces to a 1D vector or scalar in the binary classification case. The overall prediction can be derivedas the likely relevance of the related perturbation in the binary (say the related perturbation ispredicted to be relevant if ^^^ or sigmoid(^^^) exceeds a threshold, say 0 or 0.5, respectively) or the mostlikely related perturbation corresponding to the largest element of ^^^ is returned as the prediction. Thecell shapes / time points can then be returned as those corresponding to the time step n for which thecorresponding element of the relevance vector 524, in this specific implementation ^^^ ⋅ ^^^^, has thelargest value(s), for example values exceeding a relevance threshold.

[0058] With reference to Figure 6C, in an implementation that is a possible alternative for themethod of Figure 6B, attention scores and classification score vectors ^^^, ^^^^ are computed insequence rather than in parallel and the computation of the classification score vectors is dependent 14 P233173GBon the attention score vectors. Specifically, at step 614, the attention score vectors ^^^ are computed,for example as described above. At step 624, the classification score vectors ^^^^are computed usingthe attention score vectors, for example as ^^^^ = ^^^(^^^ ⋅ ^^^^^^ + ^^^), where ^^^and ^^^ have the samemeaning as described above. At step 670, the classification score vectors are then combined to form the overall prediction vector 522 as:and the respective relevance vector at each time step n is taken to be the classification score vectors^^^^ or a normalised version of the classification score vectors, for example softmax(^^^^) for the multi-class case and sigmoid(^^^^) for the binary classification case, normalising the classification scorevector at each time step across classes / vector elements (that is across the output layer of thenetwork layer generating ^^^^). Multi-binary models

[0059] As described above, the disclosed implementations can handle multi-class classification,predicting classification scores or probabilities simultaneously for several related perturbations on which the neural networks were trained (and a possible NULL class to allow for the possibility that none of the related perturbations in the training data are good matches) by outputting a classification score / probability vector. The disclosed implementations can equally handle binary classification withthe same structure apart from the output of the classifier neural network, which in that case is a singleclassification score or probability indicating a likelihood that a single possible related perturbation on which the neural networks were trained is related or not.

[0060] In some implementations, a panel of several neural networks, each with a binary classifierneural network may be used to make predictions for multiple possible related perturbations for whichtraining data is available by outputting a corresponding one-dimensional prediction vector 522 (orprediction score) for each related perturbation, in effect providing a separate combined sequence tosequence and classifier neural network head for each related perturbation. To obtain classification scores that are normalised across combined sequence to sequence and classifier neural network heads, in some implementations, the respective classification scores can be jointly passed through a softmax layer to output a scores or probabilities for the related perturbations that sum to unity. Amodel with multiple separate combined sequence to sequence and binary classifier heads is alsoreferred a multi-binary model herein.

[0061] While multi-class models have the advantage that they can be jointly trained for all theavailable training data across the perturbations in the training data, adding new related perturbation tosuch a model may be problematic in that it may involve re-training the multi-class model with the newperturbation included. On the other hand, the binary models of a multi-binary model are trainedindependently of each other for each perturbation, so that a new perturbation can readily be added by training a new corresponding binary model to be added to the set of pre-existing binary models previously trained on other perturbations. In this description, unless the context dictates otherwise,either multi-class models or multi-binary models may be used, and the terms can be readinterchangeably. 15 P233173GBBiological pathway prediction

[0062] The identification of cell shapes relevant to predicting related perturbation can, in someimplementations, be used to identify potentially relevant biological pathways. In a particularimplementation, described with reference to Figures 7A and 7B, this includes a step 710 of identifyingcell shapes relevant to the prediction of the related perturbation(s) by way of identifying corresponding shape features and / or shape sequence features with high relevancy, for example exceeding a threshold value, as described above. At step 720, a database 740 is queried with the identified cell shape(s). For example, the database 740 may store shape features 430 corresponding to cell shapes that have previously been found to be associated with corresponding biological pathways. Each storedshape feature 430 is associated with a corresponding biological pathway record 742 in the data base.The identified cell shapes are used to query the database 740 and biological pathways records corresponding to matching shape features, for example as determined by a similarity measure between the identified and stored cell features exceeding a similarity threshold, are retrieved. The retrieved biological pathway records 742 are returned at step 730, for example displayed to a user, stored or output in any other way.

[0063] In some specific implementations, the sequence to sequence network and classifier networkare trained, as described below, using cells from genetically modified cell-lines or organisms, in which certain genes have been knocked-out or -down, so that the resulting cells lack or are down-regulated in respect of the biological pathway corresponding to the gene. The resulting network can then be used to predict, for a given perturbation, for example a given applied agent or drug, applied to cells from which input data is derived, a related gene knock-out or -down, and, hence, a corresponding biological pathway that is blocked or down-regulated. This suggests the corresponding biological pathway as affected (blocked or downregulated) by the applied perturbation and the corresponding biological pathway can therefore be predicted as a related / affected biological pathway. Model training

[0064] The sequence to sequence neural network 510 and classifier neural network 520 are, insome implementation jointly trained using a process that includes a first step 810 of obtaining atraining data set. The training dataset comprises training data pairs, each comprising a sequence of shape features, obtained for example as described above at step 220 and with reference to Figures4A and 4B, from cells that are the subject of respective perturbations, for example a geneticmodification or the application of a chemical such as a drug, and a corresponding label identifying the respective perturbation for each sequence. The method comprises generating a sequence of shape sequence features 516 as described above for the sequence of each training data pair at step 820 and generating a classification score 520 from the sequence of sequence shape features, as describedabove, at step 830. A loss function is computed by comparing the classification score for the sequenceof each training pair to the corresponding label of the training pair, at step 840. The loss function may be a log probability or (binary or multi-class, as the case may be) cross-entropy function and may bedirectly computed on the logit output of the classifier network or the logit output of the classifiernetwork may first be transformed to a probability output (using a sigmoid or softmax layer, as the case may be). At step 850, the parameters of the sequence to sequence and classifier neural networks are 16 P233173GBadjusted to reduce the loss function. The process is repeated in batches or otherwise until a stoppingcriterion is met and training is completed.

[0065] To generate the data sets, in a specific implementation, shape features are extracted asdescribed above for each sequence and normalised using z-score normalisation across the data set.Training is carried out in an example implementation with 10-fold cross-validation and a 60 / 20 / 20 splitfor training / validation / test data sets. Parameters are adjusted with a base learning rate of 0.001 and aweight decay of 0.0001 and a reduce on plateau learning rate scheduler. Batch size for thisimplementation is 1 and each fold is trained for a maximum of 500 epochs with early stopping with apatience of 10 based on the validation loss. The neural networks were trained on a single NvidiaQuadro RTX 6000 GPU with 24GB of memory. The graph transformer neural network has 184,192 parameters and the classifier neural network has 2,322 parameters in a specific implementation. In this specific implementation, training on a dataset of 90000 sequences took 4.91 seconds per epochon average, with an average inference time of 3.19 seconds. The threshold parameter ^^ for the graphconstruction described above was tuned by evaluating different values on the validation data set andfor different data sets the best value was found to range from 0.7 to 1 (in effect not including similarityconnections when the threshold is set to unity).Example cell preparation and data set

[0066] Melanoma cancer cells were prepared and imaged in 96-well plates. Collagen Type 1 (fromrat tail) was prepared at a density of ~2 mg / ml. Cells were suspended in collagen, and 100 µL of the collagen-cell mixture was dispensed into each well. The target cell density was between 40,000 and 60,000 cells per well. Plates were incubated at 37 degrees for approximately 30 minutes to form a hydrogel. Cell culture media was added, and collagen-embedded cells were cultured overnight.

[0067] Treatments were applied as the perturbations. The treatments (drug agents) were: CK666(100 µM), Palbociclib (1 µM), PF228 (2 µM), Blebbisatin (10 µM). Concentrations were calculated, including the 100 µL of the collagen-cell mixture. Treatments were applied approximately 5 hours before imaging.

[0068] Cell membranes were marked by GFP-CAAX expression, and cell nuclei were marked bysir-DNA (200 nM). Images were acquired at approximately 4-6 minute intervals using oblique LSFM.This dataset consisted of 442 sequences of 3D cell images for respective cells ranging from a timeseries length of 5 to 116. Four neural network models with a binary classification output were trained,one for each applied drug treatment with the data for each model provided with a binary label indicating the respective drug vs the remaining drugs, with no untreated cells in the data sets.

[0069] Classification results on the test data sets were averaged across the four drugs and themodel was found to outperform a number of known models, specifically a LSTM model

[0048] , TransMIL

[0025] and GTP

[0032] .

[0070] Best in class or at least comparable performance was also found in an experiment trainingthe model on a classification task with a data set of 869 point cloud sequences ranging from 5 to 75time steps per sequence and coming from three different types of T cells, validating the choice of apre-trained and fixed shape feature generator (trained on Melanoma cancer cells).17 P233173GBSynthetic data example

[0071] To evaluate the ability of the disclosed models to enable high-quality interpretations by virtueof the relevancy scores, we created a synthetic dataset of 3D shapes morphing into other shapes.This dataset consists of three classes of sequences defined by their shape changes. Each time pointis a point cloud representation of a 3D shape, and each time series varies in length (between 15 and70 time points). Spheres with radii between 0.8 and 1.2 were generated and one of three shapes(torus, cube, cylinder) were injected at various time points, interpolating between the sphere and thetarget shape and back to a sphere. This interpolation happened over varying lengths in the dataset.The whole transition sequence is labelled as a class-specific instance (torus, cube, cylinder). Theimportant instances of each frame in the time series for classification are therefore known as theframes with the transition being maximal, enabling direct evaluation of the interpretability of the model.The dataset consisted of 1000 point cloud sequences., broken down as 324 torus, 354 cube, 322 cylinder sequences.

[0072] The interpretability performance based on relevance scores was compared againstTransMIL and GTP using processes described in

[0013] using two metrics: Area Over the Perturbation Curve to Random and Normalised Discounted Cumulative Gain at n. On both these measures, the model outperformed TransMIL and GTP. Information content of dynamic aspect of shape sequence data

[0073] To illustrate that the dynamics of cell shape in the sequence of shapes that are beinganalysed contains additional information that is lost when analysing static images of cells, an experiment was run in which the performance of the described model was compared for a first data set of cell shape sequences as described above and a second, jumbled data set, in which the framesof the same sequences were taken out of sequence in a random order so that the jumbled data set ismissing the sequence information of the first data set. It was found that the performance of the modelto predict related perturbations was better for the first data set as compared to the jumbled data set,indicating that the model uses the sequence information to good effect to make predictions. Computing device infrastructure

[0074] Figure 9 illustrates a block diagram of one implementation of a computing device 900 withinwhich a set of instructions, for causing the computing device to perform any one or more of the methodologies discussed herein, may be executed. In alternative implementations, the computing device may be connected (e.g., networked) to other machines in a Local Area Network (LAN), an intranet, an extranet, or the Internet. The computing device may operate in the capacity of a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The computing device may be a personal computer (PC), a tabletcomputer, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a webappliance, a server, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single computing device is illustrated, the term “computing device” shall also be taken to include any collection of machines (e.g., computers) that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein. 18 P233173GB

[0075] The example computing device 900 includes a processing device 902, a main memory 904(e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), a static memory 906 (e.g., flash memory, static random access memory (SRAM), etc.), and a secondary memory (e.g., a data storage device 918), which communicate with each other via a bus 930.

[0076] Processing device 902 represents one or more general-purpose processors such as amicroprocessor, central processing unit, or the like. More particularly, the processing device 902 may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processing device 902 may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. Processing device 902 is configured to execute the processing logic (instructions 922) for performing the operations and steps discussed herein.

[0077] The computing device 900 may further include a network interface device 908. Thecomputing device 900 also may include a video display unit 910 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 912 (e.g., a keyboard or touchscreen), a cursor control device 914 (e.g., a mouse or touchscreen), and an audio device 916 (e.g., a speaker).

[0078] The data storage device 918 may include one or more machine-readable storage media (ormore specifically one or more non-transitory computer-readable storage media) 928 on which is stored one or more sets of instructions 922 embodying any one or more of the methodologies or functions described herein. The instructions 922 may also reside, completely or at least partially, within the main memory 904 and / or within the processing device 902 during execution thereof by the computer system 900, the main memory 904 and the processing device 902 also constituting computer-readable storage media.

[0079] The various methods described above may be implemented by a computer program. Thecomputer program may include computer code arranged to instruct a computer to perform the functions of one or more of the various methods described above. The computer program and / or the code for performing such methods may be provided to an apparatus, such as a computer, on one or more computer readable media or, more generally, a computer program product. The computer readable media may be transitory or non-transitory. The one or more computer readable media could be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or a propagation medium for data transmission, for example for downloading the code over the Internet. Alternatively, the one or more computer readable media could take the form of one or more physical computer readable media such as semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disc, and an optical disk, such as a CD-ROM, CD-R / W or DVD.

[0080] In an implementation, the modules, components and other features described herein can beimplemented as discrete components or integrated in the functionality of hardware components such as ASICS, FPGAs, DSPs or similar devices. 19 P233173GB

[0081] A “hardware component” is a tangible (e.g., non-transitory) physical component (e.g., a setof one or more processors) capable of performing certain operations and may be configured or arranged in a certain physical manner. A hardware component may include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component may be or include a special-purpose processor, such as a field programmable gate array (FPGA) or an ASIC. A hardware component may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations.

[0082] Accordingly, the phrase “hardware component” should be understood to encompass atangible entity that may be physically constructed, permanently configured (e.g., hardwired), ortemporarily configured (e.g., programmed) to operate in a certain manner or to perform certainoperations described herein.

[0083] In addition, the modules and components can be implemented as firmware or functionalcircuitry within hardware devices. Further, the modules and components can be implemented in any combination of hardware devices and software components, or only in software (e.g., code stored or otherwise embodied in a machine-readable medium or in a transmission medium).

[0084] Unless specifically stated otherwise, as apparent from the following discussion, it isappreciated that throughout the description, discussions utilizing terms such as " receiving”,“determining”, “comparing ”, “enabling”, “maintaining,” “identifying”, “generating” or the like, refer to theactions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices. REFERENCES

[0085] The following documents provide background information. Some of these documents arereferred to in the above disclosure using square brackets and these document are incorporated herein by reference: 1. Bagnall, A., Dau, H.A., Lines, J., Flynn, M., Large, J., Bostrom, A., Southam, P., Keogh, E.: Theuea multivariate time series classification archive, 2018 (2018) 2. Bakal, C., Aach, J., Church, G., Perrimon, N.: Quantitative morphological signa- tures define localsignaling networks regulating cell morphology. Science (2007) 3. Barcelo, J., Samain, R., Sanz-Moreno, V.: Preclinical to clinical utility of rock inhibitors in cancer.Trends in Cancer (2023)` 4. Bier, A., Jastrzębska, A., Olszewski, P.: Variable-length multivariate time series classificationusing rocket: A case study of incident detection. IEEE Access (2022) 5. Castillo-Badillo, J.A., Gautam, N.: An optogenetic model reveals cell shape regulation through FAK and fascin. Journal of Cell Science (2021) 6. Chandrasekaran, S.N., Ceulemans, H., Boyd, J.D., Carpenter, A.E.: Image-based profiling fordrug discovery: due for a machine-learning upgrade? Nature Reviews Drug Discovery (2021)7. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.E.: A simple framework for contrastive learning of visual representations. CoRR (2020) 20 P233173GBCooper, S., Sadok, A., Bousgouni, V., Bakal, C.: Apolar and polar transitions drive the conversionbetween amoeboid and mesenchymal shapes in melanoma cells. Molecular Biology of the Cell(2015) Copperman, J., Gross, S.M., Chang, Y.H., Heiser, L.M., Zuckerman, D.M.: Morphodynamical cellstate description via live-cell imaging trajectory embedding. Communications Biology (2023)De Vries, M., Dent, L.G., Curry, N., Rowe-Brown, L., Tyson, A., Dunsby, C., Bakal, C.: 3d single-cell shape analysis of cancer cells using geometric deep learning. In: NeurIPS 2022 Workshop onLearning Meaningful Representations of Life (2022)Dent, L.G., Curry, N., Sparks, H., Bousgouni, V., Maioli, V., Kumar, S., Munro, I., Butera, F.,Jones, I., Arias-Garcia, M., Rowe-Brown, L., Dunsby, C., Bakal, C.: Environmentally dependentand independent control of 3d cell shape. Cell Reports (2024)Early, J., Cheung, G., Cutajar, K., Xie, H., Kandola, J., Twomey, N.: Inherently interpretable timeseries classification via multiple instance learning. In: The Twelfth International Conference onLearning Representations (2024)Early, J., Evers, C., Ramchurn, S.: Model agnostic interpretability for multiple instance learning.In: International Conference on Learning Representations (2022)Fourkioti, O., De Vries, M., Bakal, C.: CAMIL: Context-aware multiple instance learning forcancer detection and subtyping in whole slide images. In: The Twelfth International Conferenceon Learning Representations (2024)Gordonov, S., Hwang, M.K., Wells, A., Gertler, F.B., Lauffenburger, D.A., Bathe, M.: Time seriesmodeling of live-cell shape dynamics for image-based phenotypic profiling. Integrative Biology(2015) Heck, T., Vargas, D.A., Smeets, B., Ramon, H., Van Liedekerke, P., Van Oosterwyck, H.: Therole of actin protrusion dynamics in cell migration through a degradable viscoelastic extracellularmatrix: Insights from a computational model. PLOS Computational Biology (2020)Heinemann, T., Kornauth, C., Severin, Y., Vladimer, G.I., Pemovska, T., Hadz- ijusufovic, E.,Agis, H., Krauth, M.T., Sperr, W.R., Valent, P., Jäger, U., Simonitsch-Klupp, I., Superti-Furga, G.,Staber, P.B., Snijder, B.: Deep Morphology Learning Enhances Ex Vivo Drug Profiling-BasedPrecision Medicine. Blood Cancer Discovery (2022)Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In:Proceedings of the 35th International Conference on Machine Learning. Proceedings of MachineLearning Research (2018)Jin, M., Koh, H.Y., Wen, Q., Zambon, D., Alippi, C., Webb, G.I., King, I., Pan, S.: A survey ongraph neural networks for time series: Forecasting, classification, imputation, and anomalydetection. arXiv (2023)Li, B., Li, Y., Eliceiri, K.W.: Dual-stream multiple instance learning network for whole slide imageclassification with self-supervised contrastive learning. In: Pro- ceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition (2021)Lines, J., Taylor, S., Bagnall, A.: Time series classification with hive-cote: The hierarchical votecollective of transformation-based ensembles. ACM Trans. Knowl. Discov. Data (2018)21 P233173GBMaioli, V., Chennell, G., Sparks, H., Lana, T., Kumar, S., Carling, D., Sardini, A., Dunsby, C.:Time-lapse 3-d measurements of a glucose biosensor in multicellular spheroids by light sheetfluorescence microscopy in commercial 96-well plates. Scientific Reports (2016)Medyukhina, A., Blickensdorf, M., Cseresnyés, Z., Ruef, N., Stein, J.V., Figge, M.T.: Dynamicspherical harmonics approach for shape classification of migrating cells. Scientific Reports (2020)Nolen, B.J., Tomasevic, N., Russell, A., Pierce, D.W., Jia, Z., McCormick, C.D., Hartman, J.,Sakowicz, R., Pollard, T.D.: Characterization of two classes of small molecule inhibitors of arp2 / 3complex. Nature (2009)Shao, Z., Bian, H., Chen, Y., Wang, Y., Zhang, J., Ji, X., Zhang, Y.: TransMIL: Transformerbased correlated multiple instance learning for whole slide image classification. In: Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems (2021) Shi, Y., Huang, Z., Feng, S., Zhong, H., Wang, W., Sun, Y.: Masked label prediction: Unifiedmessage passing model for semi-supervised classification. In: Zhou, Z.H. (ed.) Proceedings ofthe Thirtieth International Joint Conference on Artificial Intelligence (2021)Tan, C., Ginzberg, M.B., Webster, R., Iyengar, S., Liu, S., Papadopoli, D., Con- cannon, J.,Wang, Y., Auld, D.S., Jenkins, J.L., Rost, H., Topisirovic, I., Hilfinger, A., Derry, W.B., Patel, N.,Kafri, R.: Cell size homeostasis is maintained by cdk4- dependent activation of p38 mapk.Developmental Cell (2021)Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph cnn forlearning on point clouds. ACM Transactions on Graphics (TOG) (2019)Wu, P.H., Gilkes, D.M., Phillip, J.M., Narkar, A., Cheng, T.W.T., Marchand, J., Lee, M.H., Li, R.,Wirtz, D.: Single-cell morphology encodes metastatic potential. Science Advances (2020)Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J.: 3d shapenets: A deeprepresentation for volumetric shapes. In: 2015 IEEE Conference on Computer Vision and PatternRecognition (CVPR) (2015)Yang, Y., Feng, C., Shen, Y., Tian, D.: Foldingnet: Point cloud auto-encoder via deep griddeformation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)Zheng, Y., Gindra, R.H., Green, E.J., Burks, E.J., Betke, M., Beane, J.E., Ko- lachalama, V.B.: Agraph-transformer for whole slide image classification. IEEE Transactions on Medical Imaging(2022)Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Y., LeCun, Y.(eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA,USA, May 7-9, 2015, Conference Track Proceedings (2015) Viana, M.P., Chen, J., Knijnenburg, T.A. et al. Integrated intracellular organization and its variations in human iPS cells. Nature 613, 345–354 (2023). https: / / doi.org / 10.1038 / s41586-Qi, C. et al. “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation.”2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016): 77- 85.22 P233173GB36. Lei, Eric et al. “WrappingNet: Mesh Autoencoder via Deep Sphere Deformation.” ArXiv abs / 2308.15413 (2023): n. Pag. 37. Yuan, Y.-J.; Lai, Y.-K.; Yang, J.; Duan, Q.; Fu, H.; and Gao, L.2020. Mesh Variational Autoencoders with Edge Contraction Pooling. In 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1105–111238. Zhou, Y.; Wu, C.; Li, Z.; Cao, C.; Ye, Y.; Saragih, J.; Li, H.; and Sheikh, Y.20 20. Fully convolutional mesh autoencoder using efficient spatially varying kernels. Advances in Neural Information Processing Systems, 33: 9251–9262. 39. Hahner, S.; and Garcke, J.2022. Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes. In 2022 IEEE / CVF Winter Conference on Applications of Computer Vision(WACV), 2344–2353. 40. Z. Hang, Y. Wang and S. Huang, "P4 Transformer: Towards Unified Programming for the Data Plane of Software Defined Network," 2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC), Madrid, Spain, 2021, pp.544-551, doi: 10.1109 / COMPSAC51774.2021.00081. 41. Xinggang Wang, Yongluan Yan, Peng Tang, Xiang Bai, and Wenyu Liu. Revisiting multiple instance neural networks. Pattern Recognition, 74:15–24, 2018.42. Syed Ashar Javed, Dinkar Juyal, Harshith Padigela, Amaro Taylor-Weiner, Limin Yu, and Aaditya Prakash. Additive MIL: Intrinsically interpretable multiple instance learning for pathology.Advances in Neural In- formation Processing Systems, 35:20689–20702, 202243. Jiajun Wu*, Chengkai Zhang*, Tianfan Xue, William T. Freeman, and Joshua B. Tenenbaum. Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling NeurIPS 2016 44. Matt De Vries, Lucas Dent, Nathan Curry, Leo Rowe-Brown, Vicky Bousgouni, Adam Tyson, Christopher Dunsby, Chris Bakal. “3D single-cell shape analysis using geometric deep learning”. bioRxiv 2022.06.17.496550; doi: https: / / doi.org / 10.1101 / 2022.06.17.496550 45. C. S. Wickramasinghe, D. L. Marino and M. Manic, "ResNet Autoencoders for Unsupervised Feature Learning From High-Dimensional Data: Deep Models Resistant to Performance Degradation," in IEEE Access, vol.9, pp.40511-40520, 2021, doi: 10.1109 / ACCESS.2021.3064819.46. Sergi Regot, Jacob J. Hughey, Bryce T. Bajar, Silvia Carrasco, Markus W. Covert,High-Sensitivity Measurements of Multiple Kinase Activities in Live Single Cells, Cell, Volume 157, Issue 7, 2014, Pages 1724-1734, ISSN 0092-8674, https: / / doi.org / 10.1016 / j.cell.2014.04.039. 47. Ismail Fawaz, H., Lucas, B., Forestier, G. et al. InceptionTime: Finding AlexNet for time series classification. Data Min Knowl Disc 34, 1936–1962 (2020). https: / / doi.org / 10.1007 / s10618-020- 00710-y 48. Sepp Hochreiter and Jürgen Schmidhuber.1997. Long Short-Term Memory. Neural Comput.9, 8 (November 15, 1997), 1735–1780. https: / / doi.org / 10.1162 / neco.1997.9.8.1735 23 P233173GB49. Thomas Zerjatke, Igor A. Gak, Dilyana Kirova, Markus Fuhrmann, Katrin Daniel, Magdalena Gonciarz, Doris Müller, Ingmar Glauche, Jörg Mansfeld, Quantitative Cell Cycle Analysis Based on an Endogenous All-in-One Reporter for Cell Tracking and Classification, Cell Reports, Volume19, Issue 9, 2017, Pages 1953-1966, ISSN 2211-1247, https: / / doi.org / 10.1016 / j.celrep.2017.05.022. FINAL REMARKS

[0086] It is to be understood that the above description is intended to be illustrative, and notrestrictive. Many other implementations will be apparent to those of skill in the art upon reading and understanding the above description. Although the present disclosure has been described with reference to specific example implementations, it will be recognized that the disclosure is not limited tothe implementations described but can be practiced with modification and alteration within the spiritand scope of the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative sense rather than a restrictive sense. The scope of the disclosure should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. 24 P233173GB

Claims

CLAIMS1. A computer-implemented method of predicting for a cell subject to a perturbation a relatedperturbation, the method comprising: obtaining a sequence over a period of time of cell shapes of a cell subject to the perturbation; generating a sequence of shape features by generating a shape feature for each cell shape in the sequence, each shape feature characterising a respective cell shape; applying the sequence of shape features as an input to a sequence to sequence neural network to generate a sequence of shape sequence features, each sequence shape feature being conditioned on a respective shape feature and one or more other shape features in the sequence; applying the sequence of shape sequence features to a classifier neural network to generate aclassification score for each of one or more classes, each class corresponding to one or more related perturbations, wherein the classifier and sequence to sequence neural networks have been jointly trained to predict respective training perturbations from sequences of cell shapes of cells subject to the respective training perturbations; and predicting one or more related perturbations using the classification scores for the one or more classes.

2. The method of claim 1, wherein the cell shapes are three-dimensional cell shapes.

3. The method of claim 1 or 2, wherein the classifier neural network comprises an attention headand is configured to apply each sequence shape feature as an input to the attention head to generate as an output of the attention head an attention score for each sequence shape feature in the sequence and the classifier neural network is further configured to combine the shape sequencefeatures weighted by their respective attention scores to generate the classification score for eachclass.

4. The method of claim 3, wherein the classifier neural network comprises a classification headand is configured to apply each sequence shape feature as an input to the classification head to generate as an output of the classification head a shape classification score for each class of the one or more classes and each sequence shape feature in the sequence and the classifier neural network is further configured to combine the generated shape classification scores weighted by their respectiveattention score to generate the classification score, thereby combining the shape sequence featureweighted by their respective attention scores.

5. The method of claim 3, wherein the classifier model is configured to apply each sequenceshape feature weighted by its respective attention score as an input to a classification head of the classifier model to generate as an output of the classification head a shape classification score for each class of the one or more classes and each sequence shape feature in the sequence and theclassifier model is further configured to combine the generated shape classification scores to generate25 P233173GBthe classification score, thereby combining the shape sequence feature weighted by their respectiveattention scores.

6. The method of any one of claims 3 to 5, comprising computing a relevance score for eachshape feature in the sequence using the respective attention score and shape classification score and identifying shapes in the sequence for which the respective relevance score meets a relevancy condition as relevant to the prediction of the one or more related perturbations.

7. The method of claim 6, comprising matching each of the identified shapes against a databaseof database cell shapes associated with known biological pathways and identifying biological pathways associated with matched database cell shapes as potentially relevant for the perturbation to which the cell is subject to.

8. The method of any preceding claim, wherein the perturbation comprises one or more of:an agent, for example a drug molecule, antibody, virus or bacteria, applied to the cell;a genetic modification of the cell; andan environmental condition of the cell.

9. The method of any preceding claim, wherein the perturbation is applied to the cell at a point intime and the time period is a time period subsequent to the point in time, when a response of the cell to the perturbation is at steady state.

10. The method of any preceding claim, wherein the shape feature is additionally characteristic ofa shape of a nucleus of the respective cell.

11. The method of any preceding claim, wherein the shape features are generated using anencoder neural network of an autoencoder neural network trained on cell shapes.

12. The method of claim any one of claims 1 to 9, wherein the shape features are generated usingan encoder neural network of an autoencoder neural network trained on cell shapes and generating the shape features comprises: applying a shape of a nucleus of the cell to the encoder neural network to generate a first output; applying the cell shape of the cell to the encoder neural network to generate a second output; concatenating the first and second outputs.

13. The method of any preceding claim, wherein the cell has been imaged to generate a set ofstacked slice images at each time point of the sequence and each cell shape has been generated from the respective set of stacked slice images.

14. The method of any preceding claim, wherein the sequence to sequence neural networkcomprises one or more graph transformer layers.26 P233173GB15. The method of claim 14, wherein the sequence of shape features is represented by a graphhaving the shape features of the sequence as nodes and respective edges between pairs of adjacentnodes.

16. The method of claim 15, wherein the graph comprises edges between pairs of nodes for whicha similarity measure between the nodes of the pair exceeds a threshold.

17. The method of any preceding claims, wherein the classifier neural network is a binaryclassifier for generating a prediction score for the relevance or not of a single related perturbation and the sequence to sequence neural network and the binary classifier jointly define one of a plurality of separate binary classifier heads, each separate binary classifier head comprising a sequence to sequence neural network providing an input to a binary classifier neural network to generate acorresponding binary classification score and each separate binary classifier head having been trainedto predict a separate respective related perturbation.

18. The method of any one of claims 1 to 16, where the classifier neural network is a multi-classclassifier.

19. The method of any preceding claim wherein the perturbation comprises an agent and thetraining perturbation comprises a genetic modification.

20. The method of any preceding claim, wherein the shape feature comprises additionalinformation.

21. The method of claim 20, wherein the additional information comprises an indicator of a stagein a cell cycle of the cell.

22. The method of claim 20 or 21, wherein the additional information comprises one or more: of afluorescent intensity of the cell or a region of the cell, a fluorescent colour of the cell or a region of the cell and a ratio of fluorescent intensity in two respective regions of the cell.

23. The method of any one of claims 20 to 22, wherein the additional information is embedded ina numeric embedding and the numeric embedding is concatenated with a numeric embeddingcharacteristic of the shape of the cell.

24. A method of jointly training the sequence to sequence and classifier neural networks used inthe method of any preceding claim, the sequence to sequence and classifier networks each being defined by a corresponding set of parameters, the method comprising: obtaining a training data set comprising a plurality of training pairs, each training pair comprising a training sequence of shape features and a corresponding label identifying a respective training perturbation, wherein obtaining the training data set comprises obtaining a training sequence over a period of time of cell shapes of a cell subject to the respective training perturbation and 27 P233173GBgenerating the training sequence of shape features by generating a shape feature for each cell shape in the sequence, each shape feature characterising a respective cell shape; applying each training sequence of shape features as an input to a sequence to sequence neural network to generate a sequence of shape sequence features; applying the sequence of shape sequence feature to the classifier neural network to generatea classification score for each of one or more classes, each class corresponding to one or more perturbations; for each training sequence of shape features, comparing the generated classification scores to the respective label to compute a loss function; and adjusting the parameters of the sets of parameters to reduce the loss function.

25. One or more computer-readable media, or a computer program product, encoding codedinstructions that, when executed on a processor implement a method as claimed in any preceding claims.

26. A system comprising one or more computer processors and a memory, the memory encodingcoded instructions that, when executed on the one or more computer processors implement a method as claimed in any one of claims 1 to 23. 28 P233173GB

Images