Quantitative morphological signatures

Abstract

The present disclosure provides an attention-based MIL model for use in extracting and characterising cell features, at both the cell level and the population level. To characterise cells, cells in a well are initially fed into the model. The cells in a well include a points cloud of cells and a point cloud of corresponding cell nuclei. The cells in the well are passed through a pretrained DFN encoder which extrapolates the cell and nuclei features from the cells in the well. These extracted features are then passed through a transformer based encoder to produced transformed versions of the extracted cell and nuclei features. These transformed features are then fed into at least two classifiers. One of the classifiers is a cell-level MLP classifier which is used to compare the cell features with known drug response to attempt to match a phenotype signature with a known drug response. One other classifier is the bag classifier which is used to compare the cell population features with that of known drug responses, again to attempt to match to a phenotype signature. The outputs of the cell classification and the bag classification are used to label the cells in the well with a drug label or a potential drug label / use case.

Description

[0001] QUANTITATIVE MORPHOLOGICAL SIGNATURES Technical Field

[0002] The present disclosure generally relates to methods and systems for use in analysing cell shape and cell characteristics at a single cell and cell population level. More specifically, the present disclosure relates to methods and systems for analysis of cell shape and cell characteristics at the single cell and cell population level using transformer based analysis. The methods and systems enable the characterization of cell perturbations such that existing drugs can be classified and the development of new drugs can be accelerated through earlier use case predictions.

[0003] The present disclosure seeks to improve the problem, described herein in how a genetic (mutation, RNAi, CRISPR) or non-genetic (small-molecule, antibody, protein expression) perturbation results in a specific cellular phenotype.

[0004] Background and Related Art

[0005] The dynamic interplay of cellular proteins, metabolites, and lipids with their environment determines the shape of single cells in populations and tissues [1, 2]. Aberrations in cell shape are often a cause and / or sign of disease [3]. As such, cell shape provides insight into the state of cells; indeed, visual examination of cells is the basis for diagnostic pathology [4], and has been shown to be predictive of gene and protein expression and the activation of signalling pathways [5, 6, 7, 8, 9]. Conventional computer vision techniques have analysed cellular phenotypes, including shapes, primarily through classical geometric descriptors like area, perimeter and eccentricity [10, 11, 12, 13, 14, 15], facilitated by user-friendly tools [16, 17, 18]. These methods, while effective in identifying major cell shape categories [6] and linking them to genetic expression and molecular pathway activity [19, 20, 21, 22, 23], often utilise two dimensional (2D) images, a limitation given the three-dimensional (3D) nature of cells in living organisms.

[0006] The evolution of 3D microscopy [24, 25] has advanced the understanding of cell geometry in relation to biological functions and disease [22, 26, 27, 28, 29]. Despite these advancements, describing 3D cell shape remain challenging. Works have extended classical descriptors to 3D [30, 31, 32, 33, 34, 35, 36], and others have using mathematical techniques such as spherical harmonics [32, 33, 34, 35, 36]. Deep learning (DL) has transformed 3D shape analysis of everyday objects, powered by extensive datasets [37, 38, 39] for classification and representation learning tasks [40, 41, 42, 43, 44, 45]. Originally, DL methods analysed Euclidean data such as 3D binary voxels

[0046] . Utilising binary voxels, a recent study has developed generative models to reconstruct 3D cell shapes from 2D images and utilise the learned representations for downstream classification tasks

[0047] . More recently, geometric deep learning (GDL) methods that generalise DL to non-Euclidean data

[0048] have leveraged graph-based autoencoders to mine local and global geometries on mesh [49, 50, 51] or point cloud representations [52, 53, 54] of 3D objects, although these methods have not yet been applied to 3D cell shape data.

[0007] In drug discovery, the ability to decode phenotypic signatures of compounds offers a promising avenue to accelerate the identification of targets and leads [55, 56, 57]. However, the heterogeneity within drug-treated cell populations often remain overlooked. Multiple instance learning (MIL), a weakly supervised learning algorithm that deals with groups (bags) of data points (instances), could offer promise. The MIL framework lends itself ideally to classifying heterogeneous populations such as small cancerous regions in whole slide images

[0058] , segmenting 2D microscopy images

[0059] , and drug activity prediction

[0060] . However, as of yet, no one has shown this in the context of 3D phenotypic screens.

[0008] To investigate how environmental factors influence 3D cell shape, we developed and applied GDL algorithms to analyse the 3D shapes of melanoma cells in hydrogel matrices. Our study demonstrates the versatility of these methods across different datasets and imaging conditions. We integrated GDL with attention-based MIL to dissect the heterogeneity of cell populations at a single-cell level, and accurately classify inhibition by clinically relevant small-molecules from 3D shape. Our models could discern features predictive of specific cellular mechanisms, such as ERK activation. By employing these models on high-throughput screening data, we could correctly predict drug targets, and genetic perturbations that up or down-regulate ERK activity using the 3D shape alone. Taken together, this suggests that GDL models can not only learn features to discriminate between 3D cell shapes, but when integrating GDL with attention-based MIL, we can learn characteristic phenotypes to discriminate treatment conditions and predict cells states.

[0009] Summary of the Invention

[0010] The present disclosure relates to an attention-based multiple instance learning (MIL) framework which is used to identify shape profiles that are hallmarks of different targeted drug reactions, which are either new or known drug reactions.

[0011] In more detail, embodiments of the present disclosure relate to an attention-based MIL model for use in extracting and characterising cell features, at both the cell level and the population level. In order to characterise cells, cells in a well are initially fed into the model. The cells in a well include a point cloud of cells and a point cloud of corresponding cell nuclei. The cells in the well are passed through a pretrained DFN encoder which extrapolates the cell and nuclei features from the cells in the well. These extracted features are then passed through a transformer-based encoder to produced transformed versions of the extracted cell and nuclei features. These transformed features are then fed into at least two classifiers. One of the classifiers is a cell-level MLP classifier which is used to compare the cell features with known drug response to attempt to match a phenotype signature with a known drug response. One other classifier is the bag classifier which is used to compare the cell population features with that of known drug responses, again to attempt to match to a phenotype signature. The outputs of the cell classification and the bag classification are used to label the cells in the well with a drug label or a potential drug label / use case.

[0012] A first aspect of the present disclosure provides a computer-implemented method for assessing the phenotype of a cell, the method comprising: feeding point clouds, indicative of cells and cell nuclei, into a pretrained encoder; extracting, by the pretrained encoder, cell features and nuclei features from the point clouds; transforming, using a transformer encoder, the cell and nuclei features into transformed cell features; classifying the cell based upon the comparison of the transformed cell features with known phenotype responses; and labelling the cell based upon the comparison of the transformed cell features with known phenotype responses. Such a computer-implemented method for assessing the phenotype of a cell has several advantages. It can accurately classify the cell based on the comparison of the transformed cell features with known phenotype responses and label the cell accordingly. This method can help in the characterization of cell perturbations, which can accelerate the classification of existing drugs and the development of new drugs through earlier use case predictions.

[0013] In one embodiment the cell features and nuclei features are deep learning features extracted using the pretrained encoder, and which do not correspond to classical 3D cell features such as size, eccentricity or sphericity. Using deep learning features extracted by the pretrained encoder, which do not correspond to classical 3D cell features such as size, eccentricity, or sphericity, has several advantages. These data-driven features may capture more information about the cell than classical features, providing a more nuanced understanding of cell morphology and its changes under various treatment conditions.

[0014] In one example the deep learning features are not 2D features derived from a 2D z-slice through the cell, but are instead derived from the 3D point clouds. In one example the pretrained encoder is a Dynamic FoldingNet Encoder, DFN.

[0015] In one example the pretrained encoder produces a feature set, the feature set being a transformation of a set of feature embeddings, Z. In particular, the output of the transformer encoder can be defined by:

[0016] Tt= Layer N orm(Zt' + FFN) where F, is the output of the transformer encoder, Z- is the normalised embeddings from the DFN and FF / V is the feed-forward network.

[0017] In one example the method further comprises deriving, by the DFN, geometric deep learning based 3D cell features from the cells and cell nuclei within the point clouds.

[0018] In one example the method further comprises concatenating, by the pretrained encoder, the cell features and nuclei features into concatenated cell and nuclei feature pairs.

[0019] In one example the method further comprises grouping features from the cell features and nuclei features that appear in a single field of view of the point clouds into a virtual bag; and labelling the virtual bag with a label indicative of an associated application.

[0020] In one example the transformer encoder is a multi-head attention-based transformer encoder. In this example the method may further comprise extracting, using the transformer encoder, the amount repeated cell features and nuclei features within the virtual bag; and extracting, using the transformer encoder, dependencies between the 3D cell features and the 3D nuclei features.

[0021] In one example the method further comprises analysing the dependencies between the 3D cell features and 3D nuclei features; and extracting, based on the dependencies, changes in the inter-relationship of the 3D cell features and 3D nuclei features. This allows for the analysis of the dependencies between the 3D cell features and 3D nuclei features and the extraction of changes in the inter-relationship of the 3D cell features and 3D nuclei features. This can provide a more nuanced understanding of cell morphology and its changes under various treatment conditions.

[0022] In one example, the method further comprises classifying, using a cell-level classifier, cell level features based upon the comparison of the transformed cell features with the known phenotype cell level features. In particular, the classifying may comprise classifying, using a bag-level classifier, bag level features based upon the comparison of the grouped features within the bag with the known phenotype bag level features. Classifying cell level features using a cell-level classifier and bag level features using a bag-level classifier can provide a more nuanced understanding of cell morphology and its changes under various treatment conditions. This is because the cell-level classifier compares the transformed cell features with known phenotype cell level features, while the bag-level classifier compares the grouped features within the bag with the known phenotype bag level features. This allows for a more detailed analysis of the cell features and their changes under different treatment conditions.

[0023] In one example the bag-level classifier comprises a multiple instance learning, MIL, aggregator and a multi-layer perceptron, MLP, classifier. In a preferred examples the multiple instance learning aggregator is a two-stream multiple instance learning aggregator, wherein a first stream comprises an instance-level classifier and the second stream comprises a bag-level classifier.

[0024] In one example an output of the instance-level classifier is defined as: wherein cinstanceis the output of the instance-level classifier, Ttis the output of the pretrained encoder, max is the max pooling function and Wois a weight vector.

[0025] Preferably, in one example an output of the bag-level classifier is defined as: wherein cbagis the bag-level, Ttis the output of the pretrained encoder, Wbis the weight vector for binary classification, b is bag embedding.

[0026] Moreover, in one example the output of the multiple instance learning aggregator is the weighted sum of the first and second stream.

[0027] In one example the method further comprises applying, using the MIL aggregator, attention weights on each cell within a bag, the attention weights being indicative of the importance of the cell and nuclei features of said cell for predicting similar phenotype responses.

[0028] In one example the label for the cell is a label indicative of a known substance, the known substance being one which has a matching phenotype signature with the cell. In another example the label for the cell is a label indicative of a clinical application, the clinical application being the clinical application of a known substance which has a matching phenotype signature with the cell.

[0029] In a further example the point clouds are obtained by: three dimensional segmentation of cells and cell nuclei; surface rendering of the three dimensional segmentation using a marching cubes algorithm; smoothing the rendered surfaces using Laplacian smoothing; and then extracting vertices of polygons forming the smoothed surfaces as the point clouds. This process allows for the accurate representation of the 3D shape of cells and cell nuclei.

[0030] A further aspect of the present disclosure relates to a computer system for implementing a cell assessment and labelling algorithm, the system comprising: one or more processor units; and a computer readable storage medium storing one or more computer programs such that when performed by the one or more processor units cause the computer system to operate in accordance with the method of any of the above examples.

[0031] A yet further aspect of the present disclosure provides a system for predicting clinical applications of a substance, the system comprising: a. input data, including point clouds indicative of cells and cell nuclei, the data being input such that it can be fed into a pretrained encoder; b. a pretrained encoder arranged to extract, cell features and nuclei features from the point clouds; c. a transformer encoder arranged to transform the cell and nuclei features into transformed cell features; d. one or more classifiers arranged to classify the cell based upon the comparison of the transformed cell features with known phenotype responses; and a processor arranged to label the cell based upon the comparison of the transformed cell features with known phenotype responses.

[0032] Further features and advantages will be apparent from the appended claims.

[0033] Brief Description of the Drawings

[0034] Further features and advantages of the present disclosure will become apparent from the following description of an embodiment thereof, presented by way of example only, and with reference to the accompanying drawings, wherein like reference numerals refer to like parts, and wherein: Figure 1 shows images relating to Single cell 3D shape analysis using geometric deep learning. Figure 1A related to single cells labelled with transgenes expressing Histone-2B (nucleus, green) and CAAX-GFP (membranes, magenta) and ERK-KTR (ERK activity, blue) were embedded in a collagen matrix and treated with a variety of small molecule drugs. The cells were imaged in 3D at a high throughput by stage scanning oblique plane microscopy (OPM). Maximum intensity projections (XY, XZ, YZ views) of cells treated with Blebbistatin (myosin inhibitor) are shown. Figure IB relates to a region of interest for a single cell whereas Figure 1C relates to cell segmentation masks; Figure ID relates to marching cubes which create meshes from segmented masks of cells and nuclei; Figure IE to point clouds extracted from single cells and nuclei meshes; and Figure IF relates to a feature respresentation learned through a Dynamic FoldingNet autoencoder. This autoencoder is trained by minimising the Chamfer Distance between the input and output point cloud;

[0035] Figure 2 shows a series of graphs relating to a convolutional foldingnet which reveals the cell shape landscape in a cancer cell line. Figure 1A is a rendered sample image of cell masks which are plotted using UMAP. Figures 2B-E show data points (representing cell shapes) coloured by their classical feature values to show how (b) Volume, (C) Eccentricity, (D) Sphericity, and (E) Roll are represented by the GDL features in UMAP. Figure 2F shows the probability density of the UMAP which was estimated using kernel density estimation on all cells, (G) DMSO-treated cells, (H) Blebbistatin-treated cells and (I) Nocodazole- treated cells. Figure 21 shows the points on the UMAP coloured by their distance relative to the coverslip. Figure 2K shows the distribution of shapes of cells on the coverslip. Figure 2L shows the distribution of the shapes of cells further away from the coverslip;

[0036] Figure 3 shows depictions relating to learned shape features to enable downstream classification tasks. Figure 3A shows hierarchical clustering of the mean features of each treatment. Figure 3B used an SVM to predict treatment for all cells using different shape feature sets. Figure 3C shows the average balanced accuracy of pairwise classification using a 10-fold cross-validation for the combined cell and nucleus 3D DFN feature set in proximal cells and (D) distal cells. The insets in the top right of panels B and C show the mean result of each treatment class across all one-vs-one comparisons. Figure 3E shows point cloud renders for cells from each class from the Red Blood Cell dataset

[0065] . Figure 3F shows point cloud renders of cells from each class of the VesselMNIST dataset

[0066] ;

[0037] Figure 4 shows an attention-based multiple instance learning framework for compound classification. Figure 4A shows an attention based multiple instance learning model used for compound classification, in accordance with embodiments of the present disclosure. A bag of instances (cells in a well treated with a certain drug) are fed through the pre-trained DFN autoencoder to extract features for each cell and nuclei. Each cell and nuclei pair are concatenated and fed through a transformer layer. The transformed features then pass through a DSMIL model to jointly learn instance classifiers, critical instances, bag classifiers, and bag embeddings. We train different models for each drug to classify a drug vs the rest. Figure 4B shows an averaged balanced accuracy of all folds for all compounds for different MIL algorithms. Figure 4C show how our cells were harbouring ERK-KTR, which is a reported of ERK activity as measured by the nuclear ERK-ratio. Figure 4D shows the ERK ratio is calculated by the ratio between the mean ERK-KTR intensity in the nucleus and the mean ERK-KTR intensity in a ring-region around the nucleus. Figure 4E shows an example of an active and inactive ERK-KTR. Figure 4F shows example cells with high attention (> 0.98) for Binimetinib-treated cells. Figure 4G shows a scatter plot of the attention score for Binimetinib and the ERK-ratio cells. Figure 4H shows the correlation between high attention scores and classical shape feature. Cells with the top 10 attentionscores for the most confident bags in each fold for each classifier are shown (100 cells per treatment). Figure 41 shows the shape space landscape, highlighting the high attention cells for each treatment (cells with an attention scores greater than 0.5);

[0038] Figure 5 shows depictions relating to an integrating GDL and MIL to characterise perturbations using 3D shape. Figure 5A shows the process for characterising a drug by RNAi. RNAi perturbations are imaged in 3D. Features are extracted from point clouds of cells and nuclei using a pre-trained autoencoder. Bags of these features are fed through each drug MIL classifier and given a probability score on how similar the gene knockdown is to the drug. Figure 5B shows the distribution of similarity to Blebbistatin when using Euclidean distances. Figure 5C shows the distribution of similarities to Blebbistatin when using MIL. Figure 5D shows further characterising Blebbistatin hits by evaluating their scores on all drug classifiers. Figure 5E relates to the MIL revealing the RNAi's most similar to Binimetinib. Figure 5F relates to Phosphorylated ERK screen reveals hits for MEK inhibition. Figure 5G shows a Venn diagram showing the overlap between the p-ERK and MIL screens. Figure 5H shows the pERK levels from our corresponding screen for Control, RHOBTB2-kd, ARHGEF9-kd, and ABR-kd cell from left to right, respectively (I) Example cells for RHOBTB2, ARHGEF9, and FARP1 RNAi imaged in 3D, respectively;

[0039] Figure 6 shows an example block diagram of a computer system for use in executing the associated methods, algorithms and programs, in accordance with embodiments of the present disclosure.

[0040] Detailed Description of the Preferred Embodiments The present disclosure provides a method and system which combines imaging and Al based pipeline to analyse the cell shape at the single cell, and population level, following perturbation by perturbation (small-molecules, antibodies, siRNA, CRIPSR). This allows the present disclosure to infer: a) Mechanism of Action (MOA) and perturbation targets. Including on-target and off-targets. b) Effect on cellular behaviours - including survival, proliferation, and differentiation. c) How targets and effects are dependent on environment and genetic backgrounds d) How perturbation will affect different cell types. e) Given (a) - (d), which diseases and patients might benefit or not benefit from the use of this perturbation.

[0041] The Sentinal4D pipeline involves:

[0042] 1. Bioengineering of matrices and culturing of cells in 3D conditions.

[0043] 2. High-throughput 3D imaging of cells. At single time point and across time.

[0044] 3. Quantification of 3D single cell morphology by Geometric Deep Learning (GDL)

[0045] 4. Transformer based analysis of 3D morphologies specific to perturbations (signatures)

[0046] 5. Classification of perturbation response through matching of signatures

[0047] 6. Integration of shape signatures with additional omics and patient data to predict target, cell-type effects, and patient benefit.

[0048] In many cases how a genetic (mutation, RNAi, CRISPR) or non-genetic (small-molecule, antibody, protein expression) functions to results in a specific cellular phenotype is very poorly understood.

[0049] Understanding how a perturbation functions is particularly essential during therapeutic development. Numerous methods, including large screens, in silico approaches, or directed chemical synthesis, can lead to the successful development of a perturbation that might inhibit a target (i.e. a small molecule that inhibits enzyme activity). But whether this perturbation will have clinical development typically requires extensive study to identify MOA, the diseases which will benefit from this novel perturbation, the cell types against which the perturbation as a desired effect, and the patients who will be sensitive / resistant. Answering these questions currently takes years of empirical study involving trial-and- error experiments across cell lines, mouse models, and patients. The current time and cost to go from drug discovery to clinical benefit is estimated to be 10-12 years at 1-2 Billion pounds. This is both unsustainable for the long term and severely affects patient health.

[0050] The present disclosure proposes to markedly accelerate drug development and massively reduce costs by characterizing novel perturbations, and making predictions as to their use, at the earliest stages. In particular, the present disclosure can cut costs and prevent waste of resources by having novel perturbations 'fail faster', as well as prioritize candidates for more detailed characterization.

[0051] The vast majority of image analysis technologies and Al models - both used in research labs or in the context of clinical pathology - analyse 2D images. The analysis of 3D images provides much more information, both increasing specificity and sensitivity.

[0052] The use of attention based models (extensively used in other fields) have not yet been implemented in the analysis of 3D cells.

[0053] Single cell gene sequencing, transcriptomics, or proteomics are also technologies to characterize perturbations. However, these technologies are: very expensive and time consuming to implement in single cells; not currently scalable; can largely not be used in living cells over time; often poor sensitivity and specificity in classifying perturbations.

[0054] In more detail, embodiments of the present disclosure relate to an attention-based MIL model for use in extracting and characterising cell features, at both the cell level and the population level. In order to characterise cells, cells in a well are initially fed into the model. The cells in a well include a point cloud of cells and a point cloud of corresponding cell nuclei. The cells in the well are passed through a pretrained DFN encoder which extrapolates the cell and nuclei features from the cells in the well. These extracted features are then passed through a transformer based encoder to produced transformed versions of the extracted cell and nuclei features. These transformed features are then fed into at least two classifiers. One of the classifiers is a cell-level MLP classifier which is used to compare the cell features with known drug response to attempt to match a phenotype signature with a known drug response. One other classifier is the bag classifier which is used to compare the cell population features with that of known drug responses, again to attempt to match to a phenotype signature. The outputs of the cell classification and the bag classification are used to label the cells in the well with a drug label or a potential drug label / use case.

[0055] The present disclosure will now be discussed in more detail in relation to the associated Figures.

[0056] As discussed above, Figure 4A shows an attention based multiple instance learning model 4 used for compound classification, in accordance with embodiments of the present disclosure. The multiple instance learning model 4 comprises a bag of instances 40 (which show cells in a well which are treated with a certain drug), a pretrained DFN encoder 41, extracted cell features 42, extracted nucleus features 43, a transformer encoder 44, the "transformed" features 45 (i.e., the transformed extracted cell features 42 and the transformed extracted nucleus features 43), a cell-level MLP classifier 46, a bag level classifier 47 including a MIL aggregator and a MLP classifier, and a cell classification unit 48.

[0057] In the model 4, a bag of instances 40 (cells in a well, which are treated with a certain drug) are fed through the pre-trained DFN autoencoder 41 to extract features, the extracted features being cell features 42 and nucleus features 43. Each cell feature 42 and nuclei feature 43 pair are concatenated and fed through a transformer encoder layer 44. This results in transformed features 45 of the cell 42 and nucleus features 43. The transformed features 45 then pass through various classifiers. For example, the transformed features 44 pass through a cell-level MLP classifier 46 which is used to identify characteristics at the cell level that will enable cell classification 48 to decipher whether the cell contains features that relate to a known drug or phenotype signature. Also, the transformed features 45 will pass through a bag classifier 47. The bag classifier 47 comprises a MIL aggregator and a MLP classifier to aid in identifying features within the cell population that may be indicative of a known drug or phenotype signature. The features identified in the cell classification 48 and the bag classifier 47 will be used to identify a likeness to a known drug or a potential use case for an unknown drug. Then, a drug label associated with the identified features will be applied to the analysis of the cells in a well 40.

[0058] Results

[0059] An automated method to profile cell shape in 3D using geometric deep learning

[0060] We used obligue plane microscopy (OPM) [24, 25] to perform light-sheet imaging of more than 65,000 WM266.4 melanoma cells harbouring CAAX-GFP, H2B, and ERK-KTR (See Methods), embedded in tissue-like collagen hydrogel matrices (Figure 1 A). Cells were physically embedded in an environment that spanned from mechanically rigid and flat (proximal to the coverslip) to mechanically soft (distal to the coverslip). To model a latent space representative of many possible shapes, we treated a subset of cells with clinically relevant inhibitors of cytoskeletal organisation and cell signalling pathways.

[0061] Cells and nuclei were segmented (Figure 1 B and C) using active contours and thresholding techniques serially (see Methods). The 3D voxel masks were transformed into mesh objects by applying the marching cubes algorithm

[0061] (Figure 1 D). Subsequently, these meshes were refined with Laplacian smoothing to eliminate irregularities. Finally, to represent each cell and nucleus in a point cloud format, points were uniformly sampled from the surfaces of the smoothed meshes (Figure 1 E) (see Methods). Point clouds were pre-processed by mean centring and scaling to assist in training our model.

[0062] To quantify single-cell shapes in 3D, we developed a GDL model (Figure 1 F). We used a graph-based autoencoder to learn 3D cell shape features in a self-supervised manner (Figure 1 F); modifying the FoldingNet

[0053] autoencoder by incorporating a Dynamic Graph Convolutional Neural Network (DGCNN)

[0040] , resulting in the formulation of a Dynamic FoldingNet (DFN). This modification achieved the highest 3D shape classification accuracy across datasets. We then used this trained autoencoder to extract 3D cell shape features for downstream analysis, including describing the latent cell shape landscapes and transferring the models to other 3D shape datasets. Finally, we use the learned features as input to a novel attention-based MIL pipeline as a drug characterisation platform (Figure 1 G).

[0063] 3D cell shape distribution is a readout of perturbation and environment

[0064] We employed our DFN model to derive GDL-based 3D shape features from all cells and nuclei in our dataset. We also extracted classical 3D cell shape features such as size, eccentricity, sphericity, and the number of protrusions, 2D shape features from maximum intensity projections (2D MIP) of cells and 2D classical features from a single z-slice (2D z-slice) through a cell (the slice that contains the largest area of the cell) using techniques described in

[0030] . Using UMAP

[0062] , we visualised the cell shape landscape derived from our GDL features (Figure 2 A), revealing that the first dimension predominantly represents cell size and eccentricity (Figure 2 B and C). Smaller, rounder cells (Figure 2 D and E) have UMAP 1 values < 4, while larger, more eccentric cells (cells with higher volume and surface area and eccentricity values) exceed UMAP 1 values of 8. Kernel density approximation assessed the shape distribution on the UMAP embedding (Figure 2 F). Notably, cells exposed to different small-molecule inhibitors exhibited varied shape distributions. Control cells (DMSO) spanned a broad shape spectrum (Figure 2 G), whereas drug-treated cells occupied more specific regions of this shape space. For instance, Blebbistatin-treated cells (Figure 2H) occupied areas of UMAP that corresponded to more protrusive and eccentric cell shapes, while Nocodazole-treated cells (Figure 2 I) primarily occupied areas of UMAP that corresponded to rounder shapes with fewer protrusions. This latent space analysis underscores that treatments generally don't introduce new geometries but shift feature distributions within a population [6, 63].

[0065] We noted the cells in the dataset varied by the depth to which they were embedded in the collagen matrix. Some adhered to the plate's 2D plastic surface and were more similar to cells plated on 2D surfaces than those suspended in 3D matrix (Figure 1 A). To discern the impact of this physical environment on cell shape, cells were colour-coded based on their distance from the coverslip (Figure 2 F). The shape distribution of coverslip-proximal cells, which essentially explored 2D substrates (Figure 2 G) contrasted starkly with that of fully collagen-embedded cells (Figure 2 H). Cells in these different depths in the collagen located in opposing areas of UMAP space. Consequently, we treated cells at different collagen depths as distinct populations. Cells with a nucleus centre of mass < 7 pm from the coverslip were termed 'Proximal', while those > 7 pm were labelled 'Distal'. This 7 pm threshold was chosen based on proximity to the coverslip and evidence suggesting cells can detect and react to rigid substrates over 3-10 pm

[0064] . These observations demonstrate that cell shape distribution data encode information regarding the cellular environment.

[0066] Learned shape features enable downstream classification tasks

[0067] Hierarchical clustering of the average features for each condition revealed high correlation between the 3D shapes of Blebbistatin and H1152, consistent with the fact that these drugs both act on the same Rho-ROCK-Myosin activation cascade (Figure 3 A). Furthermore, the 3D shapes of Blebbistatin and H1152 were dissimilar to Nocodazole- treated cells, consistent with the idea that Nocodazole promotes contractility, whereas Blebbistatin and H1152 inhibit contractility [8]. This suggests that 3D cell shape is predictive of small molecule MOA.

[0068] To determine if 3D cell shape as measured by GDL can provide information regarding the nature of different pertubagens, we developed SVM classifiers capable of performing pair- wise treatment classification based on single-cell 3D shapes using GDL features or classical features (see Methods). 3D DFN features surpassed traditional 2D and 3D classical features in predicting compound effects (Figure 3 A). 3D classical features outperformed 2D MIP features by an average of 2.5 % in terms of balanced classification accuracy across treatments. On average, using 3D DFN features to predict treatment outperformed 3D classical features by 1.85 % in all cells, suggesting that these data-driven features may be capturing relevant information of cell shape that is overlooked by classical features.

[0069] To determine if our model was sensitive to differences in environment, we made predictions separately for proximal (Figure 3 B) and distal cells (Figure 3 C). Notably, classification accuracies were, on average, superior in proximal versus distal settings. Using 3D DFN features to predict treatment outperformed 3D classical features by 1.02% in the distal setting and 3.08% in the proximal setting. In the proximal setting, we most accurately classified cells treated with Nocodazole (microtubule dynamics inhibitor), Blebbistatin (myosin-II-specific ATPase inhibitor) and Hl 152 (a Rho-associated kinase (ROCK) inhibitor) from other cells with accuracies of 80%, 72.6% and 68.3% on average and accuracies of 75.5%, 71.8% and 66.8% in the distal setting, respectively. We further split the cells in collagen into two classes, those between 15 and 60 pm from the coverslip, and those greater than 70 pm from the coverslip. Using DFN-extracted shape features, we could classify these classes with a balanced accuracy of 84.9% (std = 0.7%). This shows that we can discern where in the matrix a cell is positioned based on 3D shape, even though we are not explicitly measuring position.

[0070] DFN generalises across cell types and imaging platforms

[0071] Table 1: Classification results on the Red Blood Cell dataset and VesselMNIST

[0072] Red Blood Cell VesselMNIST

[0073] Method Shape ACC(T) AUC(T) | Method Points Smoothed ACC(T) AUC(T)

[0074] We tested our model on different cell types imaged by different imaging platforms. Using this model (trained on WM266.4 cells), we extracted 3D shape features from 825 red blood cells

[0065] (Figure 3 E) and trained an SVM to classify them into their annotated shape classes. The left-hand side of Table 1 shows the balanced accuracies (ACC) and area under the receiver operating curve (AUC) for classical features as well as the original FoldingNet architecture and our DFN with different shapes to be folded in the decoder (see Methods). The DFN features, even when trained on an unrelated cancer cell line, surpassed classical and FoldingNet features in ACC, AUC, Fl score, precision, and recall. We combined the DFN (using a plane shape with k = 20 and 2048 points sampled from an unsmoothed mesh) and classical feature sets to determine if DFN features added more information to classical features or vice versa. This combined feature set improved accuracy from the classical features alone (0.828 to 0.881); however, the combined features only outperformed DFN features in terms of AUC (0.984 to 0.987), suggesting that the DFN features may incorporate many relevant classical features.

[0075] Additionally, we evaluated our method on the VesselMNIST3D dataset (Figure 3 F), part of MedMNIST v2

[0066] . This dataset contains 1909 3D binary masks of healthy and aneurysm vessel segments split into training, validation and testing sets. The right-hand side of Table 1 shows the ACC and AUC for classical features and DFN features using a plane shape with k = 20 and different numbers of points sampled from smoothed and unsmoothed meshes. All variations of the DFN features outperform the classical features regarding ACC. The greatest ACC was seen using a DFN with 2048 points from both smoothed and unsmoothed meshes (0.843). The highest AUC was achieved when using the DFN with 4098 points sampled from an unsmoothed mesh. Combining the classical and DFN features slightly improved balanced accuracy and recall, suggesting there is some, but not complete, overlap between information captured by DFN and classical features.

[0076] An attention-based multiple instance learning framework for compound classification

[0077] While we were capable of developing SVMs that distinguished different chemical perturbations based on the shape alone, cells often exhibit highly heterogeneous responses within a perturbed population and between perturbations [6]. Single-instance learning algorithms such as SVMs forego contextual information of cell shape within a population of drug-treated cells, as each cell is treated as an independent data point. Addressing this, we approached this challenge through the lens of multiple instance learning (MIL)

[0067] (see Methods).

[0078] Using the pre-trained DFN, 3D shape feature representations were extracted from corresponding cell and nuclei pairs and concatenated. All feature representations from cell and nuclei pairs in a single field of view were grouped into bags, with each bag given the label of the well treatment. These bags were then passed through the MIL model. We utilised an adaptation to the attention-based dual-stream MIL (DSMIL)

[0068] framework, such that we employ a transformer encoder layer on the feature representations before being passed into the DSMIL model (T-DSMIL). This is intended to learn the contextual information of cells (instances) in a population (bag) as well as dependencies between cell and nuclei 3D shape features (Figure 4 A). This allowed us to use changes in the interrelationship of cell and nuclear morphology to predict drug treatments. The bag of 'transformed features' was then passed through the DSMIL architecture to give a final prediction of the treatment of the bag of cells (see Methods). We evaluated our T-DSMIL against a range of MIL algorithms, from simple pooling operators (mean, max and logsum-exponent (LSE) pooling) followed by multi-layer perceptron (MLP) classifiers to other attention-based algorithms, including TransMIL

[0069] and the original DSMIL

[0068] ).

[0079] Attention-based methods outperformed the simple pooling algorithms (Figure 4 B) with T- DSMIL classifying all drugs with balanced accuracies greater than 83% and up to 99.3% for Nocodazole. Interestingly, mean pooling outperformed max-pooling and LSE pooling in all cases. TransMIL produced higher classification accuracies than DSMIL. Including the transformer-based encoder in DSMIL significantly improved predictive accuracy. This increase in performance when adding the transformer layer before DSMIL could suggest the importance of the learning the relationship between features (including relationships between nuclear and cellular shape features) as well as modelling the distribution of cells in a population. Attention-based MIL algorithms also output attention weights for each cell in a bag, offering interpretability into the important cell and nuclear shapes when predicting certain treatments. Figure 4B shows example cells with high attention for Blebbistatin, Nocodazole, and Binimetinib.

[0080] Attention weights explain drug induced phenotypes and align with signalling activity

[0081] Extracellular signal-Regulated Kinase (ERK) is part of the Mitogen-Activated Protein Kinase (MAPK) signaling pathway, and ERK-KTR specifically monitors the activity of the ERK kinase. When ERK is active and phosphorylates the ERK-KTR, the KTR changes its location in the cell, typically moving from the nucleus to the cytoplasm, which can be visualised and guantified through fluorescence imaging

[0070] . When ERK is inactive, the ERK-KTR is more present in the nucleus than the cytoplasm (Figure 4 C). Binimetinib, a MEK inhibitor, also inhibits ERK activation, and therefore cells that have had MEK inhibited would usually have high nuclear ERK-KTR ratios (ratio of nuclear to cytosplamic ERK-KTR) (Figure 4 D). Strikingly, the attention score for cells from the Binimetinib classifier correlated with the ERK ratio with a Peason's correlation of 0.41 (p < 1 x 10-5). This suggests that our model that solely uses 3D shape, can spot MEK-inhibited-associated shapes that may also be those responding the most to treatment. This indicates that high levels of MEK inhibition are associated with shape changes, and that these shape changes are unigue and characteristic of MEK inhibition compared with the other treatments in the study. Notably, MEK1 / 2 inhibited shapes are distinguishable from CDK4 / 6 inhibition which has key functions downstream of MEK1 / 2 in the same MAPK signalling pathway. This suggests that shapes changes from MEK inhibition are mediated by direct substrates of the MEK1 / 2 kinases rather than outputs of the MAPK pathway in general.

[0082] We collected the five cells with the highest attention for the most confident prediction for each fold for each bag and extracted their 3D classical features describing their shape (Figure 4 C). For example, Nocodazole 'critical shapes' learned by the model were round shapes with an average absolute sphericity of 0.71. In contrast, the mean sphericity for all cells was 0.63 and, for all Nocodazole cells, was 0.68. The sphericity of the nucleus was 0.88 in high-attention Nocodazole cells as opposed to 0.86 in all Nocodazole cells. Furthermore, the Blebbistatin and H1152 critical shapes had high values for spreading (0.93 and 0.91) and eccentricity (0.97 and 0.94) and low values for sphericity (both 0.58). Interestingly, those cells with high attention coefficients for Binimetinib had a high nuclear ERK-KTR ratio of 3.07 as opposed to an average of 2.79 for all Binimetinib-treated cells. Conversely, cells with low attention for Binimetinib treated cells had lower nuclear ERK- KTR ratios with an average of 2.42. These findings demonstrate that MIL effectively learns accurate critical phenotypes, filtering out less informative cells in heterogeneous populations.

[0083] Figure 4 D shows cells with a coefficient greater than 0.5 for each drug classifier to explore where they existed in shape space (as described using UMAP). Interestingly, CK666 had high attention cells in areas of the shape space landscape that corresponded with both the rounder and spindle shapes. High-attention cells for Binimetinib can be found on both the rounder and protrusive areas of the shape space landscape. This suggests the model may need to see both shapes in conjunction to be confident about a predicting whether a cell has been treated with Binimetinib. The ability to leverage these types of inter-cell relationships in heterogeneous populations (semantics) highlights the power of MIL.

[0084] Integrating GPL and MIL to characterise perturbations using 3D shape

[0085] Finally, we present how GDL and attention-based MIL could be used to learn drug targets. We obtained a score on how similar the phenotypic profile of a genetic knockdown is to a treatment condition by passing bags of gene knockdowns through the trained T-DSMIL. The gene knockdown dataset comprised WM266.4 cells that had been depleted by RNA interference (RNAi) for most human Rho guanine nucleotide exchange factors (RhoGEFs), Rho GTPase activating proteins (RhoGAPs), and Rho family GTPases. This dataset contained over 35,000 single cells across 168 different treatment conditions (TCs) across two repeats. Stage scanning OPM was used to image the 3D shape of cells. Features were extracted from point clouds of these cells and nuclei using the DFN trained on the drug- treated dataset. We trained our T-DSMIL on bags of cells proximal to the coverslip to classify its chemical compound treatment. We used only cells on the coverslip from the drug-treated dataset, as the RNAi dataset consisted only of cells on the coverslip. A 10- fold cross-validation strategy was used. Bags of features of cells and nuclei with specific gene knockdowns were then fed through each pre-trained T-DSMIL drug classifier, and the average score across the folds was recorded (Figure 5 A). This score is the probability, according to the classifier, that the bags of cells with specific RNAi conditions are treated with each of the 9 small molecular inhibitors in our study. In other words, how similar the 3D cell shapes of the RNAi are to that of drug-treated cells.

[0086] RHOA, ABR, S0S1, and ARAP3 were among the top-scoring genes on the Blebbistatin MIL classifier (all > 96%). We further characterised these hits for Blebbistatin by calculating their scores on all the drug MIL classifiers (Figure 5 D). While these gene knockdowns all score highly on the Blebbistatin classifiers, they score vastly differently on the other drug classifiers. For example, RHOA also scores high on H1152 and MK1775 (75% and 62%, respectively). ABR only scores highly on the Blebbistatin classifiers (0.97). S0S1 also scores highly for Blebbistatin, but also on MK1775. Finally, ARAP3 scores highly for Blebbistatin, H1152 and CK666. These data suggest that TCs which score highly on multiple classifiers are likely to represent perturbations that target multiple processes, i.e. deplete genes that activate multiple pathways or even off-target effects. TCs with high scores on single classifiers are indicative of more targeted disruptions.

[0087] We fully characterised every chemical compound in our dataset by every RNAi in our dataset, allowing us to group RNAi and compounds with similar mechanisms of action. We further annotated each RNAi by its Rho regulator and domain to explore cluster enrichment. Although few RNAi scored relatively high on the PF228 (FAK inhibitor) classifiers, the highest-scoring RNAi was TRIO (0.81). It has been shown that TRIO may be involved in the regulation of focal adhesion dynamics as well as affecting the actin cytoskeleton [71, 72]. Top hits for Palbociclib included RND1 (0.67), for MK1775 included IGDCC4 (0.73), OBSCN (0.73), RND1 (0.63), and CDC42 (0.69), and for Nocodazole included VAV2 (0.99), CHN1 (0.99), and STARD13 (0.99).

[0088] To explore the idea that 3D shape is predictive of disruption of specific pathways, we focused our analysis on TCs that resulted in 3D shapes quantitatively similar / dissimilar to those following treatment by the MEK inhibitor, Binimetinib

[0073] . Overall, the MIL classifier ranked scores correlated with our corresponding pERK screen with a Spearman's rank correlation coefficient of 0.632 (p < 8e-15). When using the cosine similarity of the average features, we saw a rank correlation coefficient of 0.479 (p < 2e-8). We found that knocking down RHOBTB2, ARHGAP23, ARHGEF9, and ITSN2 resulted in 3D shapes most similar to those following Binimetinib treatment (Figure 5 E). To determine if these genes affected MEK activity, we compared these results to data collected from a previous unbiased screen where we measured phosphorylated-ERK levels following systematic gene depletion (Methods). ERK is the target of MEK, and phosphorylation of ERK is a direct readout of MEK activity. Depletion of RHOBTB2, ARHGAP23, ARHGEF9 and ITSN2 all caused a significant decrease in pERK levels, meaning that MEK was inhibited (Figure 5 F). Conversely, the depletion of FARP1, RHOA and ABR resulted in 3D shapes that were among the most dissimilar to Binimetinib (according to T-DSMIL). Strikingly, our corresponding screen demonstrated that knocking out these genes results in higher MEK activity.

[0089] We selected all hits from the p-ERK screen based on a z-score cutoff of below -1.28 (p(z < -1.27) = 0.1) and scores greater than 0.9 for the MIL classifier. Figure 5 G shows a Venn diagram of the p-ERK and T-DSMIL 3D shape screens. All of the knock down treatments classified by 3D MIL with a Binimetinib score above 0.9 were also hits for MEK inhibition (read-out by pERK). SYDE1 had a probability of 0.939 by the Binimetinib T- DSMIL classifier; however, it was not a hit in the MEK screen as per the -1.27 cutoff (z = -0.66). The classifier did not see several hits for MEK inhibition (read out by pERK levels) as similar to Binimetinib. For example, ARAP2 and ARHGEF40 had a probability of 0.558 and 0.635, respectively, although came up as hits in the MEK screen (z = -1.42 and z = - 1.69, respectively). This supports our finding that MEK inhibition (as measured by pERK and shown in figure 5 H) can be directly measured by 3D cell shape (Figure 5 I). These results demonstrate the utility of 3D cell shape (Figure 5 I). These results demonstrate the utility of 3D cell shape and our proposed attention-based multiple instance learning methods for phenotypic drug screening pipelines.

[0090] Discussion

[0091] In this study, we provided a comprehensive examination of the intricate relationship between 3D cell shape and state, focusing on WM266.4 melanoma cells under various environments and chemical and genetic perturbations. The utilisation of OPM enabled a detailed visualisation of 3D cell shape, creating a rich dataset for understanding cellular adaptations to their surroundings under TCs. Our introduction of GDL models, particularly our modifications to the FoldingNet, resulted in a powerful tool for single-cell shape profiling. This approach has demonstrated higher 3D shape classification accuracies across datasets, suggesting that GDL features may capture more information than classical methods. Finally, integrating GDL with attention-based MIL enabled the accurate classification of drug targets and cellular mechanism, such as ERK activation based on 3D cell shape alone.

[0092] Robust representation of the shape of 3D objects is a fundamental goal in computer vision, and there has been rapid progress in this domain. Deep learning (DL)-based shape analysis methods have been built on large-scale labelled datasets of 3D models of everyday objects [37, 38, 39]. State-of-the-art methods have analysed these 3D objects as either 3D binary voxels, meshes, or point clouds for tasks of classification [41, 42, 43], and unsupervised representation learning [46, 74, 75, 44, 45]. All three shape representations have their strengths. Voxels have a uniform structure thus making them easily to use in common DL algorithms, however, are memory intensive and computationally demanding. Meshes can represent complex surfaces with fewer data points than voxels, however, their complex structure can be difficult to process. Although point clouds lack information about how points are connected, they are arguably the simplest form of shape representation, and methods to implicity connect these points are available, such as those used in this study. This has led to in-depth research into DL architectures, which explore local geometric information using convolution, graph

[0040] , and attention mechanisms

[0076] on point clouds for improvements in 3D shape analysis. GDL on point clouds is commonly used in 3D shape analysis, from classifying everyday objects [77, 78], to predicting patient information based on the shape of brains

[0079] , to analysing cellular morphology in 3D EM data

[0080] . However, is yet to be used on the analysis of the 3D shape of cancer cells in response to perturbations. The ability of the DFN to capture both local and global 3D geometric representation of cells enables a more nuanced understanding of cell morphology and its changes under various treatment conditions.

[0093] The relationship between cell morphology and cell state has been extensively explored in the context 2D genetic [5] and small-molecule pathway screens

[0056] , predicting drug resistance

[0081] , metastasis

[0082] and metastatic potential [3] and more recently, has been integrated with chemical compound

[0083] and gene expression data

[0057] to provide complementary information for this task. However, the heterogeneity within drug-treated cell populations often remain overlooked, where image or well-averages, or singleinstance models have prevailed until now. We proposed utilising attention-based multiple instance learning (ABMIL) for compound classification. The ABMIL framework lends itself ideally to situations such as drug screening, where it is known that every cell (or instance) within a well (or bag) has been subjected to a specific treatment. Still, given the variability in their responses, it remains unclear which cells express a phenotype characteristic of that drug. We have adapted a DSMIL architecture that incorporates a transformer encoder to learn relationships between shapes and shape distributions in TCs. This T-DSMIL has outperformed the original DSMIL and another vision transformer MIL architecture (TransMIL) in compound classification. Furthermore, T-DSMIL can be employed to identify the precise phenotypes aligned with each specific drug treatment. This strategic alignment is proposed to enable the filtration of uninformative cell shapes, allowing a concentrated focus on the phenotypes distinctly associated with each treatment. More so, T-DSMIL could accurately predict drug targets, ERK signalling dynamics, and single-cell response to chemical compounds.

[0094] While our results are promising, we acknowledge certain limitations. The specificity of our target classification using T-DSMIL warrants a cautious approach to generalising these findings. Future research should focus on validating this method across diverse cell types and conditions to establish its broad applicability. The field of GDL is rapidly evolving, and future work could explore the efficacy of other emerging GDL models in capturing 3D cell morphology, including contrastive learning to obtain richer 3D shape representations. Given the study's findings on the influence of the environment on cell shape, future work could delve into incorporating this feature into the models, enhancing prediction accuracy.

[0095] The fusion of 3D shape analysis using GDL with the innovative techniques of T-DSMIL sets the stage for a transformative approach in phenotypic drug screening. This combination promises more accurate, efficient, and insightful drug discovery endeavours. Furthermore, by packaging our methods in an open-source Python package, we offer the scientific community a tool that is not only adaptable but also continuously evolving, allowing for further research in connecting 3D cell shapes to state across various domains. This study represents a significant step forward in the field of 3D cell shape analysis, opening new avenues for understanding cellular behaviour in health and disease.

[0096] Methods

[0097] Biological Preparatation

[0098] We used three new datasets in this study. The first is the small-molecule screen imaged on the ssOPM, the second is the RNAi screen imaged on the ssOPM, and the third is the RNAi screen for phosphorylated ERK, imaged

[0099] Small molecule dataset Cells used in this study were WM266.4 harbouring CAAX-EGFP (donated from the Marshall lab), with the addition of an ERK-KTR-Ruby construct (addgene #90231) and a Histone2B- iRFP670 construct (Addgene #90237).

[0100] Collagen hydrogels were prepared to a final concentration of 2mg / mL. Briefly, hydrogel solutions of dH2O, 5xDMEM, HEPES (7.5 pH), and Rat Tail Collagen IV (Corning) were prepared on ice to a final collagen concentration of 2mg / mL. Cells were re-suspended in hydrogel solution at a concentration of 4 E4 cells per 100 pL, and 100 pL of this solution was dispensed into each experimental well on a 96-well plate. After dispensing cells, the plates were incubated at 37 degrees Celsius for 1 hour, and 100 pL of DMEM was added to each well.

[0101] Treatments were added to cells 24 hours after seeding in collagen hydrogels. After 6 hours of treatment, cells were fixed in 4 % paraformaldehyde for 30 minutes at room temperature. Final concentrations for treatments were: Binimetinib (2 pM), Palbociclib (2 pM), MK1775 (1 pM), Blebbistatin (10 pM), H1152 (10 pM), PF228 (2 pM), CK666 (100 pM), Nocodazole (1 pM) and DMSO 1 in 1000. Concentrations were calculated, including the 100 pl volume of the collagen hydrogel. These cells were imaged using OPM.

[0102] High throughput RNA interference screening

[0103] RNA interference (RNAi) screens were performed in 384-well Cell Carrier plates (PerkinElmer) to which 40 nL / well small interfering RNA (siRNA) (20 pM) were plated using an Echo liguid handler (LabCyte). Before seeding cells, 10 pL of OptiMEM (Invitrogen) containing 40 nL / well Lipofectamine RNAiMAX (Invitrogen) was added using a Multidrop Combi Reagent Dispenser (ThermoFischer), and plates were incubated for 30 minutes at room temperature. 5000 cells per well were seeded in 20 pL of complete medium for all wells containing siRNA and a subset of control wells. After 48 hours, cells were fixed by adding 30 pL of pre-warmed 8% methanol-free paraformaldehyde (PFA)(ThermoFischer) and incubated for 15 minutes at room temperature. After washing 3 times with phosphate buffered saline (PBS), cells were permeabilised in 0.2%Triton X-100. Cells were blocked for 1 hour at room temperature with 0.5% bovine serum albumin (BSA) in PBS. After washing for three times with PBS, primary antibodies for phalloidin (to stain actin) 488 (green) were added in 10 pL block solution, and plates were sealed and incubated overnight at 4°C. Following three washes in PBS, secondary antibodies were added and incubated for 2 hours at room temperature. Plates were washed two times in PBS, incubated for 15 minutes with 5 pg / mL SiR-DNA, washed once with PBS, filled with 50 p PBS, and sealed for imaging. These cells were imaged using OPM. RhoGEFs and RhoGAPs phenotypic screen for phosphorylated ERK levels in melanoma cells

[0104] To determine whether RhoGEFs or RhoGAPs contribute to ERK activity and potentially ERK- mediated cell shape determination, we depleted 168 siRNAs fromWM266.4 BRAFV600D mutant cells and measured changes in basal ERK activity. The mitogen-activated protein kinase (MAPK) pathway regulates cell proliferation, apoptosis and survival

[0084] . RAS protein is activated by growth factor receptors and recruits RAF kinases, activating MEK1 / 2 and ERK1 / 2 via phosphorylation

[0085] . When MEK is phosphorylated, it then phosphorylates and activates ERK, which translocates into the nucleus

[0084] . WM266.4 harbour either the BRAFV600E or BRAFV600D mutation, which results in constitutive active ERK that drives WM266.4 proliferation and is implicated in tumour progression [86, 87].

[0105] RNAi screens were performed in 384-well Cell Carrier plates (PerkinElmer) to which 40 nL / well siRNA (20 pM) were plated using an Echo liquid handler (LabCyte). Prior to seeding cells, 10 pL of OptiMEM (Invitrogen) containing 40 nL / well Lipofectamine RNAiMAX (Invitrogen) was added using a Multidrop Combi Reagent Dispenser (ThermoFischer) and plates were incubated for 30 minutes at room temperature. For all wells containing siRNA and a subset of control wells, 5000 cells / well were seeded in 20 pL of complete medium. After 48 hours cells were fixed by adding 30 pL of pre-warmed 8% PFA (methanol free) (ThermoFischer), and incubated for 15 minutes at room temperature. After washing 3X with PBS, cells were permeabilised in 0.2% Triton X-100. Cells were blocked for 1 hour at room temperature with 0.5% BSA in PBS. After washing 3X with PBS, primary antibody was added in 10 pL block solution, and plates were sealed and incubated overnight at 4°C. Following 3X washes in PBS, secondary antibody and phalloidin were added and incubated for 2 hours at room temperature. Plates were washed 2X in PBS, incubated for 15 mins with 5 pg / mL Hoechst, washed IX with PBS, filled with 50 pL PBS, and sealed for imaging. Cells were imaged in the OPERA QEHS (PerkinElmer), with a 20x air immersion objective, on single plane.

[0106] Oblique Plane Microscopy

[0107] The Small molecule screen and the RNAi screen were imaged using OPM. Stage scanning OPM imaging was performed on a modified version of the OPM system described in [25, 30]. The primary microscope objective was a 60X / 1.2NA water immersion objective, and the secondary 373 objective was a 50X / 0.95NA air objective, and the tertiary objective was a 40x / 0.6NA air objective. The OPM angle was 35 degrees. A single sCMOS camera (pco Edge) was used in Global Reset acquisition mode with 1280 x 1000 pixels. A motorised filter wheel (FW103H / M, Thorlabs) was used to switch between filters for multichannel imaging. An OPM volume was acquired for iRFP (642 nm excitation and 731 / 137 emission filter (Semrock Brightline)) before the stage returned to the start position before the acquisition of the EGFP volume (488 nm excitation and 550 / 49 emission filter (Semrock Brightline)). Finally, a collagen scattered light volume (488 nm illumination and no emission filter) was acquired from the same start position. The laser illumination and camera exposure time were both 4 ms. The stage velocity was 0.16 pm / ms and image acquisition was triggered every 1.4 pm of stage travel. For each field of view, the x-y stage covered 4000 pm, and three regions were imaged for each well. Before analysis, raw frames were compressed using Jetraw compression software (Jetraw, Dotphoton). Volumes were then de-skewed into standard xyz coordinates

[0025] and binned such that the final voxel size was lxlxl pm3. Image reslicing was performed using bi-linear resampling similar to the methods described in

[0030] .

[0108] Image Processing

[0109] Segmentation

[0110] The OPM image acquisition produces a parallelepiped-shaped volume due to the lightsheet angle. Before segmentation, the tips of this parallelepiped were cropped out in the y direction. This process removed sections of the volume that were not imaged over the entire axial extent. The collagen channel was viewed manually before segmentation, and volumes where the collagen was absent throughout the whole volume, were rejected.

[0111] Two 3D segmentation methods were used in this paper to verify the robustness of our approach. These included Ostu's thresholding and active contours. These methods calculate a binary mask of the region of interest with minimal user input. We used both methods to segment both cells and nuclei in 3D. Otsu's thresholding was computed automatically for each field of view. The threshold for the nucleus was computed using Otsu's method with a single-level (multithresh, Image Processing Toolbox, MATLAB). There were cases in the data where brightly fluorescent cell membranes with dimly fluorescent protrusions existed. To include all these parts of the cell in the output segmentation, the lowest threshold found by a two-level Otsu method was used. For the active contours method, an initial guess of the segmentation mask was generated using a threshold value slightly above the intensity of the background (a value of 5). The final segmentation output was then calculated after 500 and 1000 iterations for nuclei and cells, respectively, of the active contours method using default parameters (Image Processing Toolbox, MATLAB). Touching nuclei were separated following a two-step process. First, an Euclidean distance transform of the inverse of the mask was calculated to determine the distance of each voxel to the edge of the 3D mask (bwdist, Image Processing Toolbox, MATLAB). Next, a watershed method (Image Processing Toolbox, MATLAB) was applied to the negative of the distance-transformed image. These steps separated nuclei based on regions where the masks began to narrow. Nuclei volumes less than 50 pm3 were rejected at this stage.

[0112] Any connected components in the cell binary mask which did not contain a nucleus were rejected. Touching cells were separated using a marker-based watershed approach. Here, the nuclei are set as the low points (value of 0), the cell cytoplasm is set as intermediate points (value of 1), and the background is set as high points (value of digital infinity). This watershed method finds the halfway point between the nuclei of touching cells. Following segmentation, cells touching the volume edges were removed. Cells with volumes less than 512 pm3 were removed. The nuclei of rejected cells and cells of rejected nuclei were removed. We further tested Cellpose

[0088] , a deep learning-based segmentation method for a wide range of microscopy images. However, after manually inspecting the results, we found that our procedure involving Ostu's thresholding and active contour with further pre- and post-processing worked better on our dataset. This was primarily due to the cases where there were brightly fluorescent cell membranes with dimly fluorescent protrusions.

[0113] There were cases where our watershed algorithms failed, and some regions of interest (ROI) contained multiple cells. As an extra filtering step, we re-segmented the nuclei channel for each ROI. If there were more than one nucleus mask from our new segmentation that was in the same voxel locations as the cell mask from the first cell segmentation, we removed this ROI from the dataset entirely. The segmentations were manually checked for several "difficult" cells, and we are confident that our method is sufficiently robust.

[0114] Outlier removal

[0115] The three repeated experiments produced a segmented dataset of over 65000 single cells. Similar to

[0030] , we removed cells and nuclei that expressed low CAAX-GFP transgene, as this caused inaccurate segmentation. We automatically removed cells with a maximum intensity less than the mean of all cell maximum intensities minus the standard deviation of all cell maximum intensities. After the above outlier removal and quality control, there was an additional round of quality control during the data analysis stage. Cells with a mean CAAX-GFP intensity of fewer than 70 units were removed from the study, and cells with a mean nuclear Histone-2B of fewer than five units were also removed from the study.

[0116] Classical shape measurements

[0117] Similar to our previous works

[0030] , cell and nuclei classical shape measurements were calculated by the regionprops3 function in MATLAB. Further measurements were derived from these outputs as described in

[0030] . Coverslip height was identified using a z profile of the nucleus intensity as described in

[0030] .

[0118] ERK-ratio measurements

[0119] Nuclear ERK-KTR intensity was defined as the mean ERK-KTR intensity inside the nucleus mask. An ERK-KTR ring region was then calculated by expanding the nuclear mask by a fixed amount (Figure 4 D). The ERK-ratio was then calculated as: mean nuclear ERK intensity

[0120] ERK Ratio = - mean ring region ERK intensity

[0121] We followed the same procedure for calculating the pERK-ratio. We selected hit siRNAs when more than 3 replicates had a z score = ± 1.27 threshold for pERK in the nuclear ring region.

[0122] Point cloud construction

[0123] Point cloud representations of 3D objects have been used extensively in 3D shape analysis using deep learning [78, 89, 90, 91, 92]. Point clouds are arguably one of the simplest forms of shape representation and are easily obtainable from mesh objects - a common output of segmentation methods. We followed a two-step procedure to create point cloud representations of cell and nuclei surfaces. First, we converted the 3D binary mask to a mesh object. We used the marching cubes algorithm

[0093] from scikit-learn [94, 95] to extract vertices, faces and normals from the 3D binary mask. Trimesh

[0096] was used to form a mesh object from these vertices, faces and normals of cell and nuclei surfaces. The marching cubes algorithm typically produces meshes with poor triangular quality

[0097] . When dealing with these meshes with poor triangular quality meshes, k-nearest neighbours graph constrictions (used in EdgeConv of our DFN autoencoder) may not give an equilateral representation of local features with irregular distances between vertices. This may adversely affect results through the topology of the 3D shape not being correct. We, therefore, used Laplacian smoothing

[0098] to obtain equilateral representations of local features. Second, we uniformly sampled points from the surface of the mesh object to create a point cloud representing the cell's shape or nuclei.

[0124] The number of points used to represent our shape data is a chosen parameter of our methods that essentially represents the resolution of our data. Since there is a large size discrepancy across the cell population, using the same number of points for all cells may lead to larger cells being under-sampled and smaller cells being over-sampled. However, several works on 3D shape understanding have used datasets of point cloud representations of objects ranging in scales from chairs to aeroplanes, which use the same number of points for all objects [40, 99, 100, 43]. A trade-off exists between the resolution of the input point cloud and the speed of training and inference of our deep learning model. We thus needed a suitable number of points representing the input data across scales and being computationally efficient. To this end, to represent each shape, we tested three different orders of magnitude of point sampling frequencies, i.e., 1024, 2048, and 4096 points. To compare the different sampling frequencies, we visually inspected the point clouds of cells to ensure intricate cellular protrusions were being captured (Figure 1 D). Furthermore, we compared the accuracies of our classification tasks using features generated from shapes represented by all three scales of point densities to show that all sampling densities were similar in terms of classification accuracy (Table 1). Ultimately, we found that 2048 points were sufficient in representing both large and small cells and were computationally efficient. We did not find cases where cell protrusions were missed with this sampling frequency. While it may be true that extremely fine details are overlooked by point cloud representations (details such as texture), global and local neighbourhood shapes are well represented by 2048 points for our datasets.

[0125] Functions from PyTorch Geometric

[0101] were used to uniformly sample points from the surface of the mesh object. We packaged our point cloud generation pipeline in a Python package called cellshape-helper.

[0126] Aligning 3D shapes to a common axis

[0127] As a step towards rotational invariance, we followed methods described in

[0102] to convert the point cloud representations of 3D cells to their PCA-based canonical poses. PCA calculates three orthogonal bases or principal axes of point cloud data. This enables us to align original point clouds to the world Cartesian plane

[0102] . Following directly from

[0102] , we performed PCA on a given point cloud, S e Rnx3rby Equation 2: where Ste IR3is the ith point of S,S e lR3is the mean of S,E is the eigenvector matrix composed of eigenvectors (el, e2, e3) (principal axes), and A = diag (Ai, A2, A3) are the corresponding eigenvalues. By aligning the principal axes to the three axes of the world coordinate, we obtain the canonical pose as Scan = SE.

[0102] proved the rotational invariant property of Scan. We calculated the canonical poses for each point cloud representation.

[0128] Machine learning

[0129] Dynamic graph convolutional foldinqnet autoencoder

[0130] The DFN autoencoder follows the design of the FoldingNet

[0053] with a dynamic graph convolutional neural network (DGCNN) as the encoder

[0040] . This encoder takes a 3D point cloud as input, constructs a local neighbourhood (k-nearest neighbour) graph on these points and applies convolution-like operations on the edges of connecting neighbouring points (EdgeConv). The authors

[0040] show translation-invariant properties of EdgeConv operations. Furthermore, other works

[0103] and our experiments found that using the DGCNN encoder outperformed the original FoldingNet encoder regarding transfer classification accuracy on ModelNet40

[0037] .

[0131] The choice of k is a hyperparameter of the model architecture, and the original DGCNN

[0040] used a value of 20 for k when using 1024 points and a value of 40 when using 2048 points for the input point cloud representation of everyday objects from ModelNet40

[0037] . A suitable value of k was considered through experimentation. A value of k too large would cause each point along the cell's surface to be connected to too many neighbouring points, resulting in redundant information and a loss of local structure. Too small a value of k will result in the model degenerating into a convolutional neural network. Furthermore, larger values for k require longer training and inference times. We tested a range of different values for k to find a value that was both computationally efficient and produced the best results on our classification tasks.

[0132] We replaced the final linear layer from the original DGCNN architecture with one that outputs a desired feature vector length. The decoder takes the feature vector, z, as input and concatenates it with "source points". Similar to efforts in

[0103] , we explored various source points. These included points sampled from a 2D plane, a Gaussian distribution or a sphere in 3D space. The feature vector concatenated with the source points is then passed through two folding operations (defined in

[0053] ) to output a reconstructed point cloud. Our model outputs 2025 points. The number of reconstructed points does not need to be the same as the input number of points. We used the extended Chamfer distance (CD) presented in

[0053] as our reconstruction error between input point cloud S and reconstructed point cloud S. The extended CD is defined in Equation 3.

[0133] Initially, the DFN was trained on the ShapeNet dataset

[0038] for 250 epochs. Then, we continued training on our point cloud representations of cells and nuclei for another 250 epochs using Adam optimiser with e-6weight decay. We used a batch size of 16 with an initial learning rate of 0.0001 and an exponential learning rate decay scheduler. The DFN model was set to extract 128 features from each point cloud. All algorithms were implemented in PyTorch. We used code from [103, 54] and packaged our DFN into a Python package called cellshape-cloud.

[0134] Classification tasks using support vector machines

[0135] We used our extracted features to train a support vector machine (SVM). For all experiments, we used one-versus-one SVM classifiers with a radial basis function kernel and an L2 regularisation parameter (C = 5), balanced class weights, and intercept scaling during training. We used scikit-learn for these methods [94, 95]. For our drug-treated dataset and 3D red blood cell dataset, we performed 10-fold cross-validation for each experiment and reported the mean balanced accuracy. For the VesselMNIST3D dataset, we used the specified training and validation set to train our model and report on the unseen test set. We report the macro-average class accuracy for all experiments to account for class imbalances.

[0136] Multiple instance learning

[0137] Multiple instance learning (MIL) is a type of weakly supervised learning algorithm where training data is arranged in bags of instances X = {xi, X2, . . . , xn} for n instances, with each instance holding a latent label yk (yk = 1 for positive, or yk = 0 for negative), which is assumed unknown. The goal of MIL is to determine whether there is at least one positive instance in a bag or not. The only information known during training is the bag label,

[0138] In this study, we used binary MIL, therefore c = 2. In the binary MIL case, negative bags only contain negative instances, while positive bags contain at least one positive instance. One solution is to assign the bag label to each instance and train a classifier to classify each instance (instance-based). Another solution is to obtain a higher-level bag representation resulting from the aggregation of instance features (bag-embeddingbased). Most recent bag-embedding approaches utilise attention mechanisms

[0104] , which assigns an attention score to each instance demonstrating its relative contribution to the bag-level representation. These attention scores have enabled automatic localisation of cancerous regions in digital pathology tasks

[0058] , and could add value in understanding phenotypic characteristics of treatment response in microscopy images.

[0139] We employ attention-based MIL to classify treatment conditions. That is, we develop an MIL classifier for each small-molecule treatment. In each case, a bag is labelled positive if it is from a well treated with the treatment of interest and negative if not. It is proposed that each bag represents the collective response of a cell population to a particular drug, with some cells displaying phenotypes typical of that drug and others being outliers and / or noise. Our MIL pipeline, dubbed T-DSMIL, followed three steps:

[0140] 1. A cell and nuclei 3D shape feature extraction module consisting of a trained and fixed DFN encoder. This transformed point clouds of cells and nuclei into lower dimensional feature representations. Each feature vector for each cell was concatenated with the feature vector extracted from the corresponding nucleus to give final cell shape feature representations: Z = {Zi, . . . ,Zj, . . . ,ZN},Zj e Rnxd, where d is 2x the embedding dimension from the DFN (cell + nucleus feature representations), n is the number of cells within a bag (n differs among bags), and N is the number of bags.

[0141] 2. A multi-head attention transformer encoder

[0105] that transforms the feature embeddings to a descriptive hidden feature representation that aggregates global concepts within the bag population of shapes. This is essential in capturing the overall pattern of shapes among multiple cells. This also allows the model to effectively integrate cell and nuclei features to learn the relationship between them. Here the feature embeddings Z are transformed into a feature set T = {Ti, . . . , T, . . . , TN}, with each T e Rnxd. Let Q,K,V be the query, key, and value matrices computed from Zi. For each head h, we have:

[0142] Where W®, 14^ , IV / are the weight matrices for the query, key, and value, respectively, for each head. The attention for each head is computed as: where dk is the dimension of the key vectors. The outputs of all heads are concatenated and linearly transformed:

[0143] MH A = Concat(heacL., ... ,headH)W°

[0144] The output of the multi-head attention is then combined with the original input Zj and normalised:

[0145] Z- = LayerNorm(Z[ + MHA~)

[0146] A feed-forward network is applied next:

[0147] FFN = maxfO^,'^ + bj W2+ b2

[0148] Finally, the output of the feed-forward network is combined with Z- and normalised:

[0149] Tt= Layer N orm(Zt' + FFN)

[0150] Here, T is the output of the Transformer encoder layer, having the same dimensions as the input Zj. These transformed features then go through a dual-stream MIL model

[0068] . The first stream involves an instance classifier that takes a bag of features of instances, Zj, and classifies each instance, followed by a max pooling on the scores: where Wo is a weight vector. The second stream involves a bag-level classifier that aggregates the instance embeddings into a bag embedding and classifies the bag. A critical instance zmof each bag is obtained from the instance classifier (that with the highest classification score), and is transformed into query (q, and value representations (v; such that: Where Wqand Wvare weight matrices. We then define the distance between an arbitrary instance and the critical instance as: with .) being the inner product of two vectors. The bag classification score from this second stream is given by:

[0151] Where Wb is a weight vector for binary classification, in our case. The bag embedding b in Equation 16 is similar to self attention, except for the fact that the query-key matching is only done between critical instances and other instances. Furthermore, each query is matched with other queries, and no key vector is learned, unlike self attention where each query is matched with an additional key vector.

[0152] 5. The final bag-level classification is the weighted sum of the instance-level and bag-level classifiers:

[0153] We compared T-DSMIL to several MIL models, including the original DSMIL, TransMIL

[0069] (a vision transformer based multiple instance learning architecture), mean pooling, max pooling and log-sum-exponent pooling. TransMIL and other pooling methods were trained using a binary cross-entropy loss on the bag prediction. DSMIL and T-DSMIL were trained using a weighted binary cross entropy loss between the bag-level and instance-level classifier. All methods were optimised using Adam optimiser with a weight-decay of le-4. We used an initial learning rate of 0.0001 with cosine annealing learning rate scheduling. All models were trained for a maximum of 250 epochs. We used early stopping to avoid over-fitting, with a patience of 50 epochs while monitoring the validation loss. We used a 10-fold cross-validation with a train / validation / test split of 50 / 20 / 30.

[0154] For the attention-based MIL algorithms (DSMIL, T-DSMIL, TransMIL), we utilised the attention scores to identify the most important cells, or 'key instances', driving the collective response to each drug. For DSMIL and T-DSMIL, we used the attention from the bag-level classifier. For TransMIL, we used the attention values of class tokens in the self-attention matrix to the rest of the feature tokens, just as the authors describe. These key instances help highlight the dominant phenotypic traits induced by each drug. Attention scores were normalised on a bag level such that anew= aa~amin, where a is the original attention maxamin coefficient for a particular instance, amin is the minimum and amax is the maximum attention coefficient in the bag.

[0155] When conducting inference on the RNAi dataset, we used MIL models trained using cells in the drug-treated dataset that were on the coverslip (labelled as proximal in previous sections), and therefore, only looking at cells in the same environment across experiments. Here, we generated point clouds of cells from the RNAi dataset and extracted 3D cell and nuclei-shape features using a pre-trained DFN autoencoder (trained on the drug-treated cells). This produced a feature vector for every cell in the RNAi dataset. We then created multiple bags of instances by grouping cells with thesame RNAi condition. These bags were then fed through pre-trained T-DSMIL classifiers (a separate model for each drug trained to classify that drug vs others and trained on only the cells proximal to the coverslip in the drug-treated dataset). This produced a score of how similar each RNAi is to each drug. We did this across ten different folds of training, validation, and testing splits, and we took the average results of the 10 folds. All MIL algorithms were implemented in PyTorch.

[0156] 3D rendering

[0157] 3D renders of intensity images are generated as a 3D projection with trilinear interpolation by using the volume viewer 2.01 Fiji plugin

[0106] . 3D renders of cell masks are presented as 3D surface representations using isosurfaces in napari

[0107] . Point cloud renderings of 3D cell shapes were done using Mitsuba2

[0108] and adapted scripts from

[0109] . Mesh renders were done using Blender v3.6.5

[0110] ..

[0158] System

[0159] Figure 6 is a block diagram illustrating an arrangement of a system according to an embodiment of the present invention. Some embodiments of the present invention are designed to run on general purpose desktop or laptop computers. Therefore, according to a first embodiment, a computing apparatus 600 is provided having a central processing unit (CPU) 606, and random access memory (RAM) 604 into which data, program instructions, and the like can be stored and accessed by the CPU. The apparatus 600 is provided with a display screen 620, and input peripherals in the form of a keyboard 722, and mouse 724. Keyboard 722, and mouse 624 communicate with the apparatus 600 via a peripheral input interface 608. Similarly, a display controller 602 is provided to control display 620, so as to cause it to display images under the control of CPU 606. 3D cell 630, 2D cell 632 and cell population data 634, can be input into the apparatus and stored via the data input 610. In this respect, apparatus 600 comprises a computer readable storage medium 612, such as a hard disk drive, writable CD or DVD drive, zip drive, solid state drive, USB drive or the like, upon which the 3D cell 630, 2D cell 632 and cell population data 634 are stored. Other datasets relating to 3D and 2D cell shape and cell population of cells that have been treated with a known drug such as the small molecule dataset 6122 and one or more RNAi datasets 6124, can also be stored. Alternatively, the data 630, 632, 634 could be stored on a web-based platform, e.g. a database (e.g. XNAT), and accessed via an appropriate network. Computer readable storage medium 612 also stores various programs, which when executed by the CPU 606 cause the apparatus 600 to operate in accordance with some embodiments of the present invention.

[0160] In particular, a control interface program 616 is provided, which when executed by the CPU 606 provides overall control of the computing apparatus, and in particular provides a graphical interface on the display 620, and accepts user inputs using the keyboard 622 and mouse 624 by the peripheral interface 608. The control interface program 616 also calls, when necessary, other programs to perform specific processing actions when required. The received 3D cell data 630, the 2D cell data 632 and the cell population data 634 provides the fundamental data required to analyse the cells in questions. The further drug datasets i.e., the small molecule dataset 6122 and the one or more RNAi datasets 6124 are used on the previously analyzed cells to aid classification of new drugs.

[0161] In particular the attention based multiple instance learning model 614 comprises a plurality of programs or algorithms that are utilized to analyse cells and then classifying their shapes at the cell level and the bag level to determine a potential drug label. Specifically, the attention based multiple learning model 614 comprises classifiers 6142, which includes the cell-level classifier and the bag classifier, a transformer encoder program 6144 and a pretrained DFN program 6146. The pretrained DFN program 6146 is used to extract cell features and nucleus features from the cells in a well images i.e., the data relating to the 3D / 2D cell shape and the cell population. The transformer-based encoder program 6144 is used to convert the extracted cell and nucleus features into a descriptive hidden feature representation which aggregates global concepts with the bag population of shapes i.e., the 3D cell data 630, the 2D cell data 632 and the cell population data 634.

[0162] The detailed operation of the computing apparatus 500 has been described in more detail in relation to Figs. 1-5 above.

[0163] Unless the context clearly requires otherwise, throughout the description and the claims, the words "comprise," "comprising," "include," "including," and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of "including, but not limited to."

[0164] The words "coupled" or "connected" or "tied", as generally used herein, refer to two or more elements or nodes that may be either directly connected, or connected by way of one or more intermediate elements. Additionally, the words "herein," "above," "below," and words of similar import, when used in this application, shall refer to this application as a whole and not to any particular portions of this application. Where the context permits, words in the Detailed Description using the singular or plural number may also include the plural or singular number, respectively. The words "or" in reference to a list of two or more items, is intended to cover all of the following interpretations of the word : any of the items in the list, all of the items in the list, and any combination of the items in the list.

[0165] Various modifications, whether by addition, substitution, or deletion will be apparent to the intended reader to provide further embodiments of the present disclosure, any and all of which are intended to be encompassed by the appended claims.

[0166] References

[0167] [1] D. W. Thompson. On growth and form, 2nd ed. On growth and form, 2nd ed. Macmillan, Cambridge University Pre, Oxford, England, 1942. Pages: ii, 1116.

[0168] [2] Sean Porazinski, Huijia Wang, Yoichi Asaoka, Martin Behrndt, Tatsuo Miyamoto, Hitoshi Morita, Shoji Hata, Takashi Sasaki, S. F. Gabriel Krens, Yumi Osada, Satoshi Asaka, Akihiro Momoi, Sarah Linton, Joel B. Miesfeld, Brian A. Link, Takeshi Senga, Atahualpa Castillo- Morales, Araxi O. Urrutia, Nobuyoshi Shimizu, Hideaki Nagase, Shinya Matsuura, Stefan Bagby, Hisato Kondoh, Hiroshi Nishina, Carl-Philipp Heisenberg, and Makoto Furutani- Seiki. YAP is essential for tissue tension to ensure vertebrate 3D body shape. Nature, 521(7551):217-221, May 2015. Number: 7551 Publisher: Nature Publishing Group.

[0169] [3] Pei-Hsun Wu, Daniele M. Gilkes, Jude M. Phillip, Akshay Narkar, Thomas Wen-Tao Cheng, Jorge Marchand, Meng-Horng Lee, Rong Li, and Denis Wirtz. Single-cell morphology encodes metastatic potential. Science Advances, 6(4):eaaw6938, January 2020. Publisher: American Association for the Advancement of Science.

[0170] [4] M.D. J. Muller. On the nature and structure of cancer, and of those morbid growths which may be confounded with it. translated from the german by Charles west, m.d. with numerous plates and engravings, part i. BMJ, sl-l(4):67-67, 1840.

[0171] [5] Chris Bakal, John Aach, George Church, and Norbert Perrimon. Quantitative morphological signatures define local signaling networks regulating cell morphology. Science (New York, N.Y.), 316(5832): 1753-1756, June 2007. Number: 5832.

[0172] [6] Zheng Yin, Amine Sadok, Heba Sailem, Afshan McCarthy, Xiaofeng Xia, Fuhai Li, Mar Arias Garcia, Louise Evans, Alexis R. Barr, Norbert Perrimon, Christopher J. Marshall, Stephen T. C. Wong, and Chris Bakal. A screen for morphological complexity identifies regulators of switch-like transitions between discrete cell shapes. Nature Cell Biology, 15(7):860-871, July 2013.

[0173] [7] Sam Cooper, Amine Sadok, Vicky Bousgouni, and Chris Bakal. Apolar and polar transitions drive the conversion between amoeboid and mesenchymal shapes in melanoma cells. Molecular biology of the cell, 26(22):4163-4170, 11 2015.

[0174] [8] Julia E. Sera, Heba Zuhair Sailem, Rico Chandra Ardy, Hannah Almuttaqi, Tongli Zhang, and Chris Bakal. Cell shape and the microenvironment regulate nuclear translocation of NF-B in breast epithelial and tumor cells. Molecular Systems Biology, ll(3):790, March 2015.

[0175] [9] Heba Z. Sailem, Julia E. Sera, and Chris Bakal. Visualizing cellular imaging data using phenoplot. Nature Communications, 6(1):5825, 2015.

[0176]

[0010] Judith M. S. Prewitt and Mortimer L. Mendelsohn. The analysis of cell images*. Annals of the New York Academy of Sciences, 128(3): 1035-1053, 1966.

[0177]

[0011] Claude Lesty, Martine Raphael, Luc Nonnenmacher, and Jacques-Louis Binet. Two statistical approaches to nuclei shape and size in a morphometric description of lymph node sections in non-hodgkin's lymphoma. Cytometry, 10(l):28-36, 1989.

[0178]

[0012] CH Thomas, JH Collier, CS Sfeir, and KE Healy. Engineering gene expression and protein synthesis by modulation of nuclear shape. Proceedings of the national academy of sciences of the United States of America, 99(4): 1972-1977, FEB 19 2002.

[0179]

[0013] Veronika Graml, Xenia Studera, Jonathan L.D. Lawson, Anatole Chessel, Marco Geymonat, Miriam Bortfeld- Miller, Thomas Walter, Laura Wagstaff, Eugenia Piddini, and Rafael E. Carazo-Salas. A genomic multiprocess survey of machineries that control and link cell shape, microtubule organization, and cell-cycle progression. Developmental Cell, 31(2):227-239, 2014.

[0180]

[0014] Saurav Basu, Soheil Kolouri, and Gustavo K. Rohde. Detecting and visualizing cell phenotype differences from microscopy images using transport-based morphometry. Proceedings of the National Academy of Sciences, lll(9):3448-3453, 2014.

[0015] Thouis R. Jones, Anne E. Carpenter, Michael R. Lamprecht, Jason Moffat, Serena J. Silver, Jennifer K. Grenier, Adam B. Castoreno, Ulrike S. Eggert, David E. Root, Polina Golland, and David M. Sabatini. Scoring diverse cellular morphologies in image-based screens with iterative feedback and machine learning. Proceedings of the National Academy of Sciences, 106(6) : 1826-1831, 2009.

[0181]

[0016] Anne E. Carpenter, Thouis R. Jones, Michael R. Lamprecht, Colin Clarke, In Han Kang, Ola Friman, David A. Guertin, Joo Han Chang, Robert A. Lindquist, Jason Moffat, Polina Golland, and David M. Sabatini. Cellprofiler: image analysis software for identifying and quantifying cell phenotypes. Genome Biology, 7(10) : R100, 2006.

[0182]

[0017] Lee Kamentsky, Thouis R. Jones, Adam Fraser, Mark-Anthony Bray, David J. Logan, Katherine L. Madden, Vebjorn Ljosa, Curtis Rueden, Kevin W. Eliceiri, and Anne E. Carpenter. Improved structure, function and compatibility for CellProfiler: modular high- throughput image analysis software. Bioinformatics, 27(8) : 1179- 1180, 02 2011.

[0183]

[0018] Claire McQuin, Allen Goodman, Vasiliy Chernyshev, Lee Kamentsky, Beth A. Cimini, Kyle W. Karhohs, Minh Doan, Liya Ding, Susanne M. Rafelski, Derek Thirstrup, Winfried Wiegraebe, Shantanu Singh, Tim Becker, Juan C. Caicedo, and Anne E. Carpenter. Cellprofiler 3.0: Next-generation image processing for biology. PLOS Biology, 16(7) : 1-17, 07 2018.

[0184]

[0019] Erik Sahai and Christopher J. Marshall. Differing modes of tumour cell invasion have distinct requirements for Rho / ROCK signalling and extracellular proteolysis. Nature Cell Biology, 5(8) :711-719, August 2003.

[0185]

[0020] Julia E. Sero and Chris Bakal. Multiparametric Analysis of Cell Shape Demonstrates that -PIX Directly Couples YAP Activation to Extracellular Matrix Adhesion. Cell Systems, 4(l) :84-96.e6, January 2017. Number: 1.

[0186]

[0021] Patricia Pascual-Vargas, Samuel Cooper, Julia Sero, Vicky Bousgouni, Mar Arias- Garcia, and Chris Bakal. RNAi screens for Rho GTPase regulators of cell shape and YAP / TAZ localisation in triple negative breast cancer. Scientific Data, 4: 170018, March 2017.

[0187]

[0022] Meghan K. Driscoll, 660 Erik S. Welf, Andrew R. Jamieson, Kevin M. Dean, Tadamoto Isogai, Reto Fiolka, and Gaudenz Danuser. Robust and automated detection of subcellular morphological motifs in 3D microscopy images. Nature Methods, 16(10) : 1037-1044, October 2019.

[0188]

[0023] Ian Jones, Lucas Dent, Tomoaki Higo, Theodoras Roumeliotis, Maria Arias Garcia, Hansa Shree, Jyoti Choudhary, Malin Pedersen, and Chris Bakal. Characterization of proteome-size scaling by integrative omics reveals mechanisms of proliferation control in cancer. Science Advances, 9(4) :eadd0636, 2023.

[0189]

[0024] C. Dunsby. Optically sectioned imaging by oblique plane microscopy. Optics Express, 16(25) :20306-20316, December 2008.

[0025] Vincent Maioli, George Chennell, Hugh Sparks, Tobia Lana, Sunil Kumar, David Carling, Alessandro Sardini, and Chris Dunsby. Time-lapse 3-d measurements of a glucose biosensor in multicellular spheroids by light sheet fluorescence microscopy in commercial 96-well plates. Scientific Reports, 6(1) :37777, Nov 2016.

[0190]

[0026] Alexandr A. Kalinin, Xinhai Hou, Alex S. Ade, Gordon-Victor Fon, Walter Meixner, Gerald A. Higgins, Jonathan Z. Sexton, Xiang Wan, Ivo D. Dinov, Matthew J. O'Meara, and Brian D. Athey. Valproic acid-induced changes of 4D nuclear morphology in astrocyte cells. Molecular Biology of the Cell, 32(18) : 1624-1633, August 2021. Publisher: American Society for Cell Biology (mboc).

[0191]

[0027] A. J. Lomakin, C. J. Cattin, D. Cuvelier, Z. Alraies, M. Molina, G. P. F. Nader, N. Srivastava, P. J. Saez, J. M. Garcia-Arcos, I. Y. Zhitnyak, A. Bhargava, M. K. Driscoll, E. S. Welf, R. Fiolka, R. J. Petrie, N. S. De Silva, J. M. Gonzalez-Granado, N. Manel, A. M. Lennon- Dumenil, D. J. Muller, and M. Piel. The nucleus acts as a ruler tailoring cell responses to spatial constraints. Science, 370(6514) :eaba2894, 2020.

[0192]

[0028] Pedro Gomez-Galvez, Pablo Vicente-Munuera, Antonio Tagua, Cristina Forja, Ana M. Castro, Marta Letran, Andrea Valencia-Exposito, Clara Grima, Marina Bermudez-Gallardo, Oscar Serrano-Perez-Higueras, Florencia Cavodeassi, Sol Sotillos, Maria D. Martin- Bermudo, Alberto Marquez, Javier Buceta, and Luis M. Escudero. Scutoids are a geometrical solution to three-dimensional packing of epithelia. Nature Communications, 9(1) : 1-14, July 2018. Number: 1 Publisher: Nature Publishing Group.

[0193]

[0029] Matheus P. Viana, Jianxu Chen, Theo A. Knijnenburg, Ritvik Vasan, Calysta Yan, Joy E. Arakaki, Matte Bailey, Ben Berry, Antoine Borensztejn, Eva M. Brown, Sara Carlson, Julie A. Cass, Basudev Chaudhuri, Kimberly R. Cordes Metzler, Mackenzie E. Coston, Zach J. Crabtree, Steve Davidson, Colette M. DeLizo, Shailja Dhaka, Stephanie Q. Dinh, Thao P. Do, Justin Domingus, RoryM. Donovan-Maiye, Alexandra J. Ferrante, Tyler J. Foster, Christopher L. Frick, Griffin Fujioka, Margaret A. Fuqua, Jamie L. Gehring, Kaytlyn A. Gerbin, Tanya Grancharova, Benjamin W. Gregor, Lisa J. Harrylock, Amanda Haupt, Melissa C. Hendershott, Caroline Hookway, Alan R. Horwitz, H. Christopher Hughes, Eric J. Isaac, Gregory R. Johnson, Brian Kim, Andrew N. Leonard, WinnieW. Leung, Jordan J. Lucas, Susan A. Ludmann, Blair M. Lyons, Haseeb Malik, Ryan McGregor, Gabe E. Medrash, Sean L. Meharry, Kevin Mitcham, Irina A. Mueller, Timothy L. Murphy-Stevens, Aditya Nath, Angelique M. Nelson, Sandra A. Oluoch, Luana Paleologu, T. Alexander Popiel, Megan M. Riel-Mehan, Brock Roberts, Lisa M. Schaefbauer, Magdalena Schwarzl, Jamie Sherman, Sylvain Slaton, M. Filip Sluzewski, Jacqueline E. Smith, Youngmee Sul, Madison J. Swain- Bowden, W. Joyce Tang, Derek J. Thirstrup, Daniel M. Toloudis, Andrew P. Tucker, Veronica Valencia, WinfriedWiegraebe, ThusharaWijeratna, Ruian Yang, Rebecca J. Zaunbrecher, Ramon Lorenzo D. Labitigan, Adrian L. Sanborn, Graham T. Johnson, Ruwanthi N. Gunawardane, Nathalie Gaudreault, Julie A. Theriot, and Susanne M. Rafelski. Integrated intracellular organization and its variations in human ips cells. Nature, 613(7943):345- 354, Jan 2023.

[0194]

[0030] L. G. Dent, N. Curry, H. Sparks, V. Bousgouni, V. Maioli, S. Kumar, I. Munro, C. Dunsby, and C. Bakal. Environmentally dependent and independent control of cell shape determination by rho gtpase regulators in melanoma. bioRxiv, 2021.

[0195]

[0031] Matheus P. Viana, Jianxu Chen, Theo A. Knijnenburg, Ritvik Vasan, Calysta Yan, Joy E. Arakaki, Matte Bailey, Ben Berry, Antoine Borensztejn, JacksonM. Brown, Sara Carlson, Julie A. Cass, Basudev Chaudhuri, Kimberly R. Cordes Metzler, Mackenzie E. Coston, Zach J. Crabtree, Steve Davidson, Colette M. DeLizo, Shailja Dhaka, Stephanie Q. Dinh, Thao P. Do, Justin Domingus, Rory M. Donovan-Maiye, Tyler J. Foster, Christopher L. Frick, Griffin Fujioka, Margaret A. Fuqua, Jamie L. Gehring, Kaytlyn A. Gerbin, Tanya Grancharova, Benjamin W. Gregor, Lisa J. Harrylock, Amanda Haupt, Melissa C. Hendershott, Caroline Hookway, Alan R. Horwitz, Chris Hughes, Eric J. Isaac, Gregory R. Johnson, Brian Kim, Andrew N. Leonard, Winnie W. Leung, Jordan J. Lucas, Susan A. Ludmann, Blair M. Lyons, Haseeb Malik, Ryan McGregor, Gabe E. Medrash, Sean L. Meharry, Kevin Mitcham, Irina A. Mueller, Timothy L. Murphy-Stevens, Aditya Nath, Angelique M. Nelson, Luana Paleologu, T. Alexander Popiel, Megan M. Riel-Mehan, Brock Roberts, Lisa M. Schaefbauer, Magdalena Schwarzl, Jamie Sherman, Sylvain Slaton, M. Filip Sluzewski, Jacqueline E. Smith, Youngmee Sul, Madison J. Swain-Bowden, W. Joyce Tang, Derek J. Thirstrup, DanielM. Toloudis, Andrew P. Tucker, Veronica Valencia, WinfriedWiegraebe, Thushara Wijeratna, Ruian Yang, Rebecca J. Zaunbrecher, , Graham T. Johnson, Ruwanthi N. Gunawardane, Nathalie Gaudreault, Julie A. Theriot, and Susanne M. Rafelski. Robust integrated intracellular organization of the human ips cell: where, how much, and how variable. bioRxiv, 2021.

[0196]

[0032] Christel Ducroz, Jean-Christophe Olivo-Marin, and Alexandre Dufour. Spherical harmonics based extraction and annotation of cell shape in 3d time-lapse microscopy sequences. In 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pages 6619-6622, 2011.

[0197]

[0033] Khaled Khairy and Jonathon Howard. Minimum-energy vesicle and cell shapes calculated using spherical harmonics parameterization. Soft Matter, 7:2138-2143, 2011.

[0198]

[0034] Xiongtao Ruan and Robert F Murphy. Evaluation of methods for generative modeling of cell and nuclear shape. Bioinformatics, 35(14):2475-2485, 12 2018.

[0199]

[0035] Anna Medyukhina, Marco Blickensdorf, Zoltan Cseresnyes, Nora Ruef, Jens V. Stein, and Marc Thilo Figge. Dynamic spherical harmonics approach for shape classification of migrating cells. Scientific Reports, 10(l):6072, Apr 2020.

[0200]

[0036] Henry Cavanagh, Daryan Kempe, Jessica K. Mazalo, Mate Biro, and Robert G. Endres. T cell morphodynamics reveal periodic shape oscillations in three-dimensional migration. Journal of The Royal Society Interface, 19(190):20220081, 2022.

[0037] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes, 2014.

[0201]

[0038] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv: 1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.

[0202]

[0039] Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Due Thanh Nguyen, and Sai- Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In International Conference on Computer Vision (ICCV), 2019.

[0203]

[0040] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions on Graphics (TOG), 2019.

[0204]

[0041] Cheng Zhang, Haocheng Wan, Shengqiang Liu, Xinyi Shen, and Zizhao Wu. Pvt: Pointvoxel transformer for 3d deep learning. arXiv preprint arXiv:2108.06076, 2021.

[0205]

[0042] Tiange Xiang, Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai. Walk in the cloud: Learning curves for point clouds shape analysis. In Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), pages 915-924, October 2021.

[0206]

[0043] Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: A simple residual MLP framework. In International Conference on Learning Representations, 2022.

[0207]

[0044] Peng-ShuaiWang, Yu-Qi Yang, Qian-Fang Zou, ZhirongWu, Yang Liu, and Xin Tong. Unsupervised 3D learning for shape analysis via multiresolution instance discrimination. In AAAI Conference on Artificial Intelligence (AAAI), 2021.

[0208]

[0045] Yi Shi, Mengchen Xu, Shuaihang Yuan, and Yi Fang. Unsupervised deep shape descriptor with point distribution learning. In The IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.

[0209]

[0046] JiajunWu, Chengkai Zhang, Tianfan Xue, William T Freeman, and Joshua B Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative- adversarial modeling. In Advances in Neural Information Processing Systems, pages 82- 90, 2016.

[0210]

[0047] Dominik J. E. Waibel, Niklas Kiermeyer, Scott Atwell, Ario Sadafi, Matthias Meier, and Carsten Marr. Shapr predicts 3d cell shapes from 2d microscopic images. bioRxiv, 2022.

[0211]

[0048] Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: Going beyond euclidean data. IEEE Signal Processing Magazine, 34(4): 18-42, 2017.

[0049] Yutong Feng, Yifan Feng, Haoxuan You, Xibin Zhao, and Yue Gao. Meshnet: Mesh neural network for 3d shape representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8279-8286, 2019.

[0212]

[0050] Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. Diffusionnet: Discretization agnostic learning on surfaces. ACM Trans. Graph., 41(3), mar 2022.

[0213]

[0051] Zhen Liu, Yao Feng, Michael J. Black, Derek Nowrouzezahrai, Liam Pauli, and Weiyang Liu. Meshdiffusion: Score-based generative 3d mesh modeling. In The Eleventh International Conference on Learning Representations, 2023.

[0214]

[0052] Haowen 769 Deng, Tolga Birdal, and Slobodan Ilic. Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors. In The European Conference on Computer Vision (ECCV), September 2018.

[0215]

[0053] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud autoencoder via deep grid deformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.

[0216]

[0054] Jiahao Pang, Duanshun Li, and Dong Tian. Tearingnet: Point cloud autoencoder to learn topology-friendly representations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

[0217]

[0055] Srinivas Niranj Chandrasekaran, Hugo Ceulemans, Justin D. Boyd, and Anne E. Carpenter. Image-based profiling for drug discovery: due for a machine-learning upgrade? Nature Reviews Drug Discovery, 20(2): 145-159, Feb 2021.

[0218]

[0056] Mohammad H. Rohban, Ashley M. Fuller, Ceryl Tan, Jonathan T. Goldstein, Deepsing Syangtan, Amos Gutnick, Ann DeVine, Madhura P. Nijsure, Megan Rigby, Joshua R. Sacher, Steven M. Corsello, Grace B. Peppier, Marta Bogaczynska, Andrew Boghossian, Gabrielle E. Ciotti, Allison T. Hands, Aroonroj Mekareeya, Minh Doan, Jennifer P. Gale, Rik Derynck, Thomas Turbyville, Joel D. Boerckel, Shantanu Singh, Laura L. Kiessling, Thomas L. Schwarz, Xaralabos Varelas, Florence F. Wagner, Ran Kafri, T.S. Karin Eisinger-Mathason, and Anne E. Carpenter. Virtual screening for small-molecule pathway regulators by imageprofile matching. Cell Systems, 13(9) :724-736.e9, 2022.

[0219]

[0057] Gregory P. Way, Ted Natoli, Adeniyi Adeboye, Lev Litichevskiy, Andrew Yang, Xiaodong Lu, Juan C. Caicedo, Beth A. Cimini, Kyle Karhohs, David J. Logan, Mohammad H. Rohban, Maria Kost-Alimova, Kate Hartland, Michael Bornholdt, Srinivas Niranj Chandrasekaran, Marzieh Haghighi, Erin Weisbart, Shantanu Singh, Aravind Subramanian, and Anne E. Carpenter. Morphology and gene expression profiling provide complementary information for mapping cell state. Cell Systems, 13(l l) :911-923.e9, 2022.

[0220]

[0058] Olga Fourkioti, Matt De Vries, and Chris Bakal. Camil: Context-aware multiple instance learning for cancer detection and subtyping in whole slide images, 2023.

[0059] Oren Z. Kraus, Jimmy Lei Ba, and Brendan J. Frey. Classifying and segmenting microscopy images with deep multiple instance learning. Bioinformatics, 32(12):i52-i59, 06 2016.

[0221]

[0060] Gang Fu, Xiaofei Nan, Haining Liu, Ronak Y. Patel, Pankaj R. Daga, Yixin Chen, Dawn E. Wilkins, and Robert J.Doerksen. Implementation of multiple-instance learning in drug activity prediction. BMC Bioinformatics, 13(15):S3, Sep 2012.

[0222]

[0061] William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '87, page 163-169, New York, NY, USA, 1987. Association for Computing Machinery.

[0223]

[0062] Leland Mclnnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction, 2018.

[0224]

[0063] Nicole S. Bryce, Tim W. Failes, Justine R. Stehn, Karen Baker, Stefan Zahler, Yulia Arzhaeva, Leanne Bischof, Ciaran Lyons, Irina Dedova, Greg M. Arndt, Katharina Gaus, Benjamin T. Goult, Edna C. Hardeman, Peter W. Gunning, and John G. Lock. High-content imaging of unbiased chemical perturbations reveals that the phenotypic plasticity of the actin cytoskeleton is constrained. Cell Systems, 9(5):496-507.e5, 2019.

[0225]

[0064] Shamik Sen, Adam J. Engler, and Dennis E. Discher. Matrix strains induced by cells: Computing how far cells can feel. Cellular and Molecular Bioengineering, 2(l):39-48, Mar 2009.

[0226]

[0065] Greta Simionato, Konrad Hinkelmann, Revaz Chachanidze, Paola Bianchi, Elisa Fermo, Richard van Wijk, Marc Leonetti, Christian Wagner, Lars Kaestner, and Stephan Quint. Red blood cell phenotyping from 3d confocal images using artificial neural networks. PLOS Computational Biology, 17(5): 1-17, 05 2021.

[0227]

[0066] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2 - a large-scale lightweight benchmark for 2d and 3d biomedical image classification. arXiv, 2021.

[0228]

[0067] Oded Maron and Tomas Lozano-Perez. A framework for multiple-instance learning. In M. Jordan, M. Kearns, and S. Solla, editors, Advances in Neural Information Processing Systems, volume 10. MIT Press, 1997.

[0229]

[0068] Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 14318-14328, 2021.

[0230]

[0069] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in Neural Information Processing Systems, 34:2136-2147, 2021.

[0070] Takamasa Kudo, Stevan Jekni '822 c, Derek N. Macklin, Sajia Akhter, Jacob J. Hughey, Sergi Regot, and Markus W. Covert. Live-cell measurements of kinase activity in single cells using translocation reporters, Jan 2018.

[0231]

[0071] Quintus G. Medley, Elizabeth G. Buchbinder, Kouichi Tachibana, Hai Ngo, Carles Serra- Pages, and Michel Streuli. Signaling between focal adhesion kinase and trio*. Journal of Biological Chemistry, 278(15): 13265- 13270, 2003.

[0232]

[0072] David D Schlaepfer, Satyajit K Mitra, and Dusko Ilic. Control of motile and invasive cell phenotypes by focal adhesion kinase. Biochimica et Biophysica Acta (BBA) - Molecular Cell Research, 1692(2):77-102, 2004. Cell Adhesion and Signalling.

[0233]

[0073] Brian Tran and Mark S. Cohen. The discovery and development of binimetinib for the treatment of melanoma. Expert Opinion on Drug Discovery, 15(7):745-754, 2020. PMID: 32249628.

[0234]

[0074] Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas J Guibas. Learning representations and generative models for 3d point clouds. arXiv preprint arXiv: 1707.02392, 2017.

[0235]

[0075] Yongheng Zhao, Tolga Birdal, Haowen Deng, and Federico Tombari. 3d point capsule networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

[0236]

[0076] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R. Martin, and Shi-Min Hu. Pct: Point cloud transformer. Computational Visual Media, 7(2): 187-199, Jun 2021.

[0237]

[0077] Siddharth Srivastava and Gaurav Sharma. Exploiting local geometry for feature and graph construction for better 3d point cloud processing with graph neural networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 12903- 12909, 2021.

[0238]

[0078] Seyed Saber Mohammadi, Yiming Wang, and Alessio Del Bue. Pointview-gcn: 3d shape classification with multi-view point clouds. In 2021 IEEE International Conference on Image Processing (ICIP), pages 3103-3107, 2021.

[0239]

[0079] Pierre Besson, Todd Parrish, Aggelos K. Katsaggelos, and S. Kathleen Bandt. Geometric deep learning on brain shape predicts sex and age. Computerized Medical Imaging and Graphics, 91: 101939, 2021.

[0240]

[0080] Valentyna Zinchenko, Johannes Hugger, Virginie Uhlmann, Detlev Arendt, and Anna Kreshuk. Morphofeatures for unsupervised exploration of cell types, tissues, and organs in volume electron microscopy. eLife, 12:e80918, feb 2023.

[0241]

[0081] M.E. Kelley, A.Y. Berman, D.R. Stirling, B.A. Cimini, Y. Han, S. Singh, A.E. Carpenter, T.M. Kapoor, and G.P. Way. High-content microscopy reveals a morphological signature of bortezomib resistance. bioRxiv, 2023.

[0242]

[0082] Assaf Zaritsky, Andrew R. Jamieson, Erik S. Welf, Andres Nevarez, Justin Cillay, Ugur Eskiocak, Brandi L. Cantarel, and Gaudenz Danuser. Interpretable deep learning uncovers cellular properties in label-free live cell images that are predictive of highly metastatic melanoma. Cell Systems, 12(7):733-747.e6, 2021.

[0243]

[0083] Nikita Moshkov, Tim Becker, Kevin Yang, Peter Horvath, Vlado Dancik, Bridget K. Wagner, Paul A. Clemons, Shantanu Singh, Anne E. Carpenter, and Juan C. Caicedo. Predicting compound activity from phenotypic profiles and chemical structures. Nature Communications, 14(1): 1967, Apr 2023.

[0244]

[0084] Zoi Karoulia, Yang Wu, Tamer A. Ahmed, Qisheng Xin, Julien Bollard, Clemens Krepler, Xuewei Wu, Chao Zhang, Gideon Bollag,Meenhard Herlyn, James A. Fagin, Amaia Lujambio, Evripidis Gavathiotis, and Poulikos I. Poulikakos. An integrated model of raf inhibitor action predicts inhibitor activity against oncogenic braf signaling. Cancer Cell, 30(3):485-498, 2016.

[0245]

[0085] Teresa Delgado-Goni, Maria Falck Miniotis, Slawomir Wantuch, Harold G. Parkes, Richard Marais, Paul Workman, Martin O. Leach, and Mounia Beloueche-Babari. The BRAF Inhibitor Vemurafenib Activates Mitochondrial Metabolism and Inhibits Hyperpolarized Pyruvate-Lactate Exchange in BRAF-Mutant Human Melanoma Cells. Molecular Cancer Therapeutics, 15(12):2987-2999, 12 2016.

[0246]

[0086] Maria Karasarides, Antonio Chiloeches, Robert Hayward, Dan Niculescu-Duvaz, Ian Scanlon, Frank Friedlos, Lesley Ogilvie, Douglas Hedley, Jan Martin, Christopher J. Marshall, Caroline J. Springer, and Richard Marais. B-raf is a therapeutic target in melanoma. Oncogene, 23(37):6292-6298, Aug 2004.

[0247]

[0087] David Dankort, David P. Curley, Robert A. Cartlidge, Betsy Nelson, Anthony N. Karnezis, William E. Damsky Jr, Mingjian J. You, Ronald A. DePinho, Martin McMahon, and Marcus Bosenberg. Brafv600e cooperates with pten loss to induce metastatic melanoma. Nature Genetics, 41(5): 544-552, May 2009.

[0248]

[0088] Carsen Stringer, Tim Wang, Michalis Michaelos, and Marius Pachitariu. Cellpose: a generalist algorithm for cellular segmentation. Nature Methods, 18(1): 100-106, Jan 2021.

[0249]

[0089] Haoxi Ran, Jun Liu, and Chengjie Wang. Surface representation for point clouds. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18942-18952, June 2022.

[0250]

[0090] Antonio Montanaro, Diego Valsesia, and Enrico Magli. Rethinking the compositionality of point clouds through regularization in the hyperbolic space, 2022.

[0251]

[0091] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip H.S. Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), pages 16259-16268, October 2021.

[0252]

[0092] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point- bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19313-19322, June 2022.

[0093] William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. SIGGRAPH Comput. Graph., 21(4): 163-169, August 1987.

[0253]

[0094] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.

[0254]

[0095] Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, AndreasMueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Arnaud Joly, Brian Holt, and Gael Varoquaux. API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Mining and Machine Learning, pages 108-122, 2013.

[0255]

[0096] Dawson-Haggerty et al. trimesh.

[0256]

[0097] Lis Custodio, Sinesio Pesco, and Claudio Silva. An extended triangulation to the marching cubes 33 algorithm. Journal of the Brazilian Computer Society, 25(1):6, Jun 2019.

[0257]

[0098] J. Vollmer, R. Mencl, and H. Muller. Improved laplacian smoothing of noisy surface meshes. Computer Graphic Forum, 18(3): 131-138, 1999.

[0258]

[0099] Seyed Saber Mohammadi, Yiming Wang, and Alessio Del Bue. Pointview-gcn: 3d shape classification with multi-view point clouds. In 2021 IEEE International Conference on Image Processing (ICIP), pages 3103-3107. IEEE, 2021.

[0259]

[0100] Benjamin Eckart, Wentao Yuan, Chao Liu, and Jan Kautz. Self-supervised learning on 3d point clouds by learning discrete generative models. In 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8244-8253, 2021.

[0260]

[0101] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.

[0261]

[0102] Feiran Li, Kent Fujiwara, Fumio Okura, and Yasuyuki Matsushita. A closer look at rotation-invariant deep point cloud analysis. In Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), pages 16218-16227, October 2021.

[0262]

[0103] An Tao. Unsupervised point cloud reconstruction for classific feature learning. https: / / github.com / AnTao97 / UnsupervisedPointCloudReconstruction, 2020.

[0263]

[0104] Maximilian Use, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2127-2136. PMLR, 10-15 Jul 2018.

[0264]

[0105] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.

[0265]

[0106] Johannes Schindelin, Ignacio Arganda-Carreras, Erwin Frise, Verena Kaynig, Mark Longair, Tobias Pietzsch, Stephan Preibisch, Curtis Rueden, Stephan Saalfeld, Benjamin Schmid, Jean-Yves Tinevez, Daniel James White, Volker Hartenstein, Kevin Eliceiri, Pavel Tomancak, and Albert Cardona. Fiji: an open-source platform for biological-image analysis. Nature Methods, 9(7):676-682, Jul 2012.

[0266]

[0107] Nicholas Sofroniew, Talley Lambert, Kira Evans, Juan Nunez-Iglesias, Grzegorz Bokota, Matthias Bussonnier, Gonzalo Pena-Castellanos, Philip Winston, Kevin Yamauchi, Draga Doncila Pop, Pam, Ziyang Liu, Ahmet Can Solak, alisterburt, Genevieve Buckley, Andy Sweet, Lorenzo Gaifas, Gregory Lee, Jaime Rodriguez-Guerra, Nathan Clack, Jordao Bragantini, Lukasz Migas, Volker Hilsenstein, Melissa Weber Mendonga, Robert Haase, Hector, Jeremy Freeman, Peter Boone, Alan R Lowe, and Christoph Gohlke. napari / napari: 0.4.13rc0, January 2022.

[0267]

[0108] Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. Mitsuba 2: A retargetable forward and inverse Tenderer. ACM Trans. Graph., 38(6), nov 2019.

[0268]

[0109] Tolga Birdal. Mitsuba2PointCloudRenderer. https: / / github. com / tolgabirdal / Mitsuba2PointCloudRenderer , 2020.

[0269]

[0110] Blender Online Community. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.

Claims

Claims1. A computer-implemented method for assessing the phenotype of a cell, the method comprising: a. feeding point clouds, indicative of cells and cell nuclei, into a pretrained encoder; b. extracting, by the pretrained encoder, cell features and nuclei features from the point clouds; c. transforming, using a transformer encoder, the cell and nuclei features into transformed cell features; d. classifying the cell based upon the comparison of the transformed cell features with known phenotype responses; and e. labelling the cell based upon the comparison of the transformed cell features with known phenotype responses.

2. The method of claim 1, wherein the cell features and nuclei features are deep learning features extracted using the pretrained encoder, and which do not correspond to classical 3D cell features such as size, eccentricity or sphericity, but encompass such features.

3. The method of claim 2, wherein the deep learning features are not 2D features derived from a 2D z-slice through the cell, but are instead derived from the 3D point clouds.

4. The method of any preceding claims, wherein the pretrained encoder is a Dynamic FoldingNet Encoder, DFN.

5. The method of any preceding claims, wherein the pretrained encoder produces a feature set, the feature set being a transformation of a set of feature embeddings, Z.

6. The method of claim 4, wherein the output of the transformer encoder can be defined by:Tt= Layer N orm(Zt' + FFN) where T, is the output of the transformer encoder, Z- is the normalised embeddings from the DFN and FF / V is the feed-forward network.

7. The method of claim 4, the method further comprising:deriving, by the DFN, geometric deep learning based 3D cell features from the cells and cell nuclei within the point clouds.

8. The method of any of the preceding claims, the method further comprising: concatenating, by the pretrained encoder, the cell features and nuclei features into concatenated cell and nuclei feature pairs.

9. The method of any of the preceding claims, the method further comprising: grouping features from the cell features and nuclei features that appear in a single field of view of the point clouds into a virtual bag; and labelling the virtual bag with a label indicative of an associated application.

10. The method of any of the preceding claims, wherein the transformer encoder is a multi-head attention-based transformer encoder.

11. The method of claim 7, the method further comprising: extracting, using the transformer encoder, the amount repeated cell features and nuclei features within the virtual bag; and extracting, using the transformer encoder, dependencies between the 3D cell features and the 3D nuclei features.

12. The method of claim 11, the method further comprising: analysing the dependencies between the 3D cell features and 3D nuclei features; and extracting, based on the dependencies, changes in the inter-relationship of the 3D cell features and 3D nuclei features.

13. The method of any preceding claims, the method further comprising: classifying, using a cell-level classifier, cell level features based upon the comparison of the transformed cell features with the known phenotype cell level features.

14. The method of claim 9, the method further comprising: classifying, using a bag-level classifier, bag level features based upon the comparison of the grouped features within the bag with the known phenotype bag level features.

15. The method of claim 14, wherein the bag-level classifier comprises a multiple instance learning, MIL, aggregator and a multi-layer perceptron, MLP, classifier.

16. The method of claim 15, wherein the multiple instance learning aggregator is a two-stream multiple instance learning aggregator, wherein a first stream comprises an instance-level classifier and the second stream comprises a bag-level classifier.

17. The method of claim 16, wherein an output of the instance-level classifier is defined as:wherein cinstanceis the output of the instance-level classifier, Ttis the output of the pretrained encoder, max is the max pooling function and Wois a weight vector.

18. The method of claim 16, wherein an output of the bag-level classifier is defined as:wherein cbagis the bag-level, Ttis the output of the pretrained encoder, Wbis the weight vector for binary classification, b is bag embedding.

19. The method of claim 16, wherein the output of the multiple instance learning aggregator is the weighted sum of the first and second stream.

20. The method of claim 15, the method further comprising: applying, using the MIL aggregator, attention weights on each cell within a bag, the attention weights being indicative of the importance of the cell and nuclei features of said cell for predicting similar phenotype responses.

21. The method of any preceding claims, wherein the label for the cell is a label indicative of a known substance, the known substance being one which has a matching phenotype signature with the cell.

22. The method of any of claims 1-21, wherein the label for the cell is a label indicative of a clinical application, the clinical application being the clinical application of a known substance which has a matching phenotype signature with the cell.

23. The method of any of the preceding claims, wherein the point clouds are obtained by:three dimensional segmentation of cells and cell nuclei; surface rendering of the three dimensional segmentation using a marching cubes algorithm; smoothing the rendered surfaces using Laplacian smoothing; and then extracting vertices of polygons forming the smoothed surfaces as the point clouds.

24. A computer system for implementing a cell assessment and labelling algorithm, the system comprising: one or more processor units; and a computer readable storage medium storing one or more computer programs such that when performed by the one or more processor units cause the computer system to operate in accordance with the method of any of claim 1 to 22.

25. A system for predicting clinical applications of a substance, the system comprising: a. input data, including point clouds indicative of cells and cell nuclei, the data being input such that it can be fed into a pretrained encoder; b. a pretrained encoder arranged to extract, cell features and nuclei features from the point clouds; c. a transformer encoder arranged to transform the cell and nuclei features into transformed cell features; d. one or more classifiers arranged to classify the cell based upon the comparison of the transformed cell features with known phenotype responses; and e. a processor arranged to label the cell based upon the comparison of the transformed cell features with known phenotype responses.

Images