Methods, devices, and systems for spatial transcriptome slide alignment

Abstract

Techniques are provided for aligning spatial transcriptome slides of biological tissues. Alignment of images and spot coordinates within a transcriptome profile of a biological tissue can utilize a machine learning-enabled alignment model that can receive both the displacement image to be aligned and a reference image as input. The alignment model can learn a diffeomorphic transformation between these images. Such a transformation can be easily applied to align the displacement image and the displacement coordinates of spots within the transcriptome profile of the transcriptome. In some cases, transcriptome data can be parsed and converted into an image-like format, which can then be used as supplemental input to the alignment model. The alignment model can be trained in a tissue-independent manner, such as by using synthetic images.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 381,813, filed November 1, 2022, the entire text of which is incorporated herein by reference. Summary of the Invention

[0002] It should be understood that both the following general description and the following detailed description are merely illustrative and explanatory, and not limiting. In one embodiment, the present disclosure provides a computer-implemented method. The computer-implemented method includes receiving a reference image of biological tissue and receiving a displaced image of the biological tissue. The computer-implemented method also includes applying a normalization process to the reference image to obtain a normalized reference image and applying the normalization process to the displaced image to obtain a normalized displaced image. The computer-implemented method further includes performing a first registration of the normalized displaced image with respect to the normalized reference image. The first registration results in a set of parameters defining a coarse transformation, and further results in a second displaced image. The computer-implemented method further includes performing a second registration of the second displaced image with respect to the reference image by providing the normalized reference image to a machine-learning alignment model, providing the second displaced image to the machine-learning alignment model, and applying the machine-learning alignment model to the reference image and the second displaced image. Applying the machine-learning alignment model generates a deformation vector field representing an alignment transformation between the reference image and the second displaced image.

[0003] Additionally, the computer-implemented method can further include receiving reference spatial coordinates of spots in a first transcriptome profile of the biological tissue. The first transcriptome profile corresponds to the reference image. Each spot includes one or more cells. The computer-implemented method can also include receiving displacement spatial coordinates of spots in a second transcriptome profile of the biological tissue, the second transcriptome profile corresponding to the displacement image. The computer-implemented method can include performing a first alignment of the displacement spatial coordinates with the reference spatial coordinates based on the coarse transformation, obtaining second displacement spatial coordinates of the spots in the second transcriptome profile, and performing a second alignment of the second displacement spatial coordinates of the spots in the second transcriptome profile based on the alignment transformation.

[0004] In yet another embodiment, the present disclosure provides another computer-implemented method. The computer-implemented method includes generating a plurality of training image pairs based on a plurality of training label map pairs, each pair of the plurality of training image pairs including a training reference image and a training displacement image. The computer-implemented method also includes determining a solution to an optimization problem related to a loss function based on a similarity measure of the pair of training label maps and a deformation vector field representing an alignment transformation between a first training reference image and a first training displacement image of a pair of the plurality of training image pairs. The solution defines an alignment model for aligning the assessment displacement image of the biological tissue with the assessment reference image of the biological tissue.

[0005] Additional elements and advantages of the present disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the present disclosure. The advantages of the present disclosure may be achieved by means of the elements and combinations particularly pointed out in the appended claims.

[0006] This summary is not intended to identify key or essential features of the disclosure, but merely to summarize certain features and variations thereof. Other details and features are described in the following sections. Moreover, both the foregoing summary and the following detailed description are exemplary and explanatory only and are not intended to limit the scope of the present disclosure.

[0007] The accompanying drawings are an integral part of this disclosure and are incorporated herein. The drawings illustrate exemplary embodiments of the present disclosure and, together with the specification and claims, serve to explain, at least in part, various principles, elements, or aspects of the present disclosure. Embodiments of the present disclosure are described more fully below with reference to the accompanying drawings. However, the various elements of the present disclosure may be implemented in many different forms and should not be construed as limited to the implementations set forth herein. Like numbers refer to like elements throughout. [Brief explanation of the drawings]

[0008] [Figure 1] 1 illustrates an example of a computing system in accordance with one or more embodiments of the present disclosure. [Figure 1A] 1 illustrates an example of a workflow according to an embodiment of the present disclosure. [Figure 1B] 1 shows examples of reference and displacement images of biological tissue and their respective normalized images, according to embodiments of the present disclosure; [Figure 1C] 1 illustrates an example set of reference spot coordinates in a reference slide and a set of displacement spot coordinates in a displacement slide according to aspects described herein. [Figure 2] 1 illustrates an example of an alignment model, in accordance with one or more embodiments of the present disclosure. [Figure 3] 1 illustrates another example of a computing system in accordance with one or more embodiments of the present disclosure. [Figure 4A] 1 shows examples of input images of biological tissue, one of the illustrated images being a reference image and the other an affine transformed displacement image. [Figure 4B] 1 illustrates an example of a tile image in accordance with one or more embodiments of the present disclosure. [Figure 5A] 10 illustrates an example of four tile transformation vector fields in accordance with one or more embodiments of the present disclosure. [Figure 5B] 10 illustrates an example of a deformation vector field obtained from a multiple tile deformation vector field in accordance with one or more embodiments of the present disclosure. [Figure 6] 1 illustrates an example of a computing system for training an alignment model using synthetic training data, in accordance with one or more embodiments of the present disclosure. [Figure 6A] 1A-1C illustrate an example process flow for generating reference and displacement label maps and generating synthetic training reference and displacement images according to aspects described herein. [Figure 7A] 10 shows an example of alignment performance of a small slide using simulated deformation, according to an embodiment of the present disclosure. [Figure 7B] 10 shows an example of alignment performance of a small slide using simulated deformation, according to an embodiment of the present disclosure. [Figure 8A] 10 shows an example of the alignment performance of a whole-space slide of a mouse brain using simulated deformation, according to an embodiment of the present disclosure. [Figure 8B] 10 shows an example of the alignment performance of a whole-space slide of a mouse brain using simulated deformation, according to an embodiment of the present disclosure. [Figure 9] 10 shows an example of full-space slide alignment performance of a human lymph node using simulated deformation, according to an embodiment of the present disclosure. [Figure 10] 10 shows an example of alignment performance for more than two slides with simulated deformation, according to an embodiment of the present disclosure. [Figure 11] 1 illustrates the application of an alignment model according to an embodiment of the present disclosure to the alignment of two slides of a mouse brain with realistic deformations. [Figure 12]1 illustrates an example computer-implemented method for aligning images and spatial coordinates of spots in a transcriptome profile of biological tissue to a common coordinate system, in accordance with one or more embodiments of the present disclosure. [Figure 13] 1 illustrates an example computer-implemented method for normalizing an image of biological tissue in accordance with one or more embodiments of the present disclosure. [Figure 14] 1 illustrates an example computer-implemented method for normalizing images of biological tissue using transcriptome profiling data, in accordance with one or more embodiments of the present disclosure. [Figure 15] 1 illustrates an example of a computer-implemented method for training a machine learning (ML) alignment model, in accordance with one or more embodiments of the present disclosure. [Figure 16] 1 illustrates an example computer-implemented method for generating synthetic training data for training an ML alignment model, in accordance with one or more embodiments of the present disclosure. [Figure 17] 1 illustrates an example of a computer-implemented method for obtaining an alignment model and aligning an image of biological tissue to a reference coordinate system, in accordance with one or more embodiments of the present disclosure. [Figure 18A] 10 shows alignment performance results according to an embodiment of the present disclosure. [Figure 18B] 10 shows alignment performance results according to an embodiment of the present disclosure. [Figure 18C] 10 shows alignment performance results according to an embodiment of the present disclosure. [Figure 18D] 10 shows alignment performance results according to an embodiment of the present disclosure. [Figure 18E] 10 shows alignment performance results according to an embodiment of the present disclosure. [Figure 18F] 10 shows alignment performance results according to an embodiment of the present disclosure. [Figure 18G] 10 shows alignment performance results according to an embodiment of the present disclosure. [Figure 18H] 10 shows alignment performance results according to an embodiment of the present disclosure. [Figure 18I]10 shows alignment performance results according to an embodiment of the present disclosure. [Figure 18J] 10 shows alignment performance results according to an embodiment of the present disclosure. [Figure 19] 1 illustrates an example of a computing system for performing slide alignment in accordance with one or more embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present disclosure recognizes and addresses, among other technical challenges, the problem of aligning images and spatial coordinates of spots in a transcriptome profile of a slide of biological tissue relative to a common coordinate system. The biological tissue may include, for example, a section of a human organ or other type of animal organ. Embodiments of the present disclosure include computer-implemented methods, computing devices, computing systems, and computer program products, which, individually or in combination, enable a set of images and spatial coordinates of spots in a transcriptome profile of a slide of biological tissue and another set of images and spatial coordinates of spots in a transcriptome profile of another slide of biological tissue to be registered relative to a common coordinate system. More specifically, embodiments of the present disclosure, individually or in combination, utilize machine-learned alignment models. The alignment model may include, for example, a convolutional neural network, which may receive as input both a displaced image to be aligned and a reference image, and learn a diffeomorphic transformation between the images. For simplicity, such a transformation is referred to as a registration transformation and can be easily applied to align displaced images and displaced spatial coordinates of spots in the transcriptome profiles. Furthermore, transcriptome data can be parsed and converted into an image-like format to serve as auxiliary input for alignment models, which can be trained in a tissue-independent manner using synthetic images, thus making the alignment models broadly applicable.

[0010] Embodiments of the present disclosure may provide various improvements and efficiencies over existing techniques for automated alignment of displaced spatial coordinates of spots in transcriptome profiles of biological tissue slides. For example, embodiments of the present disclosure may utilize a graphics processing unit (GPU) instead of or in addition to a central processing unit (CPU). As a result, processing speeds may be superior to existing techniques. Additionally or alternatively, optical imaging may be combined with spatial transcriptome profiling to align image and spot spatial coordinates, resulting in more reliable alignment than existing techniques. As yet another example, alignment of image and spot spatial coordinates by training and applying a machine learning model to be used as an alignment model avoids the use of landmarks and other types of fiducial markers, as well as manual alignment. Additionally or alternatively, the machine learning model can be trained based on synthetic images and is tissue-independent, greatly increasing the flexibility and applicability of the image and spot alignment described herein. Furthermore, the tiling and stitching technique allows for easier analysis of images larger than the image size used to train the machine learning model.

[0011] FIG. 1 illustrates an example computing system 100 in accordance with one or more embodiments of the present disclosure. The computing system 100 includes a data collection platform 104 that enables measurement and collection of various types of data. Specifically, the data collection platform 104 may include an optical imaging device 106 and a transcriptome profiling device 108. The optical imaging device 106 enables generation of images of biological tissue slides using light (visible or other light). The transcriptome profiling device 108 enables generation of spatial transcriptome profiling data of the biological tissue slides. The transcriptome profiling device 108 may generate such data at one or more spatial resolutions, including near single-cell resolution, single-cell resolution, and / or sub-cell resolution. The computing system 100 also includes one or more memory devices 110 (referred to as data storage 110) that may hold image data representing the images of the biological tissue slides. The image data may be held in one or more files within a file system configured within the data storage 110. Each of the one or more files may be formatted according to a format suitable for image storage. In some cases, the image may be a histological staining image. Such an image may be obtained using H&E (hematoxylin and eosin solution) staining. Embodiments of the present disclosure are not limited in this respect and may also be applied to images obtained using other types of histological staining techniques. In some cases, the image may be an immunohistochemical staining (IHC) image. In still other cases, the image may be an in situ hybridization (ISH) image, including a fluorescent in situ hybridization (FISH) image. In still other cases, the image may be a single molecule fluorescent in situ hybridization (smFISH) image. Thus, the data generated by the data acquisition platform 104 and maintained in the data storage 110 may include various types of data.In some embodiments, the single-cell data may include single-cell RNA-seq data (scRNA-seq) or single-nuclear RNA-seq data (snRNA-seq). In some embodiments, the single-cell data may include single-cell ChIP. In some embodiments, the single-cell data may include single-cell ATAC-seq. In some embodiments, the single-cell data may include single-cell proteomics. In some embodiments, the spatial data may be coarse-grained. In some embodiments, the spatial data may include STARmap and / or MERFISH. In some embodiments, the single-cell data is multimodal single-cell data. In some embodiments, the multimodal data is single-cell RNA-seq and chromatin accessibility data (SHARE-seq). In some embodiments, the multimodal data is single-cell RNA-seq and proteomics data (CITE-seq). In some embodiments, the multimodal data is single-cell RNA-seq and patch-clamp electrophysiological recordings and morphological analysis of single neurons (patch-seq).

[0012] The data storage 110 can also hold transcriptome profiling data including multiple datasets corresponding to respective images of slides of biological tissue. Any one of the multiple datasets including transcriptome profiling data for each of the slides of biological tissue can be acquired contemporaneously with optical imaging of the respective slide. That is, a first dataset including transcriptome profiling data for a first slide of biological tissue can be acquired contemporaneously with optical imaging of the first slide, a second dataset including transcriptome profiling data for a second slide of biological tissue can be acquired contemporaneously with optical imaging of the second slide, etc.

[0013] Computing system 100 also includes a computing device 120 operatively coupled to data storage 110 via a communications architecture 114 (e.g., wireless network(s), wired network, wireless link, wired link, server device, router device, gateway device, combinations thereof, etc.). Computing device 120 may include computing resources (not shown for clarity) including, for example, one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), memory, disk space, incoming bandwidth and / or outgoing bandwidth, interface(s) (e.g., I / O interfaces and APIs, or both), controller device(s), power source, combinations of the foregoing, and / or similar resources.

[0014] The computing device 120 may include an ingestion module 130 that can access images and transcriptome profiling data of biological tissue slides from the data storage 110. Accessing image data may include, for example, programmatically downloading the image and / or transcriptome profiling data from the data storage 110. For example, the ingestion module 130 may download the image and / or transcriptome profiling data via an application programming interface (API). The ingestion module 130 may receive image files (or data constituting the files) corresponding to the images by executing one or more function calls of the API and retain the image files (or data constituting the files) in one or more non-volatile memory devices 170 (referred to as memory 170). As another example, the ingestion module 130 may download the image and / or transcriptome profiling data by executing a script that copies the image files from a file system in the data storage 110 to the memory 170. The memory 170 may be integrated into the computing device 250.

[0015] Referring to workflow 101 shown in FIG. 1A, capture module 130 can receive an image of a slide of biological tissue. The image can be referenced to reference image 103 (or a fixed image) and can be used to establish a reference two-dimensional (2D) coordinate system. The location of pixels in reference image 103 can be defined with respect to that 2D coordinate system. Reference image 103 may also be referred to as a "fixed image." Additionally, capture module 130 can receive an image of another slide of biological tissue. Such an image may also be referred to as a displaced image 105 because multiple slides of a biological tissue section may be displaced relative to one another. The location of pixels in displaced image 105 need not be defined within a reference 2D coordinate system. Thus, a pixel at a particular location in displaced image 105 does not necessarily correspond to a pixel at the same location in reference image 103. Displaced image 105 may also be referred to as a "moving image." Further examples of reference image 103 and displaced image 105 are shown in FIG. 1B.

[0016] The computing device 120 can align the displacement image 105 and the displacement spatial coordinates of the spots on the displacement slide where the transcriptome was profiled with respect to a reference 2D coordinate system. Prior to this alignment, the computing device 120 can process the reference image 103 and the displacement image 105 by applying a normalization process. The normalization process is applied to generate a normalized image that represents the tissue mask image. To that end, the computing device 120 can include a normalization module 140 that can apply the normalization process.

[0017] As part of the normalization process, the normalization module 140 can determine whether to apply color normalization. Color normalization can be applied when significant color deviations from an expected color palette and / or color mismatches between the reference image 103 and the displaced image 105 are detected. To make such a determination, the normalization module 140 can apply color deviation criteria and / or color mismatch criteria to both the reference image 103 and the displaced image 105. In some cases, neither the color deviation criterion nor the color mismatch criterion is met, and thus the normalization module 140 can determine not to apply color normalization. In some cases, in response to at least one criterion being met, the normalization module 140 can determine to apply color normalization to both the reference image 103 and the displaced image 105. Thus, the image component 142 included in the normalization module 140 can receive the reference image 103 and apply color normalization to the reference image 103. The reference image 103 can be received from the memory 170, for example. Additionally, image component 142 may receive displaced image 105 and apply color normalization to displaced image 105. Displaced image 105 may be received from memory 170. Applying color normalization may include scaling each RGB color channel so that the intensity value of each channel is within the range of [0, 1] for each pixel of the input image (reference image or displaced image).

[0018] Regardless of whether color normalization is applied, the image component 142 can further apply a normalization process to the reference image 103 and the displaced image 105. As a result, the image component 142 generates a normalized reference image and a normalized displaced image. To apply the normalization process, the image component 142 can receive an input image (e.g., the reference image 101 or the displaced image 103). The input image can be received from memory 170. The image component 142 can then generate a tissue mask image of the input image by separating tissue foreground pixels and non-tissue background pixels. If the input image is a histological image of biological tissue, the input image can be a red-green-blue (RGB) color image. The image component 142 can separate tissue foreground pixels and non-tissue background pixels by performing operations on the RGB color image. In some cases, it may be difficult to identify non-tissue background pixels using the RGB color image. In such cases, the image component 142 can convert the RGB color image to a grayscale image. Next, the image component 142 can use the grayscale image to determine a threshold intensity that allows tissue foreground pixels to be distinguished from non-tissue background pixels. Specifically, the image component 142 can identify pixels in the grayscale image that have an intensity less than the threshold intensity and set such pixels as non-tissue background pixels. The image component 142 can assign a null intensity to the non-tissue background pixels in the grayscale image. That is, the non-tissue background pixels can be set as black pixels in the grayscale image. Accordingly, the image component 142 can identify pixels in the RGB color image that correspond to the non-tissue background pixels in the grayscale image. The image component 142 can set such identified pixels as non-tissue background pixels in the RGB color image. After identifying the non-tissue background pixels in the input image, the image component 142 can, for example, set the non-tissue background pixels as black pixels in the tissue mask image.The tissue mask image with black pixels constitutes the normalized input image. An example of a normalized reference image 103A and a normalized displacement image 105A is shown in Figure 1B.

[0019] The normalization module 140 may also optionally apply one or more shaping operations to the tissue mask image, which may include cropping the tissue mask image to remove unwanted background portions and resizing the tissue mask image to obtain a predetermined resolution suitable for input into the alignment models described herein.

[0020] As mentioned above, transcriptome profiling data can be obtained from the same slides used to acquire images of biological tissues (called histological images). Therefore, transcriptome profiling data correspond to and complement the information contained in histological images. Some existing platforms can perform gene expression measurements on hundreds to tens of thousands of tissue subregions, called spots. In some cases, spots cover an area spanning one cell, resulting in measurements with single-cell spatial resolution. In other cases, spots cover an area spanning several to tens of cells, resulting in measurements with near-single-cell spatial resolution. ISH-based platforms can quantify messenger RNA (mRNA) transcripts with subcellular resolution. However, gene expression obtained from a single or small number of cells in each spot may result in low numbers of detected mRNA transcripts and may be susceptible to non-spatial confounding factors such as batch effects and variations in cell state.

[0021] To mitigate or even completely avoid limitations due to batch effects and variations in cellular state, the normalization module 140 can convert the transcriptome profiling data into image-formatted input data. To do so, the acquisition module 130 can receive spatially resolved transcriptome (ST) profiling data corresponding to a first slide and a second slide of a biological tissue. The first slide can correspond to a reference image, and the second slide can correspond to a displacement image. The transcriptome profiling data defines the transcriptome profile of each spot within each of the first and second slides. The location of a spot within the first slide can be defined by the location of a pixel within the reference image that is located beneath the center of the spot and is therefore defined within a reference 2D coordinate system. The location of a spot within the second slide is defined by the location of a pixel within the displacement image that is located at the center of the spot and is therefore not necessarily defined within a reference 2D coordinate system. Therefore, a spot at a particular location within the second slide does not necessarily have to correspond to a spot at the same location within the first slide. An example set of reference spot coordinates 107 and a set of displacement spot coordinates 109 are shown in Figure 1C. As described herein, the computing device 120 can align the displacement space coordinates of the spots on the displacement slide with respect to a reference 2D coordinate system.

[0022] The ingestion module 130 can pass (or send) the transcriptome profiling data to the normalization module 140. The transcriptome component 146, which can be part of the normalization module 140, can assemble the transcriptome profiles of all spots in a particular slide into count matrices, such as the count matrix 111 (for the reference image 103) and the count matrix 113 (for the displacement image 104) shown in FIG. 1A. The columns of the count matrices 111 and 113 correspond to respective spot identifiers (IDs), or in some cases, cell IDs. The rows of the count matrices 111 and 113 correspond to respective genes. The transcriptome component 146 can normalize the count matrices 111 and 113 to ensure that the total number of counts per column is equal. The transcriptome component 146 can then natural logarithm-transform the count matrices 111 and 113. The transcriptome component 146 can also center and scale the count matrices 111 and 113 to have values ​​ranging from 0 to 1. The transcriptome component 146 can then determine the top N variable genes across spots within a particular slide. The parameter N is configurable. In one example, N=2000.

[0023] The transcriptome component 146 can integrate the transcriptome data corresponding to the first slide and the second slide. The transcriptome component 146 can perform dimensionality reduction by scaling the integrated data and applying principal component analysis (PCA). The transcriptome component 146 can perform mutual nearest neighbor clustering based on the top M principal components. A resolution parameter is set to identify major clusters. The parameter M is configurable. In one example, M=30.

[0024] After clusters such as cluster 115 shown in FIG. 1A are identified, transcriptome component 146 can generate a label map based on the transcriptome profiling data. The label map spans the same defined region as the input image (e.g., reference image 103 or displacement image 105) of the biological tissue associated with the transcriptome profiling data. The label map can include a first label associated with multiple first pixels in the input image, a second label associated with multiple second pixels in the input image, and so on. To generate the label map, transcriptome component 146 can assign cluster labels to pixels located at the centroid of each spot (or each cell in the case of single-cell resolution measurements). In some cases, the pixels of the input image can be partitioned into a Voronoi diagram using the centroid pixel as the generating point. The transcriptome component 146 can assign predefined cluster labels, such as cluster 1 or cluster 2, to pixels within each partition polygon of the Voronoi diagram, as shown in cluster 115 in FIG. 1A. Here, the predefined cluster labels are the same as the cluster labels assigned to the centroid pixels that define the partition polygon's generating point. Such a label assignment process is sometimes called label dilation.

[0025] The transcriptome component 146 can generate a contour map based on the label map. This contour map reflects the spatial organization of tissue spots (or cells in the case of single-cell resolution measurements) based on the transcriptome. Figure 1A shows an example of a contour map 111A for the reference image 103 and an example of a contour map 113A for the displacement image 105, each with three boundaries (e.g., based on clusters 115). In this regard, the boundary between any two regions with different cluster labels can be used to define a contour map. In fact, a first contour line in the contour map defines a boundary separating a first subset of pixels with a first cluster label from a second subset of pixels with a second cluster label. In some cases, high-complexity regions in the contour map (e.g., 111A and / or 113A) may be merged to avoid overfitting the model.

[0026] The transcriptome component 146 can overlay the contour maps (e.g., 111A and / or 113A) on the normalized input image. For example, as shown in FIG. 1A, the contour map 111A can be overlaid on the normalized reference image 111C (e.g., normalized reference image 103A), and the contour map 113A can be overlaid on the normalized displacement image 113C (e.g., normalized displacement image 105A). The contour lines in the counter maps (e.g., 111A and / or 113A) can be overlaid on the normalized input image at the pixels on which the contour lines are based. To do so, the transcriptome component 146 can identify specific pixels in the normalized input image that correspond to each contour line. The transcriptome component 146 can then update the normalized input image by changing the respective values ​​of the specific pixels. The transcriptome component 146 can determine each value as a linear combination of a first value from the normalized input image and a second value based on the transcriptome profiling data, where the ratio of the second value to the first value in the linear combination can be, for example, 0.3:0.7.

[0027] Converting transcriptome data into a contour map and incorporating the contour map into a tissue image can minimize the effects of noise and batch effects in aligning a pair of biological tissue images. The contour map derived from the transcriptome data described herein provides orthogonal, complementary information to the tissue image, especially when the tissue image is homogeneous and lacks information about the spatial organization / orientation of the slide from which the image originates. However, normalizing the reference and displacement images to overlay the transcriptome-based contour map is optional. Indeed, in some scenarios, such as when the tissue image contains sufficient local texture detail or when the transcriptome data quality is low, the use of transcriptome data may not be desirable.

[0028] The normalization module 140 can pass (or transmit) the normalized reference image 111C and the normalized displaced image 113C to the transformation module 150. As shown in FIG. 1, the transformation module 150 can be included in the computing device 120. The transformation module 150 can perform two types of alignment of the normalized displaced image 113C to the normalized reference image 111C. To that end, an affine alignment component 154, which is part of the transformation module 150, can perform a first alignment of the normalized displaced image 113C to the normalized reference image 111C. The first alignment can provide a rough alignment of the normalized displaced image 113C to a reference 2D coordinate system associated with the reference image 103. Performing the first alignment can include determining an affine transformation that can be determined between the normalized displaced image 113C and the normalized reference image 111C. Determining the affine transformation includes determining a set of parameters that define the transformation. As a result of performing the first alignment, the affine alignment component 154 can determine a first deformation vector field associated with the affine alignment based on the affine transformation. Such first deformation vector field can define a new position to which each pixel of the normalized displacement image 113C should move in order to at least partially align the normalized displacement image 113C with the normalized reference image 111C. The transformation module 150 can retain the first deformation vector field in memory 170 (shown in FIG. 1A as spot alignment deformation vector field(s) 151).

[0029] As part of applying the first alignment, the alignment module 160 can apply a first deformation vector field to the normalized displaced image 113C. As a result, the alignment module 160 can generate a second displaced image (not shown). The alignment module 160 can pass (or transmit) the second displaced image to the transformation module 150. The transformation module 150 can receive the second displaced image and provide the normalized reference image 111C and the second displaced image to the dense registration component 156. The dense registration component 156 can perform a second alignment of the second displaced image with respect to the reference image 103 by applying an alignment model 158 to the normalized reference image 111C and the second displaced image. Applying the alignment model in this manner can generate a second deformation vector field representing the alignment transformation between the reference image and the second displaced image. The dense registration component 156 can be configured (e.g., program coded and constructed) to include the alignment model 158. Alternatively, in some embodiments, the dense registration component 156 can load or otherwise obtain the alignment model 158 from memory 170. To that end, in such embodiments, the memory 170 can hold the alignment model 158.

[0030] As part of applying the alignment model 158, the dense registration component 156 can determine a corresponding inverse deformation vector field by integrating the inverse gradient (also called negative flow) of the deformation vector field. The alignment module 160 can then apply the inverse deformation vector field to the second displaced image to obtain an aligned normalized image.

[0031] The alignment model 158 may be a machine learning model including a neural network with one or more first layers of a first type and one or more second layers of a second type. The one or more first layers may be configured to extract features from the normalized reference image and the second displaced image. The one or more second layers may be configured to output a deformation vector field and an inverse deformation vector field corresponding to the deformation vector field. In some cases, as shown in FIG. 2 , the machine learning model may include a convolutional neural network (CNN) 200 with an ingestion module 210, an encoder / decoder block 220 including an encoder module for feature extraction, a single bottleneck level representing a latent space, and a decoder module, and a field construction module 230 configured to output the deformation vector field and the inverse deformation vector field. The ingestion module allows image data defining the reference image 204 and image data defining the displaced image 206 to be simultaneously received. The reference image and the displaced image may both be normalized images according to aspects described herein. For illustrative purposes only, in CNN 200, the ingestion module 210 is implemented with a Siamese input processor, and the encoder / decoder block 220 is implemented with a U-Net backbone. Furthermore, CNN 200 is shown configured for images with a size of 256 x 256 pixels. Of course, the present disclosure is not limited in this respect, and other image sizes are also contemplated. The encoder module includes four stages of downsampling (or reduction), and the decoder module includes four stages of upsampling (or expansion). A skip connection from the encoder module to the decoder module is also included at each stage. The final stage of the decoder module is connected to a field synthesis module to output a deformation field for aligning the input displacement image, or a set of displacement coordinates of spots on a displacement slide from which a transcriptome profile was acquired, to an input reference. It should be noted that the present disclosure is not limited to a U-Net backbone.Indeed, other CNNs may form part of the alignment model, each of which contains a module (e.g., a visual transformer) capable of performing feature extraction from image data and mapping the extracted features to a deformation field and a corresponding inverse deformation field.

[0032] As an example, a component of the alignment model 158 (e.g., the encoder module of the encoder / decoder block 220) may perform feature extraction from image data as a pre-training step or pre-training process. In addition to using the transcriptome profile of the input reference, or if the transcriptome profile of the input reference is unavailable, the encoder module may use self-supervised learning (e.g., label-less learning) to extract features from the input reference (e.g., raw histological image(s)) and map the extracted features to deformation fields and corresponding inverse deformation fields as described herein. The encoder module may use such self-supervised pre-training to extend the existing feature extraction capabilities / functionality of the encoder described herein. For example, the encoder module may autonomously identify and interpret patterns in the input reference using representation learning, mask image modeling, a combination thereof, etc. This is done by predicting one or more features (e.g., features associated with a first portion) using other features derived therefrom (e.g., features associated with another portion(s)). The encoder module's use of self-supervised learning can improve alignment performance with higher accuracy. Furthermore, the encoder module uses self-supervised learning for feature extraction, combined with subsequent supervised fine-tuning on synthetic data, as further described herein, can also improve the precision and accuracy of the model.

[0033] The alignment model 158 has a defined size (e.g., N p ×N p In one example, the image may be configured to receive an image of N p= 256. However, the ML-based alignment described herein is not limited in this respect. Indeed, in some embodiments, computing device 120 may align images of a size larger than the defined size. To that end, as shown in FIG. 3, transformation module 150 may divide the input image into sections of a defined size. To that end, transformation module 150 may divide M p ×M p (M p >N p ), which can generate a tiling of the input image of size equal to

[0034] More specifically, the reference image and the displacement image are each M p ×M p , the tessellation component 310 can generate a tiling of the normalized reference image, where each tile image of the normalized reference image tiling has a defined size N p ×N p The tile images can be evenly spaced along the width and height of the normalized reference image. As shown in Figure 4A, each tile image in the normalized reference image tiling partially spatially overlaps all other tile images adjacent to it. In some cases, the number of adjacent tile images can be as few as two and as many as four. Figure 4B shows an example of image tiles for the reference image (or fixed image) shown in Figure 4A and the affinely transformed displaced image.

[0035] Instead of processing the input displaced image, the tessellation component 310 can generate a tiling of the second displaced image. The tiling of the second displaced image can be generated in the same manner as the tiling of the reference image. Therefore, the tiling of the second displaced image has the same structure as the tiling of the reference image. Therefore, the tiled second displaced image and the normalized reference image have the same number of tile images. Each tile image in the tiling of the second displaced image has exactly one matching tile image in the tiling of the normalized reference image, covering the same spatial region in each of the second displaced image and the normalized reference image.

[0036] The tile images of the normalized tiling of the reference image and the tile images of the tiling of the second displaced image can be provided to the alignment model 158. The tile images can be provided sequentially, and a pair of tile images can be provided at the same time. A pair of tile images consists of a tile image of the normalized reference image and a tile image of the tiling of the second displaced image that matches the tile image of the normalized reference image in terms of placement and size. That is, both tile images of the pair cover the same spatial region in their respective full images.

[0037] The transformation module 150, via the dense registration component 156, can apply an alignment model 158 to each tile image of the tiling of the normalized reference image, along with a tile image of the tiling of the second displaced image that matches the tile image in the normalized reference image, as described above. As a result, applying the alignment model 158 results in N T N tile deformation vector fields can be generated, where N T represents the total number of tiles in the tiling of the normalized reference image, which is equal to the total number of tiles in the tiling of the second displacement image. Figure 5A shows an example of four tile deformation vector fields.

[0038] The alignment module 160 TCombine the tile transformation vector fields to create a size M p ×M p A deformation vector field can be formed corresponding to the displacement image having N T Each tile deformation field and all of its neighboring tile deformation fields have a common defined area. Thus, alignment module 160 aligns N tile deformation vector fields by determining, for each pixel within the common defined area, a weighted average of the tile deformation vector fields that overlap at that pixel and assigning the weighted average to the deformation vector field for each pixel within the common defined area. T Tile deformation vector fields can be combined. In some cases, the numerical weight applied to a tile deformation field at a particular pixel is inversely proportional to the distance between the center of that tile deformation field and the particular pixel. Figure 5B shows an example of a deformation vector field obtained by combining multiple tile deformation vector fields according to aspects described herein.

[0039] The alignment model 158 can be trained in a tissue-independent manner using synthetic training data. In this way, the alignment model 158 can be applied to any tissue type and / or image acquisition protocol (e.g., various histological staining techniques, non-histological fluorescence-based IHC imaging, and ISH imaging). Furthermore, the use of synthetic training data addresses the common problem of limited available training data in deep neural network training. In contrast to existing techniques, the image synthesis approach described herein is designed for the tissue alignment task solved by embodiments of the present disclosure.

[0040] Each instance of a training data record includes a set of four composite images: (a) a color reference image, (b) a first segmentation mask image associated with the color reference image, (c) a color displacement image aligned with the color reference image, and (d) a second segmentation mask image associated with the color displacement image. In this disclosure, the segmentation mask image may be referred to as a label map. Pixels in the segmentation mask image can be classified into a finite number of different classes, with each pixel assigned a specific class label. These classes can be abstract with respect to the training data but can reflect spatially separated regions of tissue microenvironment and / or cellular composition in the context of a tissue slide.

[0041] 6 illustrates an example computing system 600 for training an alignment model, such as alignment model 158, using synthetic images. Computing system 600 includes an image generation module 610 that can generate sets of four synthetic images to create training data for training the alignment model. Image generation module 610 can generate training image pairs, each pair of the training image pairs including a training reference image and a training displacement image. The training image pairs can be generated based on label maps.

[0042] More specifically, for a particular set of four composite images, the image generation module 610 may include a label map generation component 614 (referred to as label map generator 614) that can assign a label to each pixel across a region of a defined size. The labels can be assigned to multiple layers corresponding to the regions of the defined size. Specifically, the labels include multiple label sets corresponding to each of the multiple layers. That is, N L For layers, the first label set corresponds to the first layer, the second label set corresponds to the second layer, and N L The Nth layer corresponds to the L This continues until the Nth label set. L= 5. Therefore, the generation module 310 assigns N L N labels can be assigned. L Each of the labels corresponds to a different layer, which is sometimes called a channel.

[0043] Labels within a layer can be randomly set. To this end, the label map generator 614 can set multiple arbitrarily seeded simplex noise distributions within each layer. As a result, for each layer, a numerical weight is associated with a label at each pixel within a region of a defined size. The label map generator 614 can determine the numerical weight as a random value according to the simplex noise distribution at that pixel in that layer. Thus, the label map generator 614 can use such simplex noise distribution to determine the numerical weight of each of multiple predefined labels at a particular pixel within the region. The label map generator 614 can then assign a first label corresponding to the first numerical weight with the largest value among the respective numerical weights to the particular pixel.

[0044] Based on the set labels, the label map generator 614 can generate a base label map spanning a region of a defined size. Then, the label map generator 614 can generate a reference label map by warping the base label map using the simplex noise field as a warping field. Such warping may be referred to as first simplex noise warping. Furthermore, the label map generator 614 can also generate a displacement label map by warping the base label map using another simplex noise field as a warping field. Such warping may be referred to as second simplex noise warping.

[0045] The image generation module 610 may include a color image generation component 618 (referred to as a color image generator 618) that can generate a specific training reference image based on the reference label map. To do so, for each label in the first label map, the color image generator 618 can randomly select a color (e.g., an RGB color) for the label and configure a group of pixels corresponding to the label to the selected color. As a result, the color image generator 618 generates a color image. Furthermore, the color image generator 618 can perform operations on the color image to generate the specific training reference image. The operations on the color image may include blurring the color image to generate a blurred image and applying a bias intensity field to the blurred image. The blur applied to the color image may be a Gaussian blur. The bias intensity field spans an area of ​​a defined size.

[0046] The color image generator 618 can also generate specific training displacement images based on the displacement label map. To do so, for each label in the second label map, the color image generator 618 can randomly select a color (e.g., an RGB color) for the label and set a group of pixels corresponding to the label to have the selected color. As a result, the color image generator 618 generates a color image. Furthermore, the color image generator 618 can perform an operation on the color image to generate the specific training image. As described herein, performing an operation on the color image can include blurring the color image to generate a blurred image and applying a bias intensity field to the blurred image. The blur applied to the color image can be a Gaussian blur. The bias intensity field spans an area of ​​a defined size.

[0047] The image generation module 610 can construct a reference label map, a displacement label map, a specific training reference image, and a specific training displacement image associated with a particular instance of a training data record. The image generation module 610 can retain the first label map, the second label map, the specific training reference image, and the specific training displacement image as part of a composite image 634 in one or more storage devices 630 (referred to as image storage devices 630).

[0048] 6A illustrates an example process flow 650 for generating reference and displacement label maps (e.g., via the image generation module 610) and generating synthetic training reference and displacement images according to embodiments described herein.

number

number

number

number

number

number

[0049] The computing system 600 may include a training module 620 that can train an alignment model for aligning an assessment displacement image of the biological tissue with an assessment reference image of the biological tissue based on multiple label maps and multiple training image pairs (e.g., those generated by process flow 650). As described herein, the alignment model may be a machine learning model that includes a convolutional neural network (e.g., CNN 200 (FIG. 2)) with an encoder module for feature extraction, a single bottleneck level representing a latent space, a decoder module, and a field construction module configured to output a deformation vector field and an inverse deformation vector field corresponding to the deformation vector field. The field construction module may also be referred to as a deformation field formation module. In some examples, as described herein, the encoder module for feature extraction can perform feature extraction from image data, such as the assessment reference image of the biological tissue, as a pre-training step or process. In addition to using the transcriptome profile(s) of the evaluation reference image, or if the transcriptome profile(s) of the evaluation reference image are not available, the encoder module may autonomously identify and interpret patterns in the evaluation reference image by using self-supervised learning (e.g., label-less learning) to predict one or more features (e.g., associated with an initial portion(s)) using other features derived therefrom (e.g., associated with another portion(s)).

[0050] To train the alignment model, the training module 620 can iteratively determine a solution to an optimization problem with a loss function based on the similarity metrics of pairs of label maps and the deformation vector fields associated with the training reference images and training displacement images. Such a solution defines the trained machine learning alignment model. More specifically, the loss function can be defined as:

number

[0051] In this equation (1), f, s f represent the training reference image and the reference label map associated with that image, respectively. Furthermore, m and s m are the estimated deformation vector fields output by the alignment model, respectively.

number

number

number

[0052] In equation (1), the regularization term

number

[0053] The learning module 620 can persist the learned alignment models in an alignment model library 644 in one or more memory devices 640 (referred to as model storage 640). The learning module 620 can also configure an interface, such as an API, to allow access to the stored learned alignment models via function calls.

[0054] In embodiments of the present disclosure, a trained alignment model (e.g., alignment model 158) can infer conservative deformations in each inference run (e.g., applying the trained alignment model to a pair of evaluation images). Thus, a computing device (e.g., computing device 120 (FIG. 1)), or in some cases a system of a computing device, can apply the trained alignment model to a pair of evaluation images multiple times until the displacement coordinates stabilize or a defined number of iterations (e.g., 5 iterations) is reached. The defined number of iterations may be configurable based on user-specified input. The displacement coordinates can be considered stable if the mean squared error between the current coordinates at the nth iteration and the coordinates at the previous n-1th iteration is less than a predetermined percentage (e.g., 1%) of the mean squared error between the n-1th iteration and the n-2th iteration.

[0055] In embodiments of the present disclosure, three or more slides can be aligned in either a "template-based" mode or a "template-less" mode. The template-based mode involves setting one of the slides as a fixed reference and aligning the remaining slides to that fixed reference. The operations for aligning a pair of slides described herein can be applied to each of the non-reference slides to generate a set of aligned tissue slides.

[0056] The "template-less" mode involves scaling and centering each slide so that the spot coordinates detected on each slide are within the same range along the x- and y-axes. For example, assuming a spatial transcriptome slide size of 512 x 512, the minimum (x,y) coordinate of a spot on each slide can be (35,35), and the maximum (x,y) coordinate can be (477,477) or less. The "template-less" mode scales and centers ... s Each slide in the slides i For the remaining (N s -1) Slides j (j≠i) Each j is used as a fixed reference, and the trained alignment model s i The resulting (N-1) pairs of aligned coordinates are then compared with the initial coordinates s i (assuming no change since it is aligned to itself) and then average the slides i The final aligned coordinates can be output.

[0057] Figures 7-10 show alignment performance in various scenarios with different slide types and slide configurations, according to embodiments of the present disclosure. In some performance results, embodiments of the present disclosure are collectively referred to as "ML aligners" simply for nomenclature convenience. More specifically, Figures 7A and 7B show examples of alignment performance for small slides, according to embodiments of the present disclosure. Each of these small slides has 115 spots. Alignment performance is measured in terms of mean squared error (MSE). At simplex noise intensities of 5, 10, and 15, the performance of the ML aligner outperforms existing techniques for aligning slides. Without being bound by interpretation, it appears that although the GPSA at a simplex noise intensity of 20 produces a lower MSE than the ML aligner at such an intensity, the improved performance of the GPSA in this case is an artifact. Specifically, the number of spots within a small slide appears to mitigate the limited sensitivity of GPSA in distinguishing spots with similar transcriptome profiles at a simplex noise intensity of 20. Figures 8A and 8B show an example of alignment performance for a full spatial slide, according to an embodiment of the present disclosure. There are approximately 3,000 spots in such a full spatial slide. At all illustrated simplex noise intensities, the performance of ML aligners is superior to that of existing technologies for aligned slides. Again, referring to a simplex noise intensity of 20, it can be seen that GPSA's limitations in distinguishing spots with similar transcriptome profiles can result in reduced performance compared to ML aligners. Figure 9 shows an example of alignment performance for a full spatial slide of a human lymph node, according to an embodiment of the present disclosure. As shown in bar graphs 910, 920, and 930, regardless of the amount of warping, the performance of ML aligners is superior to that of existing technologies for aligned slides. Figure 10 shows an example of alignment performance for three or more slides. The slides are unordered and correspond to repeated samples.As shown in bar graph 1010, the performance of the ML aligner is superior to that of existing technology for alignment slides.

[0058] 11 illustrates the application of an alignment model according to an embodiment of the present disclosure to the alignment of two images of a mouse brain, where the images correspond to successive slices / slides of the mouse brain.

[0059] In view of the aspects described herein, exemplary methods that may be performed in accordance with the present disclosure may be better understood with reference to the flowcharts in FIGS. 12-17, for example. For ease of explanation, the exemplary methods disclosed herein are presented and described as a series of blocks (e.g., each block represents an action or operation in the method). However, the exemplary methods are not limited by the order of the blocks and associated actions or operations, as some blocks may occur in a different order than shown and described herein and / or concurrently with other blocks. Furthermore, not all illustrated blocks and associated action(s) may be required to perform an exemplary method according to one or more aspects of the present disclosure. Two or more of the exemplary methods (and any other methods disclosed herein) may be performed in combination with each other. It should be noted that the exemplary methods (and any other methods disclosed herein) may alternatively be represented as a series of interrelated states or events, such as in a state diagram.

[0060] Methods according to the present disclosure can be carried on an article of manufacture or a computer-readable non-transitory storage medium, which can enable or facilitate transport and transfer of such methods to a computing device or system of computing devices (such as a desktop computer, a laptop computer, a blade server, etc.), where they will be executed by, and therefore implemented in, one or more processors of the computing device(s), or stored in, or operatively coupled to, one or more memory devices of the computing device. In one aspect, one or more processors, such as processor(s) that implement (e.g., execute) one or more of the disclosed methods, can be employed to execute program code (e.g., processor-executable instructions) carried on a memory device or any computer- or machine-readable medium to perform one or more of the disclosed methods. Such program code can provide a computer-executable or machine-executable framework for performing the methods described herein.

[0061] FIG. 12 is a flowchart of an exemplary method 1200 for registering images and spatial coordinates of spots in a transcriptome profile of biological tissue to a common coordinate system, in accordance with one or more embodiments of the present disclosure. As described herein, the images can be histological staining images, IHC images, or ISH images (e.g., FISH images). A computing device, or a system of computing devices, can perform the exemplary method 1200 in whole or in part. To that end, each computing device includes computing resources capable of executing at least one of the blocks included in the exemplary method 1200. Computing resources include, for example, a CPU, a GPU, a tensor processing unit (TPU), memory, disk space, incoming and / or outgoing bandwidth, interface(s) (e.g., I / O interfaces and APIs, or both), controller device(s), power supply, combinations of the foregoing, and / or similar resources. In one example, one or more computing systems can include programming interface(s), an operating system, software for configuring and / or controlling a virtualization environment, firmware, and similar resources. A system of computing devices may be referred to as a computing system.

[0062] In some cases, a computing device executing example method 1200 may host, among other software components / modules, ingest module 130, normalization module 140, transformation module 150 (possibly including tessellation component 310), and alignment module 160. The computing device may execute example method 1200, for example, by executing one or more instances of one or a combination of ingest module 130, normalization module 140, transformation module 150 (possibly including tessellation component 310), and alignment module 160. Thus, in response to execution, ingest module 130, normalization module 140, transformation module 150 (possibly including tessellation component 310), and alignment module 160, individually or in combination, may perform operations corresponding to the blocks of example method 1200.

[0063] In block 1210, the computing device may receive a reference image of the biological tissue. The reference image may be acquired by optical imaging of a reference slide of the biological tissue. In block 1215, the computing device may receive a displacement image of the biological tissue. The displacement image may also be acquired by optical imaging of a displacement slide of the biological tissue. The reference slide and the displacement slide may be acquired, for example, by serially slicing an organ of the subject. As described herein, the position of a pixel in the reference image may be defined relative to a reference 2D coordinate system, and the position of a pixel in a particular displacement image need not be defined within the reference 2D coordinate system. Thus, a pixel at a particular location in the displacement image need not correspond to a pixel at the same location in the reference image.

[0064] At block 1220, the computing device may apply a normalization process to the particular reference image to obtain a normalized reference image. At block 1225, the computing system may apply a normalization process to the displacement image to generate a normalized displacement image. The normalization process may include various operations involving the tissue image and, optionally, transcriptome profiling data acquired on the same slide used to acquire the tissue image. To apply the normalization process to the reference image and the displacement image, the computing device may perform exemplary method 1300 ( FIG. 13 ) and, optionally, exemplary method 1400 ( FIG. 14 ), described below. When performing these exemplary methods, the reference image and the displacement image individually serve as input tissue images in each application of the normalization process.

[0065] At block 1230, the computing device may perform a first alignment of the normalized displaced image with respect to the particular normalized reference image. As described herein, performing the first alignment includes determining a coarse transformation between the normalized displaced image and the normalized reference image. Performing such an initial alignment generates a group of parameters defining the coarse transformation and also generates a second displaced image. The coarse transformation may be, for example, an affine transformation.

[0066] In block 1235, the computing device can provide the reference image to the alignment model. In block 1240, the computing device can provide a second specific displaced image to the alignment model. The reference image and the second displaced image can be provided to the alignment model simultaneously. The alignment model can be an ML model trained using synthetic images. Thus, the alignment is tissue-independent. The ML model includes a neural network having one or more layers configured to extract features from the normalized reference image and the second displaced image, and one or more second layers configured to output a deformation vector field and an inverse deformation vector field corresponding to the deformation vector field. In some cases, the ML model includes a CNN (e.g., CNN 200 (FIG. 2)). The deformation vector field represents a registration transformation between the reference image and the second displaced image. In one example, the alignment model is alignment model 158 (FIG. 1).

[0067] In block 1245, the computing device may perform a second alignment of the second displaced image with respect to the reference image by applying an alignment model to the reference image and the second displaced image. This application generates a deformation vector field that represents an alignment transformation between the reference image and the second displaced image. As part of applying the alignment model, the computing device (e.g., via dense alignment component 156 (FIG. 1)) may also determine a corresponding inverse deformation vector field by integrating the inverse gradient (also called negative flow) of the deformation vector field.

[0068] In block 1250, the computing device can receive reference spatial coordinates of spots in a first transcriptome profile of a biological tissue. The first transcriptome profile corresponds to a reference image. In practice, the first transcriptome profile can be obtained from a reference slide (e.g., via the transcriptome profiling device 108 (FIG. 1)). Each spot contains one or more cells.

[0069] In block 1255, the computing device can receive displacement spatial coordinates of spots in a second transcriptome profile of the biological tissue. The second transcriptome profile corresponds to the displacement image. In practice, the second transcriptome profile can be obtained from the displacement slide (e.g., via the transcriptome profiling device 108 (FIG. 1)). As previously described, each spot contains one or more cells.

[0070] At block 1260, the computing device performs a first alignment of the displacement space coordinates relative to the reference space coordinates based on the coarse transformation to obtain second displacement space coordinates of the spots in the second transcriptome profile. As part of the first alignment, the computing device may apply a coarse transformation (e.g., an affine transformation) to the displacement space coordinates to move the spots to the second displacement space coordinates.

[0071] At block 1265, the computing device may perform a second alignment of the second displacement space coordinates of the spots in the second transcriptome profile based on the alignment transformation. As part of the second alignment, the computing device may apply the alignment transformation to the second displacement space coordinates to further move the spots in the second transcriptome profile to defined end positions relative to the reference 2D coordinate system.

[0072] By performing the exemplary method 1200, a computing device can align spots from a reference slide and a displaced slide by relying, as described above, on coarse and alignment transformations determined by aligning the reference and displaced images according to aspects described herein.

[0073] 13 is a flowchart of an exemplary method 1300 for normalizing an image of biological tissue, in accordance with one or more embodiments of the present disclosure. Again, the image may be a histological stain image, an IHC image, or an ISH image (such as a FISH image). A computing device, or in some cases a computing system, performing exemplary method 1200 (FIG. 12) may perform exemplary method 1300.

[0074] At block 1310, the computing device may receive an image of the biological tissue. The image may be a histological staining image. Such an image may be obtained using any of a variety of histological staining techniques. Thus, the image may be an RGB color image. In some cases, the image of the biological tissue is a reference image of the biological tissue. In some cases, the image of the biological tissue is a displacement image of the biological tissue.

[0075] At block 1320, the computing device may determine whether to apply color normalization to the input image. To do so, the computing device may apply a color deviation criterion and / or a color mismatch criterion to the input image. In response to a positive determination (the "Yes" branch), the computing device may normalize the color of the image at block 1330 and then direct the flow of the exemplary method 1300 to block 1340. Color normalization of the input image may include scaling each RGB color channel so that, for each pixel of the image, the intensity value of each channel is within the interval [0, 1]. In response to a negative determination (the "No" branch), the computing device may direct the flow of the exemplary method to block 1340.

[0076] At block 1340, the computing device may generate a tissue mask image for the input image. The tissue mask image may be generated by separating tissue foreground pixels and non-tissue background pixels, as described herein.

[0077] At block 1350, the computing device may set non-tissue background pixels as black pixels. The tissue mask image with black pixels constitutes a normalized input image. The exemplary method 1300 may optionally include block 1360, in which the computing device may apply one or more shaping operations to the normalized input image. As described herein, shaping operations may include cropping and resizing. The normalized input image may be cropped to remove unwanted background portions and resized to a size suitable for input into the ML alignment model referenced in exemplary method 1200.

[0078] 14 is a flowchart of an exemplary method 1400 for normalizing an image of biological tissue using transcriptome profiling data, in accordance with one or more embodiments of the present disclosure. As described herein, the image may be a histological staining image, an IHC image, or an ISH image (such as a FISH image). A computing device, or in some cases a computing system, performing exemplary method 1200 (FIG. 12) may perform exemplary method 1400.

[0079] At block 1410, a computing device may receive transcriptome profiling data corresponding to an image of the biological tissue. The transcriptome profiling data may be obtained from a biological slide used to obtain the image of the biological tissue.

[0080] At block 1420, the computing device may generate, based on the transcriptome profiling data, a label map including first labels associated with a plurality of first pixels and second labels associated with a plurality of second pixels.

[0081] In block 1430, the computing device can generate a contour map based on the label map. As described herein, the contour map reflects the transcriptome-derived spatial organization of a tissue spot (or a cell in a single-cell resolution measurement). In effect, a first contour line in the contour map defines a boundary separating a first subset of pixels having a first cluster label from a second subset of pixels having a second cluster label.

[0082] At block 1440, the computing device may identify particular sets of pixels in the normalized version of the image that correspond to each contour line in the contour map. Thus, the computing device may identify a first set of pixels that corresponds to a first contour line in the contour map, and may also identify a second set of pixels that corresponds to a second contour line in the contour map.

[0083] Based on the identified particular pixel set, the computing device may overlay a contour map on a normalized version of the image at block 1450. To do so, the computing device may update the normalized version of the image by changing the values ​​of each of the pixels that make up the particular pixel set, where each value may be generated from a linear combination of a first value from the normalized version of the image and a second value based on the transcriptome profiling data.

[0084] As mentioned above, converting transcriptome data into a contour map and incorporating the contour map into tissue images can minimize the impact of noise and batch effects in the alignment of a pair of tissue images.

[0085] FIG. 15 is a flowchart of an example method 1500 for training an ML alignment model (also referred to as an alignment model) in accordance with one or more embodiments of the present disclosure. A computing device, or a system of computing devices, can execute the example method 1500 in whole or in part. To that end, each computing device includes computing resources that can execute at least one of the blocks included in the example method 1500. The computing resources include, for example, a CPU, a GPU, a TPU, memory, disk space, incoming and / or outgoing bandwidth, interface(s) (such as an I / O interface or API, or both), controller device(s), power supply, combinations of the foregoing, and / or similar resources. In one example, one or more computing devices may include programming interface(s), an operating system, software for configuring and / or controlling a virtualization environment, firmware, and similar resources. A system of computing devices may be referred to as a computing system.

[0086] In some cases, a computing system executing exemplary method 1500 may host, among other software components / modules, image generation module 610 ( FIG. 6 ) and learning module 620 ( FIG. 6 ). The computing system may execute exemplary method 1500, for example, by executing one or more instances of either image generation module 610 or learning module 620. Thus, in response to execution, image generation module 610 and learning module 620 may execute operations, individually or in combination, corresponding to the blocks of exemplary method 1500.

[0087] At block 1510, the computing system may generate a plurality of training image pairs based on the plurality of training label map pairs. Each pair of the plurality of training image pairs includes a training reference image and a training displacement image. Each pair of the plurality of training label map pairs comprises a training reference label map and a training displacement label map. The computing system may generate the plurality of training image pairs by implementing the example method 1600 shown in FIG. 16 and described below.

[0088] At block 1520, the computing system may train an ML alignment model for aligning the evaluation displaced image of the biological tissue with the evaluation reference image of the biological tissue based on the plurality of label map pairs and the plurality of training image pairs. The ML alignment model may generate a deformation vector field representing an alignment transformation between the evaluation reference image and the evaluation displaced image. The ML alignment model may also generate an inverse deformation vector field corresponding to the deformation vector field. Training the ML alignment model may include determining a solution to an optimization problem for a loss function based on (i) a similarity metric for the training label map pair and (ii) a deformation vector field associated with a first reference training image and a first displaced training image pair among the plurality of training image pairs. Such a solution defines a trained machine learning alignment model. In one example, the trained ML alignment model may be alignment model 158 (FIG. 1).

[0089] A solution to the optimization problem can be determined iteratively. Accordingly, such solution determination can continue iteratively until a termination criterion is met. Determining a solution to the optimization problem may include generating a current deformation field vector by applying a current alignment model to a first training image pair. The current alignment model is set in the current iteration of determining a solution to the optimization problem. Furthermore, determining a solution to the optimization problem may include applying the current deformation field vector to a displacement label map of the first training label map pair to generate an aligned displacement label map. Furthermore, determining a solution to the optimization problem may include determining a value of a loss function (e.g., L(m,f) shown in Equation (1)) based on (i) a reference label map of the first training label map pair, (ii) the aligned displacement label map, and (iii) the current deformation field vector. Furthermore, determining a solution to the optimization problem may include generating a next deformation field vector based on the value of the loss function.

[0090] At block 1530, the computing system may provide the learned alignment model. Providing the learned alignment model may include storing the model in a data storage (e.g., model storage 640 (FIG. 6)) and, in some cases, configuring an interface (such as an API) to allow access to the stored learned alignment model via one or more function calls.

[0091] 16 is a flowchart of an example method 1600 for generating synthetic training data for training an ML alignment model, in accordance with one or more embodiments of the present disclosure. In some cases, a computing device that performs example method 1500 (FIG. 15) may also perform example method 1600. As described herein, the synthetic training data defines instances of training records that include respective quadruplets of synthetic images. Each quadruplet consists of a pair of training label maps and a pair of training images. Generating the synthetic training data includes generating multiple such quadruplets.

[0092] In block 1610, the computing system may construct a label for each pixel that spans a region of a defined size. The region may be a square with N pixels on one side and N pixels on the other side. In some configurations, N=2 q , q is a natural number. For example, N can be 256.

[0093] At block 1620, the computing system may generate a base label map based on the constructed labels. The base label is a composite image spanning a region of a defined size. As described herein, the base label map can be used to generate a first label map and a second label map. Specifically, at block 1630, the computing system may generate a reference label map by warping the base label map using a first simple noise field. Furthermore, at block 1640, the computing system may generate a displaced label map by warping the base label map using a second simple noise field. Although not shown, the computing system may associate the reference label map and the displaced label map with a particular set of four composite images.

[0094] At block 1650, the computing system may generate a training reference image based on the reference label map. Further, at block 1660, the computing system may generate a training displacement image based on the displacement label map.

[0095] At block 1670, the computing system may associate a training reference image and a training displacement image with a training image pair within a particular set of four composite images.

[0096] In block 1680, the computing system may associate and establish a reference label map and a displacement label map with the training label map pair within a particular set of four synthetic images.

[0097] FIG. 17 illustrates an exemplary method 1700 for obtaining an alignment model and aligning images of biological tissue to a common coordinate system in accordance with one or more embodiments of the present disclosure. As described herein, the images can be histological stain images, IHC images, or ISH images (e.g., FISH images). A computing device, or a system of computing devices, can perform the exemplary method 1700 in whole or in part. To that end, each computing device includes computing resources that can execute at least one of the blocks included in the exemplary method 1700. Computing resources include, for example, a CPU, a GPU, a TPU, memory, disk space, incoming and / or outgoing bandwidth, interface(s) (e.g., I / O interface or API, or both), controller device(s), power supply, combinations of the foregoing, and / or similar resources. In one example, one or more of the systems of computing devices can include programming interface(s), an operating system, software for configuring and / or controlling a virtualization environment, firmware, and similar resources. A system of computing devices can be referred to as a computing system.

[0098] In some cases, a computing system implementing exemplary method 1700 may host, among other software components / modules, image generation module 610 ( FIG. 6 ), learning module 620 ( FIG. 6 ), ingestion module 130, normalization module 140, transformation module 150 (possibly including tessellation component 310), and alignment module 160. The computing system may implement exemplary method 1700 by executing, for example, one or more instances of one or a combination of image generation module 610 ( FIG. 6 ), learning module 620 ( FIG. 6 ), ingestion module 130, normalization module 140, transformation module 150 (possibly including tessellation component 310), and alignment module 160. Thus, in response to execution, the image generation module 610 (FIG. 6), the learning module 620 (FIG. 6), the capture module 130, the normalization module 140, the transformation module 150 (which may include the tessellation component 310), and the alignment module 160 may, individually or in combination, perform operations corresponding to the blocks of the exemplary method 1700.

[0099] At block 1710, the computing system may generate a plurality of training image pairs based on the plurality of label map pairs, each pair of the plurality of training image pairs including a training reference image and a training displacement image, and each pair of the plurality of training label map pairs including a training reference label map and a training displacement label map.

[0100] At block 1720, the computing system may train an ML alignment model for aligning an assessment displacement image of the biological tissue with an assessment reference image of the biological tissue based on the plurality of label map pairs and the plurality of training image pairs. The machine learning alignment model may generate a deformation vector field representing an alignment transformation between the assessment reference image and the assessment displacement image. Training the ML alignment model may include determining a solution to an optimization problem for a loss function based on a similarity metric for the label map pairs and the deformation vector field associated with a first pair of reference training image and first displacement training image of the plurality of training image pairs. Such a solution defines a trained machine learning alignment model.

[0101] At block 1730, the computing system may receive a particular reference image of the biological tissue and a particular displacement image of the biological tissue.

[0102] At block 1740, the computing system can apply a normalization process to the particular reference image and the particular displaced image. As described herein, the normalization process can include various operations involving the tissue image and, optionally, transcriptome profiling data acquired on the same slide used to acquire the tissue image. To apply the normalization process to the reference image and the displaced image, the computing device can perform exemplary method 500 ( FIG. 5 ) and, optionally, exemplary method 600 ( FIG. 6 ), described below. When performing these exemplary methods, the reference image and the displaced image individually serve as input tissue images in each application of the normalization process.

[0103] At block 1750, the computing system may perform a first registration of the particular normalized displacement image with respect to the particular normalized reference image. As described herein, performing the first registration includes determining an affine transformation between the normalized displacement image and the normalized reference image. Performing such a first registration results in a second displaced image.

[0104] At block 1760, the computing system may provide the particular reference image and the second particular displaced image to the trained ML alignment model.

[0105] At block 1770, the computing system may perform a second alignment of the second particular displaced image to the particular reference image by applying the machine learning alignment model to the particular reference image and the second particular displaced image.

[0106] 18A-18J show additional alignment performance and / or validation data associated with various scenarios having different slide types and slide placements in accordance with embodiments of the present disclosure. In Figures 18A-18J, performance and / or validation statistics associated with embodiments of the present disclosure are collectively referred to as "ML Aligner" solely for nomenclature convenience.

[0107] Figures 18A and 18B compare the performance of the ML aligner trained using the synthetic images described herein with that of previously published methods, referred to herein and in the figures as "PASTE" and "GPSA." Figure 18A shows a synthetic / reference image 1801, or a histological image of a real tissue, that was not used for training. This image 1801 was generated by adding simplex noise of various amplitudes. Figure 18B shows a reference image 1803 of a coronal slice / slide of a mouse hindbrain. Image 1803 was generated using simplex noise wrapping and manual wrapping to create a set of displaced images. As shown in column (A) of Figures 18A and 18B, images 1801 and 1803 were digitally distorted to low, medium, or high levels or manually warped to generate a set of displaced / displaced images. The original, unwarped images (1801 and 1803) were used as reference images. The degree of distortion is quantified as a "dynamic" NCC score in bar graphs 1805 and 1807 shown in Figures 18A and 18B, respectively. The ML aligner was used in conjunction with affine methods (e.g., affine transformation) and nonlinear alignment methods provided by the Advanced Normalization Tool (ANTs) to align each shifted / displaced image to the corresponding reference image 1801 or 1803. The aligned images from each method are shown in Figures 18A and 18B, and the NCC scores after alignment are shown in bar graphs 1805 and 1807. The NCC values ​​for the ML aligner and ANTs are averages over 10 replicate runs. The associated error bars are also plotted but are very small. The statistical significance of the increase in the NCC value for the ML aligner compared to the other methods is indicated by an asterisk and explained in the p-value annotations 1809 (Figure 18A) and 1811 (Figure 18B) to the left.

[0108] Figure 18C shows the evaluation of the alignment of digitally warped spatial transcriptome slices / slides / images of a mouse brain using ML Aligner. Figure 18C was compared with the performance of previously published methods, "PASTE" and "GPSA," using reference image 1811. Specifically, reference image 1811 shows a mouse hindbrain sagittal slice / slide / image profiled using the 10xGenomics Visium platform ("Visium"). Reference image 1811 was digitally warped using simplex noise to low levels (noise amplitude = 5, NCC = 0.606 for the warped image), medium levels (noise amplitude = 10, NCC = 0.566 for the warped image), or high levels (noise amplitude = 20, NCC = 0.534 for the warped image), generating the set of shifted slices / slides / images shown in column (A) of Figure 18C (noise frequency remained at 1 throughout the warping process). As shown in legend 1813, gray dots correspond to spots in reference image 1811. Legend 1813 also indicates that blue crosses correspond to spots in the series of shifted slices / slides / images shown in column (B) of Figure 18C, and red crosses correspond to spots in the aligned images in columns (B)–(D) of Figure 18C. The discrepancy between the spatial coordinates of spots in each shifted slice / slide / image and those in the reference slice / slide / image was quantified by the mean squared error (mSE), shown in bar graph 1815. Each shifted spatial transcriptome slice / slide / image was aligned to reference image 1811 using ML Aligner in conjunction with the previously published PASTE and GPSA methods. Figure 18C shows the spot coordinates before alignment (blue crosses shown in legend 1813) and after alignment (red crosses shown in legend 1813), along with the spot coordinates in reference image 1811 (gray dots shown in legend 1813), for easy visual comparison. The MSE after alignment by each method is shown in bar graph 1815. Values ​​for ML aligner and GPSA are averages over 10 runs and are shown in bar graph 1815 with small error bars.

[0109] Figure 18D shows a graph 1819 illustrating the spatial consistency of the class labels 1817 of tissue spots in the reference image 1811 and the slide aligned with the ML aligner in Figure 18C. The presented data is taken from the highly distorted instance in Figure 18C (simple noise amplitude = 20, NCC of the deformed image = 0.534). Spots in the reference image 1811 (shown as circles in Figure 18C) and the transferred image (shown as crosses in Figure 18C) share the same color code for class 1817. The displayed spots on the transferred slide were aligned with the ML aligner to achieve a MSE in the top 10% of the magnitude compared to the reference image. Most of the crosses shown in Figure 18C are correctly positioned in the region containing the same colored circle, indicating that the spatial location is valid with respect to the class label 1817.

[0110] FIG. 18E shows graphs 1821A-F depicting GPSA loss trajectories for the moving slides shown in column (D) of FIG. 18C when aligned at low strain (1821A, 1821B), medium strain (1821C, 1821D), and high strain (1821E, 1821F). Graphs 1821A and 1821B depict GPSA loss trajectories for the moving slides shown in FIG. 18D aligned at low strain. Graphs 1821C and 1821D depict GPSA loss trajectories for the moving slides shown in FIG. 18D aligned at medium strain. Graphs 1821D and 1821E depict GPSA loss trajectories for the moving slides shown in FIG. 18D aligned at high strain. In graphs 1821A, 1821C, and 1821E, GPSA was run on the Visium platform with parameters 1823 (m_X_per_view=200, m_G=200, N_GENES=10). According to the loss trajectory, GPSA was run for over 20,000 epochs before reaching convergence. Graphs 1821B, 1821D, and 1821F show the loss trajectory of GPSA when the same moving slide (shown in Figure 18D) was aligned with parameters 1825 (N_GENES=100, while keeping m_X_per_view=200 and m_G=200).

[0111] Figure 18F shows graphs 1827A-1827D. Each graph in Figure 18F is a comparison of the GPSA alignment spot coordinates (shown as red crosses) and the reference coordinates (shown as gray dots) when aligning the moving slide shown in column (D) of Figure 18C. For graph 1821A, the Visium platform parameter settings were m_X_per_view=200, m_G=200, and N_GENES=100. For graph 1821B, the Visium platform parameter settings were m_X_per_view=100, m_G=100, and N_GENES=100. For graph 1821C, the Visium platform parameter settings were m_X_per_view=50, m_G=50, and N_GENES=100. In graph 1821D, the parameter settings for the Visium platform were m_X_per_view = 100, m_G = 100, and N_GENES = 10. In all cases, GPSA produced aggregated spatial coordinates, as shown in Figure 18F.

[0112] Figure 18G shows the performance of ML Aligner in aligning digitally warped spatial transcriptome slices / slides / images of a human lymph node. Figure 18G was used as a reference image 1829 to compare the performance of previously published methods, "PASTE" and "GPSA." Specifically, reference image 1829 is a human lymph node cross-section profiled on the 10X Genomics Visium platform. Reference image 1829 was digitally warped using simplex noise to low (noise amplitude = 5, NCC = 0.621 for the warped image), medium (noise amplitude = 10, NCC = 0.540 for the warped image), or high (noise amplitude = 20, NCC = 0.504 for the warped image) levels to generate the set of displaced slices / slides / images shown in column (A) of Figure 18G (noise frequency remained 1 throughout the warping process). As shown in legend 1831, gray dots correspond to spots in reference image 1829. Legend 1831 also indicates that blue crosses correspond to spots in the series of shifted slices / slides / images shown in column (B) of Figure 18G, and red crosses correspond to spots in the aligned images in columns (B)–(D) of Figure 18G. The discrepancy between the spatial coordinates of spots in each shifted slice and the spatial coordinates of spots in the reference slice / slide / image was quantified by the MSE (method) shown in bar graph 1833. ML Aligner was applied to align each shifted spatial transcriptome slice / slide / image in column (B) of Figure 18G to the reference image 1829 using the previously published methods PASTE and GPSA. Figure 18G also shows the spot coordinates before alignment (blue crosses shown in legend 1831) and after alignment (red crosses shown in legend 1831), along with the spot coordinates in the reference image 1829 (gray dots shown in legend 1831), for easy visual comparison. The MSE after alignment by each method is shown in bar graph 1833. Values ​​for ML aligner and GPSA are averages over 10 runs and are shown in bar graph 1833 with small error bars.

[0113] Figure 18H shows the performance of the ML aligner in de novo alignment of spatial transcriptome slices / slides / images. The top row of Figure 18H displays four moving spatial transcriptome slices / slides / images independently warped from a reference image 1811 acquired from the mouse hindbrain described above. The top row of Figure 18H displays spot coordinates as crosses in the corresponding images (slide 1: red, slide 2: green, slide 3: blue, slide 4: orange, as indicated by the legend 1835). For Figure 18H, warping of the reference image 1811 was performed using simplex noise with a random seed, amplitude of 15, and frequency of 1. The average pairwise NCC between tissue images in the moving slices / slides / images is 0.198. The average pairwise MSE between spot coordinates within the moving slices / slides / images is 0.10. The bottom row of Figure 18H shows the spot coordinates for the four slices / slides / images shown in the top row before alignment ("unaligned coordinates") and after alignment by ML Aligner, PASTE, and GPSA, respectively, using the same colors and cross symbols as shown in the top row of Figure 18H. The average MSE after alignment across all slice pairs is 0.046, 0.105, and 0.55 for ML Aligner, PASTE, and GPSA, respectively.

[0114] Figure 18I shows the results of applying the ML aligner to actual spatial transcriptome slices / slides / images. The top row 1837 of Figure 18I shows two consecutive sagittal sections of a mouse brain (profiled in Visium). The two slices / slides / images are labeled "Slide 1" and "Slide 2," respectively. The differences between the slices / slides / images shown in 1837 before alignment are shown in the overlaid histological image in 1839. The lower (red) section of 1839 corresponds to slide 1 of 1837, and the upper (green) section of 1839 corresponds to slide 2 of 1837. The spatial consistency of the tissue spot clusters between the slices / slides / images shown in 1837 was assessed in the form of overlaid class label maps before (1841) and after (1843) alignment. The slices / slides / images in 1837 contain identical classes, which are shown in Figure 18I. In Figure 18I, shades of yellow and blue are seen in 1841 and 1843, respectively. Each overlay map shown in 1841 and 1843 was generated by combining the colors of both slices / slides / images (1837) with a blending ratio of 0.5 for each RGB channel. The various shades of gray in 1841 and 1843 indicate matching class labels between the two slices / slides / images in 1837. The Dice score between the unaligned label maps (1841) is 0.794. After alignment with the ML aligner, the Dice score increases to 0.867 (1843).

[0115] The bottom row of Figure 18I, 1845, shows four biological replicates of a dissected mouse olfactory bulb as slides 1–4. To demonstrate spatial mismatch, the bottom row of Figure 18I, 1849, overlays the tissue slices / slides / images from 1845, representing each slice / slide / image by color ("Slide 1" is gray, "Slide 2" is red, "Slide 3" is green, and "Slide 4" is blue). Each class in 1849 (the label map) is represented by a different shade of gray, red, green, and blue, respectively. In the label map overlay, the colors of all four slices / slides / images in 1847 are blended with an equal blending ratio of 0.25 for each RGB channel. Areas with different shades of gray between class labels across the four slices / slides / images in 1847 indicate agreement. Before alignment with ML Aligner, the Dice score was 0.498 (shown in 1851 of Figure 18I). After alignment with the ML aligner, the Dice score improves to 0.819 (shown as 1853 in Figure 18I).

[0116] Figure 18J shows two graphs (1855 and 1857) demonstrating batch effects in the mouse olfactory bulb spatial transcriptome dataset. Graph 1855, shown on the left side of Figure 18J, shows the tissue spots from the four aforementioned mouse olfactory bulb slides (slides 1–4, shown in 1845 in Figure 18I) clustered based on gene expression profiles and visualized using UMAP. The spots in graph 1855 form four slide-specific clusters, indicating the existence of non-negligible batch effects across slides. As shown in legend 1855A of graph 1855, the red cluster corresponds to slide 1 of 1845, the green cluster corresponds to slide 2 of 1845, the blue cluster corresponds to slide 3 of 1845, and the purple cluster corresponds to slide 4 of 1845. Graph 1857, shown on the right side of Figure 18J, shows the tissue spot coordinates for slides 1–4, shown in 1845 in Figure 18I, after performing de novo alignment using GPSA. As shown in legend 1857A of graph 1855, the red cross in 1857 corresponds to slide 1 of slide 1845, the green cross in 1857 corresponds to slide 2 of 1845, the blue cross in 1857 corresponds to slide 3 of 1845, and the purple cross in 1857 corresponds to slide 4 of 1845. Graph 1857 shows significant deviations of the coordinates from the expected square grid, highlighting the limitations of GPSA. Furthermore, the poor agreement of spot coordinates between aligned slides (average pairwise MSE = 24.09) suggests that GPSA is susceptible to batch effects in transcriptome data.

[0117] Methods and systems for image alignment according to aspects described herein can be implemented on a computing system 1900 shown in FIG. 19 , as described below. The computer-implemented methods, devices, and systems disclosed herein can utilize one or more computing devices to perform one or more functions at one or more locations. FIG. 19 is a block diagram illustrating an example of a computing system 1900 for performing the disclosed methods and / or implementing the disclosed systems. The computing system 1900 is merely an example of an operating environment and is not intended to suggest any limitation as to the scope of use or functionality of the operating environment architecture. Neither should the operating environment be interpreted as having any dependency or requirement relating to any one or any combination of components illustrated in the exemplary operating environment. The computing environment 1900 shown in FIG. 19 can embody or constitute the computing system 100 (FIG. 1 or FIG. 3). The computing system 1900 can implement various functions described herein related to aligning images of biological tissue to a common coordinate system. For example, one or more computing devices forming computing system 1900 may include capture module 130, normalization module 140, transformation module 150 (possibly including tessellation component 310), and alignment module 160. Additionally, or in some embodiments, one or more computing devices forming computing system 2500 may also include image generation module 610 and learning module 620 as described herein.

[0118] Computer-implemented methods, devices, and systems according to the present disclosure can operate with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with the systems and methods include, but are not limited to, personal computers, server computers, laptop devices, and multiprocessor systems. Additional examples include set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0119] The processing of the disclosed computer-implemented methods, devices, and systems may be performed by software components. The disclosed systems, devices, and computer-implemented methods may be described in the general context of computer-executable instructions (such as program modules) being executed by one or more computers or other processing devices. Generally, program modules include computer code, routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The disclosed computer-implemented methods may also be practiced in a grid-based, distributed computing system in which tasks are performed by remote processing devices linked through a communications network. In a distributed computing system, program modules may be located in both local and remote computer storage media, including memory storage devices.

[0120] Additionally, the systems and computer-implemented methods disclosed herein may be performed via a general-purpose computing device in the form of a computing device 1901. Components of the computing device 1901 may include one or more processors 1903, a system memory 1912, and a system bus 1913 that operatively couples various system components, including the one or more processors 1903, to the system memory 1912. The system may utilize parallel computing.

[0121] The system bus 1913 represents one or more of several possible types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, or a local bus, using any of a variety of bus architectures. The system bus 1913, and all buses specified herein, may be implemented over a wired or wireless network connection, and each of the subsystems, including one or more processors 1903, mass storage device 1904, operating system 1905, software 1906, data 1907, network adapter 1908, system memory 1912, input / output interface 1910, display adapter 1909, display device 1911, and human-machine interface 1902, may be contained within one or more remote computing devices 1914a, b, c, at physically separate locations, connected through this type of bus, effectively implementing a fully distributed system.

[0122] Computing device 1901 typically includes a variety of computer-readable media. Exemplary readable media may be any available media that is accessible by computing device 1901, including, for example, both volatile and nonvolatile media, removable and non-removable media. Main memory 1912 includes computer-readable media in the form of volatile memory, such as random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM). Main memory 1912 typically includes data, such as data 1907, and / or program modules, such as operating system 1905 and software 1906, that are immediately accessible to and / or presently being operated on by one or more processors 1903. For example, software 1906 may include capture module 130, normalization module 140, transformation module 150 (which may include tessellation component 310), and alignment module 160. Additionally, or in other embodiments, software 1906 may also include image generation module 610 and learning module 620. The operating system 1905 may be implemented, for example, in one of the Windows operating systems, Unix, or Linux.

[0123] In another aspect, computing device 1901 may also include other removable / non-removable, volatile / non-volatile computer storage media. For example, Figure 19 illustrates a mass storage device 1904 that can provide non-volatile storage of computer code, computer-readable instructions, data structures, program modules, and other data for computing device 1901. For example, without limitation, mass storage device 1904 can be a hard disk, a removable magnetic disk, a removable optical disk, a magnetic cassette or other magnetic storage device, a flash memory card, a CD-ROM, a digital versatile disk (DVD) or other optical storage, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0124] Any number of program modules may be stored on mass storage device 1904, including, by way of example, operating system 1905 and software 1906. Operating system 1905 and software 1906 (or some combination thereof) may each include elements of programming and software 1906. Data 1907 may be stored on mass storage device 1904. Data 1907 may be stored in any of one or more databases known in the art. Examples of such databases include DB2®, Microsoft® Access, Microsoft® SQL Server, Oracle®, mySQL, PostgreSQL, SQLite, etc. Databases may be centralized or distributed across multiple systems. Mass storage device 1904 and main memory 1912, individually or combined, may embody or include memory 170. Additionally, or in some embodiments, mass storage device 1904 may embody or include image storage 630 and model storage 640. Additionally or in yet other embodiments, a combination of mass storage devices 1904 that may be present in remote computing devices 1914 a, b, c may embody or include the data storage 110 .

[0125] In another aspect, a user may enter commands and information into the computing device 1901 through input devices (not shown). Examples of such input devices include, but are not limited to, keyboards, pointing devices (e.g., a “mouse”), microphones, joysticks, scanners, tactile input devices such as gloves, other body coverings, etc. These and other input devices may be connected to the one or more processors 1903 through a human-machine interface 1902 coupled to the system bus 1913, but may also be connected by other interface and bus structures such as a parallel port, a game port, an IEEE 1394 port (also known as a Firewire port), a serial port, or a universal serial bus (USB).

[0126] In yet another aspect, a display device 1911 may also be connected to the system bus 1913 via an interface, such as a display adapter 1909. It is envisioned that the computing device 1901 may have more than one display adapter 1909, and that the computing device 1901 may have more than one display device 1911. For example, the display device 1911 may be a monitor, an LCD (liquid crystal display), or a projector. In addition to the display device 1911, other output peripheral devices may include components such as speakers (not shown) and a printer (not shown), which may be connected to the computing device 1901 via the input / output interface 1910. Any operations and / or results of the present methods may be output in any form to an output device. Such output may be any form of visual representation, including, but not limited to, text, graphics, animation, audio, tactile, etc. The display device 1911 and the computing device 1901 may be part of a single device or separate devices.

[0127] The computing device 1901 can operate in a networked environment using logical connections to one or more remote computing devices 1914a, b, c. For example, the remote computing devices may be personal computers, portable computers, smartphones, server devices, router devices, network computers, peer devices, or other common network nodes. The logical connections between the computing device 1901 and the remote computing devices 1914a, b, c may be established over a network 1915, such as a LAN and / or general WAN. Such network connections may be through a network adapter 1908. The network adapter 1908 may be implemented in both wired and wireless environments.

[0128] For purposes of illustration, application programs and other executable program components, such as the operating system 1905, are illustrated herein as separate blocks, with the understanding that such programs and components may reside at various times in different storage components of the computing device 1901 and be executed by one or more processors 1903 of the computer. The executable aspects of the software 1906 may be stored on or transmitted across some form of computer-readable media. Any of the disclosed methods may be performed by computer-readable instructions embodied on a computer-readable medium. A computer-readable medium may be any available medium that can be accessed by a computer. By way of example, and not limitation, computer-readable media may include “computer storage media” and “communications media.” “Computer storage media” includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Exemplary computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer.

[0129] It is to be understood that the methods and systems described herein are not limited to the particular acts, processes, components, or structures described, or to the ordering or specific combinations of such acts or components described. It is also to be understood that the terminology used herein is for the purpose of describing example embodiments only, and is not intended to be limiting or restrictive.

[0130] As used herein, the singular forms "a," "an," and "the" include both singular and plural referents unless the context clearly dictates otherwise. Values ​​expressed as approximations by use of antecedents such as "about" or "approximately" are intended to include a reasonable variation from the referenced value. When such approximations fall within a range, not only are the endpoints considered approximations, but the magnitude of the range is also considered approximations. Lists are to be considered exemplary and are not limited or restricted to the elements comprising the list or to the order in which the elements are listed, unless the context clearly dictates otherwise.

[0131] Throughout the specification and claims of this disclosure, the following words have the meanings specified. "Comprise," and variations of the terms such as "comprising" and "comprises," mean, for example, the inclusion of, but not limitation to, other accessories, components, elements, or operations, and are not intended to exclude them. Variations of the terms such as "include" and "including" are not intended to mean restricted or limited to what is shown to be included, or to exclude what is not shown. "May" means permissive but not restrictive or limiting. "Optional" or "optionally" means something that may or may not be included without changing the outcome or what is being described. Variations of the terms such as "prefer," "preferably," or "preferably" are exemplary and mean something that is more ideal, but not required. "Etc." means something that serves merely as an example.

[0132] The operations and components described herein as being used to perform the disclosed methods and to construct the disclosed systems are illustrative unless the context explicitly dictates otherwise. When combinations, subsets, interactions, groups, etc. of these operations and components are disclosed, specific reference to the various individual and collective combinations and permutations of each of these may not be explicitly disclosed, but it is understood that each is specifically contemplated and described herein for all methods and systems. This applies to all aspects of this application, including, but not limited to, the operations in the disclosed methods and / or components disclosed in the systems. Thus, where there are various additional operations that may be performed or components that may be added, it is understood that each of these additional operations can be performed and components can be added in any specific embodiment or combination of embodiments of the disclosed systems and methods.

[0133]

[0013] Embodiments of the present disclosure may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the methods and systems may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied in the storage medium. Any suitable computer-readable storage medium may be utilized, such as a hard disk, CD-ROM, optical storage device, magnetic storage device, memristor, non-volatile random access memory (NVRAM), flash memory, or combinations thereof (whether internal, network, or cloud-based).

[0134] Embodiments of the present disclosure are described with reference to diagrams, flowcharts, and other illustrations of computer-implemented methods, systems, devices, and computer program products. Each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, respectively, may be implemented by processor-accessible instructions. Such instructions may include, for example, computer program instructions (e.g., processor-readable and / or processor-executable instructions). The processor-accessible instructions may be built (e.g., linked and compiled) and stored in processor-executable form in one or more memory devices or one or many other processor-accessible non-transitory storage media. These computer program instructions (built or otherwise) may be loaded into a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine. The loaded computer program instructions may be accessed and executed by one or more processors or other types of processing circuitry. In response to execution, the loaded computer program instructions provide the functionality described with respect to the flowchart blocks (individually or in specific combinations) or block diagram blocks (individually or in specific combinations). Thus, such instructions, when executed on a computer or other programmable data processing apparatus, create means for performing the functions specified in the blocks of the flowchart (individually or in a specific combination) or in the blocks of the block diagram (individually or in a specific combination).

[0135] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a specific manner, whereby the instructions stored in the computer-readable memory produce an article of manufacture that includes processor-accessible instructions (e.g., processor-readable instructions and / or processor-executable instructions) for implementing the functions specified in the flowchart blocks (individually or in a specific combination) or in the blocks in the block diagrams (individually or in a specific combination). Computer program instructions (built or otherwise) can also be loaded into a computer or other programmable data processing apparatus to cause the computer or other programmable device to perform a series of operations to produce a computer-implemented process. The series of operations can be performed in response to execution by one or more processors or other types of processing circuits. Thus, such instructions executed on a computer or other programmable device provide operations for performing the functions specified in the flowchart blocks (individually or in a specific combination) or in the blocks in the block diagrams (individually or in a specific combination).

[0136] Thus, the blocks in the block diagrams and flowchart illustrations support combinations of means for performing the specified functions, combinations of operations for performing the specified functions, and program instruction means for performing the specified functions associated with such diagrams and / or flowchart illustrations. Each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or combinations of dedicated hardware and computer instructions.

[0137] The methods and systems may employ artificial intelligence techniques such as machine learning and iterative learning, examples of which include, but are not limited to, expert systems, case-based reasoning, Bayesian networks, behavior-based AI, neural networks, fuzzy systems, evolutionary computing (e.g., genetic algorithms), swarm intelligence (e.g., ant algorithms), and hybrid intelligent systems (e.g., expert inference rules generated through neural networks or generative rules from statistical learning).

[0138] Although the computer-implemented methods, apparatus, devices, and systems have been described in connection with preferred embodiments and specific examples, the scope is not intended to be limited to the particular embodiments described, as the embodiments herein are intended in all respects to be illustrative and not restrictive.

[0139] Unless expressly stated otherwise, any method described herein is in no way intended to be construed as requiring that its operations be performed in a particular order. Thus, if a method claim does not actually recite the order in which its operations are to be followed, or if the claim or specification does not specifically state that the operations are to be limited to a particular order, no order is intended to be inferred in any respect. This holds for any possible implicit basis for interpretation, including the arrangement of operations or operation flow, the apparent meaning derived from grammatical construction or punctuation, and logical matters regarding the number or type of embodiments described in the specification.

[0140] It will be apparent to those skilled in the art that various modifications and variations can be made without departing from the scope or spirit. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the disclosures herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit being indicated by the following claims. [Example]

[0141] Embodiment 1. A computer-implemented method comprising: receiving a reference image of biological tissue; receiving a displaced image of the biological tissue; applying a normalization process to the reference image to generate a normalized reference image; applying the normalization process to the displaced image to obtain a normalized displaced image; performing a first alignment of the normalized displaced image to the normalized reference image to generate a group of parameters defining a coarse transformation and further generating a second displaced image; feeding the normalized reference image to a machine learning alignment model; feeding the second displaced image to the machine learning alignment model; and performing a second alignment of the second displaced image to the reference image by applying the machine learning alignment model to the reference image and the second displaced image, wherein the application generates a deformation vector field representing the alignment transformation between the reference image and the second displaced image.

[0142] Embodiment 2. The computer-implemented method of embodiment 1, further comprising: receiving reference spatial coordinates of spots in a first transcriptome profile of the biological tissue, the first transcriptome profile corresponding to a reference image, each spot comprising one or more cells; receiving displacement spatial coordinates of spots in a second transcriptome profile of the biological tissue, the second transcriptome profile corresponding to the displacement image; performing a first alignment of the displacement spatial coordinates to the reference spatial coordinates based on a coarse transformation, resulting in second displacement spatial coordinates of the spots in the second transcriptome profile; and performing a second alignment of the second displacement spatial coordinates of the spots in the second transcriptome profile based on the alignment transformation.

[0143] Embodiment 3. A computer-implemented method as described in any one of embodiments 1 or 2, wherein the step of providing the normalized reference image includes the steps of generating a tiling of the normalized reference image, the tiling of the normalized reference image consisting of a first defined number of tile images, and providing each tile image of the tiling of the normalized reference image in turn to the machine learning alignment model.

[0144] Embodiment 4. The computer-implemented method of embodiment 3, wherein the step of providing the second displaced image includes the steps of generating a tiling of the second displaced image, the tiling of the second displaced image consisting of a second defined number of tile images, the second defined number being equal to the first defined number, and sequentially providing each tile image of the tiling of the second displaced image to the machine learning alignment model, wherein a first tile image of the tiling of the second displaced image is provided simultaneously with a first tile image of the tiling of the normalized reference image, wherein the first tile image of the tiling of the second displaced image spans a section of the second displaced image, the first tile image of the tiling of the normalized reference image spans a section of the tiling of the normalized reference image, and the section of the second displaced image and the section of the normalized reference image are identical to each other in terms of arrangement and size.

[0145] Embodiment 5. The computer-implemented method of embodiment 4, wherein applying the machine learning alignment model includes applying the machine learning alignment model to a first tile image of a tiling of the normalized reference image and a first tile image of a tiling of the second displaced image.

[0146] Embodiment 6. The computer-implemented method of embodiment 5, wherein applying the machine learning alignment model generates a first tile deformation vector field and a second tile deformation vector field, and wherein the method further includes combining the first tile deformation vector field and the second tile deformation vector field to at least partially form the deformation vector field.

[0147] Embodiment 7. A computer-implemented method as described in embodiment 6, wherein the first tile deformation vector field and the second tile deformation vector field and each of their adjacent tile deformation vector fields have a common defined area, and the combining step includes, for each pixel within the common defined area, determining a weighted average of one of the first tile deformation vector field or the second tile deformation vector field and each of the adjacent tile deformation vector fields that overlap at that pixel, and assigning the weighted average to the deformation vector field in common for each pixel within the defined area.

[0148] Embodiment 8. The computer-implemented method of any one of the preceding embodiments, wherein the coarse transformation is an affine transformation, and performing the first alignment includes determining an affine transformation between the normalized displacement image and the normalized reference image.

[0149] Embodiment 9. A computer-implemented method according to any one of the preceding embodiments, wherein the normalization process includes receiving an input image of biological tissue, generating a tissue mask image for the input image, and setting non-tissue pixels in the tissue mask image as black pixels, wherein the tissue mask image having black pixels constitutes the normalized input image.

[0150] Embodiment 10. The computer-implemented method of any one of the preceding embodiments, wherein the normalization process further comprises: receiving transcriptome profiling data corresponding to the input image; generating, based on the transcriptome profiling data, a label map including first labels associated with a plurality of first pixels and second labels associated with a plurality of second pixels; generating a contour map based on the label map, wherein the first contour lines in the contour map define boundaries separating a subset of the plurality of first pixels and a subset of the plurality of second pixels; identifying, in the normalized input image, specific pixels corresponding to the first contour lines; and updating the normalized input image by modifying respective values ​​of the specific pixels, wherein each respective value is generated from a linear combination of a first value from the normalized input image and a second value based on the transcriptome profiling data.

[0151] Embodiment 11. The computer-implemented method of any one of the preceding embodiments, wherein the machine learning alignment model comprises a convolutional neural network (CNN) having an encoder module, a decoder module, and a field construction module configured to output a deformation vector field.

[0152] Embodiment 12. The computer-implemented method of any one of the preceding embodiments, wherein the machine learning alignment model comprises a neural network including one or more first layers (of a first type) configured to extract features from the normalized reference image and the second displaced image, and one or more second layers (of a second type) configured to output a deformation vector field.

[0153] Embodiment 13. A computer-implemented method comprising: generating a plurality of training image pairs based on a plurality of training label map pairs, each pair of the plurality of training image pairs including a training reference image and a training displacement image; and determining a solution to an optimization problem relating to a loss function based on a similarity index of the pair of training label maps and a deformation vector field representing an alignment transformation between a first training reference image and a first training displacement image of a pair of the plurality of training image pairs, the solution defining an alignment model for aligning an evaluation displacement image of a biological tissue with an evaluation reference image of the biological tissue.

[0154] Embodiment 14. The computer-implemented method of embodiment 12, wherein determining a solution to the optimization problem includes: generating a current deformation field vector by applying a current alignment model to a first pair of training images, wherein the current alignment model is configured for the current iteration that determines the solution to the optimization problem; applying the current deformation field vector to a displacement label map of the first pair of training label maps to generate an aligned displacement label map; determining a value of a loss function based on (i) the reference label map of the first pair of training label maps, (ii) the aligned displacement label map, and (iii) the current deformation field vector; and generating a next deformation field vector based on the value of the loss function.

[0155] Embodiment 15. The computer-implemented method of embodiment 13, wherein the loss function comprises a Dice similarity coefficient and a gradient of the deformation vector field, the gradient being weighted by a regularization factor.

[0156] Embodiment 16. A computer-implemented method described in any one of embodiments 13 to 15, wherein the alignment model includes a convolutional neural network (CNN) including an encoder module, a decoder module, and a field construction module configured to output a deformation vector field.

[0157] Embodiment 17. A computer-implemented method described in any one of embodiments 13 to 16, wherein the machine learning alignment model includes a neural network including one or more first layers (of a first type) configured to extract features from the normalized reference image and the second displaced image, and one or more second layers (of a second type) configured to output a deformation vector field.

[0158] Embodiment 18. A computer-implemented method described in any one of embodiments 13 to 17, wherein the step of generating multiple pairs of training images includes the steps of constructing labels for each pixel across an area of ​​a defined size, and generating a base label map across the area of ​​the defined size based on the constructed labels.

[0159] Embodiment 19. The computer-implemented method of embodiment 18, wherein the configuring step includes configuring a plurality of sets of simplex noise distributions within each layer, each layer corresponding to a region of a defined size, wherein a first set of the plurality of sets includes a first simplex noise distribution centered at a respective position within the first layer of each layer, and a second set of the plurality of sets includes a second simplex noise distribution centered at a respective position within the second layer; configuring a plurality of predefined labels for each pixel within the region, each of the plurality of predefined labels corresponding to a particular layer of each layer; using the plurality of sets of simplex noise distributions to determine a numerical weight for each of the plurality of predefined labels at a particular pixel within the region; and assigning to the particular pixel a first label corresponding to the first numerical weight having the largest magnitude among the respective numerical weights.

[0160] Embodiment 20. The computer-implemented method of embodiment 19, wherein generating the base label map includes merging each layer having labeled pixels into a single predefined layer that defines the base label map.

[0161] Embodiment 21. The computer-implemented method of embodiment 18, wherein generating a plurality of training image pairs further includes: generating a reference label map by warping the base label map using a first simplex noise field; generating a specific training reference image based on the reference label map and generating a colored image by randomly setting the color of each label in the reference label map; blurring the colored image to generate a blurred image; and applying a bias field to the blurred image, the bias strength field spanning an area of ​​a defined size.

[0162] Embodiment 22. The computer-implemented method of embodiment 21, wherein the step of generating a plurality of training image pairs further includes the steps of: generating a displacement label map by warping the base label map using a second simplex noise field; generating specific training displacement images based on the displacement label map and generating a second colored image by randomly setting the color of each label in the second label map; blurring the second colored image to generate a second blurred image; and applying a bias strength field to the second blurred image.

[0163] Embodiment 23. A computer-implemented method as described in embodiment 22, wherein the step of generating a plurality of training image pairs further includes a step of associating a specific training reference image and a specific training displacement image with a particular pair of the plurality of image pairs.

[0164] Embodiment 24. The computer-implemented method of embodiment 23, further comprising the step of associating and setting a reference label map and a displacement label map with a particular pair of the plurality of training label maps.

[0165] Embodiment 25. A computer-implemented method, comprising: generating a plurality of training image pairs based on a plurality of training label map pairs, each pair of the plurality of training image pairs including a training reference image and a training displaced image; learning a machine learning alignment model for aligning an evaluation displaced image of a biological tissue with an evaluation reference image of the biological tissue based on the plurality of label map pairs and the plurality of training image pairs, the alignment model generating a deformation vector field representing an alignment transformation between the evaluation reference image and the evaluation displaced image; and a specific displaced image of the specific reference image; applying a normalization process to the specific reference image and the specific displaced image; performing a coarse alignment of the normalized specific displaced image to the normalized specific reference image to generate a group of parameters defining a coarse transformation and further generating a second specific displaced image; feeding the specific reference image and the second specific displaced image to a trained machine learning alignment model; and performing a fine alignment of the second specific displaced image to the specific reference image by applying the machine learning alignment model to the specific reference image and the second specific displaced image.

[0166] Embodiment 26. The computer-implemented method of embodiment 25, further comprising the steps of: receiving reference spatial coordinates of spots in a first transcriptome profile of the biological tissue, the first transcriptome profile corresponding to a reference image, each spot comprising one or more cells; receiving specific displacement spatial coordinates of spots in a second transcriptome profile of the biological tissue, the second transcriptome profile corresponding to the displacement image; performing coarse alignment of the specific displacement spatial coordinates with the specific reference spatial coordinates based on the coarse transformation to obtain second specific displacement spatial coordinates of the spots in the second transcriptome profile; and performing fine alignment of the second specific displacement spatial coordinates of the spots in the second transcriptome profile based on the alignment transformation.

[0167] Embodiment 27. A computer-implemented method described in any one of embodiments 25 or 26, wherein the training includes determining a solution to an optimization problem for a loss function based on a similarity metric of a first training label map pair of a plurality of training label map pairs and a deformation vector field associated with a first training reference image and a first training displacement image of a pair of a plurality of training image pairs, the solution defining a trained machine learning alignment model.

Claims

1. 1. A computer-implemented method comprising: receiving a reference image of the biological tissue; receiving a displacement image of the biological tissue; applying a normalization process to the reference image to generate a normalized reference image; applying the normalization process to the displacement image to generate a normalized displacement image; performing a first registration of the normalized displaced image to the normalized reference image to generate a group of parameters defining a coarse transformation and to generate a second displaced image; feeding the normalized reference image to a machine learning alignment model; providing the second displaced image to the machine learning alignment model; performing a second alignment of the second displaced image to the reference image by applying the machine learning alignment model to the reference image and the second displaced image, wherein the applying generates a deformation vector field representing an alignment transformation between the reference image and the second displaced image.

2. receiving reference spatial coordinates of spots in a first transcriptome profile of the biological tissue, the first transcriptome profile corresponding to the reference image, and each spot comprising one or more cells; receiving displacement spatial coordinates of spots in a second transcriptome profile of the biological tissue, the second transcriptome profile corresponding to the displacement image; performing a first alignment of the displacement space coordinates to the reference space coordinates based on the coarse transformation, resulting in the second displacement space coordinates of the spots in the second transcriptome profile; 2. The computer-implemented method of claim 1, further comprising: performing a second alignment of the second displacement space coordinates of the spots in the second transcriptome profile based on the alignment transformation.

3. providing the normalized reference image, generating a tiling of the normalized reference image, the tiling of the normalized reference image consisting of a first defined number of tile images; and feeding each tile image of the tiling of the normalized reference image in turn to the machine learning alignment model.

4. providing the second displacement image; generating a tiling of the second displacement image, the tiling of the second displacement image consisting of a second defined number of tile images, the second defined number being equal to the first defined number; supplying each tile image of the tiling of the second displaced image to the machine learning alignment model in sequence, wherein a first tile image of the tiling of the second displaced image is supplied simultaneously with a first tile image of the tiling of the normalized reference image; Including, the first tile image of the tiling of the second displaced image spans a section of the second displaced image, and the first tile image of the tiling of the normalized reference image spans a section of the tiling of the normalized reference image; The computer-implemented method of claim 3 , wherein the section of the second displacement image and the section of the normalized reference image are identical to one another in terms of placement and size.

5. 5. The computer-implemented method of claim 4, wherein applying the machine learning alignment model comprises applying the machine learning alignment model to the first tile image of the tiling of the normalized reference image and the first tile image of the tiling of the second displaced image.

6. 6. The computer-implemented method of claim 5, wherein applying the machine learning alignment model generates a first tile deformation vector field and a second tile deformation vector field, and the method further comprises combining the first tile deformation vector field and the second tile deformation vector field to at least partially form the deformation vector field.

7. the first tile transformation vector field, the second tile transformation vector field, and each of the adjacent tile transformation vector fields have a common defined area; The bonding step comprises: determining, for each pixel within the common defined region, a weighted average of one of the first tile deformation vector field or the second tile deformation vector field and one of the respective adjacent tile deformation vector fields that overlap at the pixel; and The computer-implemented method of claim 6 , comprising: for each pixel within the common defined region, assigning the weighted average to the deformation vector field.

8. 10. A computer-implemented method according to any one of the preceding claims, wherein the coarse transformation is an affine transformation, and wherein the performing of the first registration comprises determining the affine transformation between the normalized displacement image and the normalized reference image.

9. The normalization process receiving an input image of the biological tissue; generating a tissue mask image for the input image; setting non-tissue pixels in the tissue mask image as black pixels; Including, 10. A computer-implemented method according to any one of the preceding claims, wherein the tissue mask image with the black pixels constitutes a normalized input image.

10. The normalization process receiving transcriptome profiling data corresponding to the input image; generating a label map based on the transcriptome profiling data, the label map including first labels associated with a plurality of first pixels and second labels associated with a plurality of second pixels; generating a contour map based on the label map, wherein a first contour line in the contour map defines a boundary separating the first subset of pixels and the second subset of pixels; identifying a particular pixel in the normalized input image that corresponds to the first contour line; 10. The computer-implemented method of claim 1, further comprising: updating the normalized input image by modifying a value of each of the particular pixels, wherein each of the respective values ​​is generated from a linear combination of a first value from the normalized input image and a second value based on the transcriptome profiling data.

11. 10. The computer-implemented method of any one of the preceding claims, wherein the machine learning alignment model comprises a convolutional neural network (CNN) having an encoder module, a decoder module, and a field construction module configured to output a deformation vector field.

12. the machine learning alignment model: one or more first layers of a first type configured to extract features from the normalized reference image and the second displaced image; and one or more second layers of a second type configured to output the deformation vector field.

13. 1. A computer-implemented method comprising: generating a plurality of training image pairs based on a plurality of training label map pairs, each pair of the plurality of training image pairs including a training reference image and a training displacement image; determining a solution to an optimization problem for a loss function based on a similarity measure of a pair of training label maps and a deformation vector field representing an alignment transformation between a first training reference image and a first training displaced image of a pair of the plurality of training image pairs, the solution defines an alignment model for registration of an estimated displacement image of the biological tissue to an estimated reference image of the biological tissue.

14. determining the solution to the optimization problem, generating a current deformation field vector by applying a current alignment model to a first pair of training images, the current alignment model comprising the current iteration of determining the solution to the optimization problem; applying the current deformation field vector to a displacement label map of a first training label map pair to generate an aligned displacement label map; determining a value of the loss function based on (i) a reference label map of the first training label map pair, (ii) the aligned displacement label map, and (iii) the current deformation field vector; and generating a next deformation field vector based on the value of the loss function.

15. 14. The computer-implemented method of claim 13, wherein the loss function comprises a Dice similarity coefficient and a gradient of the deformation vector field, the gradient being weighted by a regularization factor.

16. 16. The computer-implemented method of any one of claims 13 to 15, wherein the alignment model comprises a convolutional neural network (CNN) including an encoder module, a decoder module, and a field construction module configured to output the deformation vector field.

17. the machine learning alignment model: one or more first layers of a first type configured to extract features from the normalized reference image and the second displaced image; and one or more second layers of a second type, the second layers configured to output the deformation vector field.

18. generating the plurality of training image pairs, constructing a label for each pixel across an area of ​​a defined size; and generating the base label map across the region of the defined size based on the constructed labels.

19. The constituent steps include: constructing a plurality of sets of simplex noise distributions within each layer, any one of the layers corresponding to the region of the defined size, a first set of the plurality of sets including a first simplex noise distribution centered at a respective location within the respective first layer, and a second set of the plurality of sets including a second simplex noise distribution centered at a respective location within the second layer; constructing a plurality of predefined labels for each pixel in the region, any one of the plurality of predefined labels corresponding to a particular one of the respective layers; using the plurality of sets of simplex noise distributions to determine a numerical weight for each of the plurality of predefined labels at a particular pixel within the region; and assigning to the particular pixel a first label corresponding to a first numerical weight having a largest magnitude among the respective numerical weights.

20. 20. The computer-implemented method of claim 19, wherein generating the base label map comprises merging the respective layers having labeled pixels into a single predefined layer that defines the base label map.

21. generating the plurality of training image pairs, generating a reference label map by warping the base label map using a first simplex noise field; Based on the reference label map, randomly setting the color of each label in the reference label map to generate a colored image; blurring the colored image to generate a blurred image; and 20. The computer-implemented method of claim 18, further comprising: generating a specific training reference image by applying a bias field to the blurred image, the bias strength field spanning the region of the defined size.

22. generating the plurality of training image pairs, generating a displacement label map by warping the base label map using a second simplex noise field; Based on the displacement label map, randomly setting the color of each label in the second label map to generate a second colored image; blurring the second colored image to generate a second blurred image; and applying a bias intensity field to the second blurred image; and generating the specific training displacement image by:

23. 23. The computer-implemented method of claim 22, wherein generating the plurality of training image pairs further comprises: associating the particular training reference image and the particular training displacement image with a particular pair of the plurality of image pairs.

24. 24. The computer-implemented method of claim 23, further comprising: configuring the reference label map and the displacement label map to be associated with a particular pair of a plurality of training label map pairs.

25. 1. A computer-implemented method comprising: generating a plurality of pairs of training images based on the plurality of pairs of training label maps, each pair of the plurality of pairs of training images including a training reference image and a training displacement image; a step of training a machine learning alignment model for aligning a displaced evaluation image of a biological tissue with a reference evaluation image of the biological tissue based on a plurality of pairs of label maps and the plurality of pairs of training images, wherein the alignment model generates a deformation vector field representing an alignment transformation between the reference evaluation image and the displaced evaluation image; receiving a specific reference image of the tissue and a specific displacement image of the tissue; applying a normalization process to the particular reference image and the particular displaced image; performing a coarse registration of the normalized specific displaced image to the normalized specific reference image to generate a group of parameters defining a coarse transformation and to generate a second specific displaced image; providing the specific reference image and the second specific displaced image to the trained machine learning alignment model; and performing a fine alignment of the second particular displaced image to the particular reference image by applying the machine learning alignment model to the particular reference image and the second particular displaced image.

26. receiving reference spatial coordinates of spots in a first transcriptome profile of the biological tissue, the first transcriptome profile corresponding to the reference image, each spot comprising one or more cells; receiving specific displacement spatial coordinates of spots in a second transcriptome profile of the biological tissue, the second transcriptome profile corresponding to the displacement image; performing a coarse alignment of the specific displacement space coordinates with the specific reference space coordinates based on the coarse transformation to obtain second specific displacement space coordinates of the spots in the second transcriptome profile; 26. The computer-implemented method of claim 25, further comprising: performing a fine registration of the second specific displacement spatial coordinate of the spot in the second transcriptome profile based on the registration transformation.

27. 27. The computer-implemented method of claim 25, wherein the training step comprises determining a solution to an optimization problem for a loss function based on a similarity metric for a first training label map pair of a plurality of training label map pairs and a deformation vector field associated with a first training reference image and a first training displacement image of one pair of the plurality of training image pairs, the solution defining the trained machine-learned alignment model.

Images