Method, apparatus and system for spatial transcriptome slide alignment

Abstract

Techniques for aligning spatial transcriptome slides of biological tissue are provided. Alignment of coordinates of an image and a spot within a transcriptome map of the biological tissue may utilize an alignment model, which may be machine-learned and may receive as input both a shifted image and a reference image to be aligned. The alignment model may learn differential homeomorphic transforms between the images. Such transformations can be readily applied to alignment of the shifted images and alignment of shifted coordinates of spots within the transcriptome map of a transcriptome. In some cases, transcriptome data may be analyzed and converted similar to the format of an image, and may then be used as supplemental input to the alignment model. The alignment model may be trained in an organization-independent manner, such as by using a composite image.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 381,813, filed on Nov. 1, 2022, the entire content of which is incorporated herein by reference. SUMMARY OF THE INVENTION

[0003] It should be understood that the following general description and the following detailed description are merely illustrative and explanatory and are not restrictive. In one embodiment, the present disclosure provides a computer-implemented method. The computer-implemented method includes: receiving a reference image of a biological tissue; and receiving a shifted image of the biological tissue. The computer-implemented method further includes: applying a normalization process to the reference image to produce a normalized reference image; and applying a normalization process to the shifted image to produce a normalized shifted image. The computer-implemented method further includes: performing a first registration of the normalized shifted image relative to the normalized reference image. Performing such first registration produces a set of parameters defining a coarse transformation and further produces a second shifted image. The computer-implemented method further includes: supplying the normalized reference image to a machine learning alignment model; supplying the second shifted image to the machine learning alignment model; and performing a second registration of the second shifted image relative to the reference image by applying the machine learning alignment model to the reference image and the second shifted image. Applying the machine learning alignment model produces a deformation vector field representing the registration transformation between the reference image and the second shifted image.

[0004] In addition, the computer-implemented method may further include: receiving reference spatial coordinates of spots within a first transcriptomic map of the biological tissue. The first transcriptomic map corresponds to the reference image. Each of the spots includes one or more cells. The computer-implemented method may further include: receiving shifted spatial coordinates of spots within a second transcriptomic map of the biological tissue, the second transcriptomic map corresponding to the shifted image. The computer-implemented method may further include: performing a first registration of the shifted spatial coordinates relative to the reference spatial coordinates based on the coarse transformation to produce second shifted spatial coordinates of the spots within the second transcriptomic map; and performing a second registration of the second shifted spatial coordinates of the spots within the second transcriptomic map based on the registration transformation.

[0005] In yet another embodiment, the present disclosure provides another computer-implemented method. The another computer-implemented method includes: generating a plurality of pairs of training images based on a plurality of pairs of training label maps, wherein each pair of the plurality of pairs of training images includes a training reference image and a training shifted image. The another computer-implemented method further includes: determining a solution to an optimization problem regarding a loss function based on a similarity metric of a pair of training label maps and a deformation vector field representing a registration transformation between a first training reference image and a first training shifted image in a pair of the plurality of pairs of training images. The solution defines an alignment model for registering an evaluation shifted image of a biological tissue relative to an evaluation reference image of the biological tissue.

[0006] Additional elements and advantages of the present disclosure will be set forth in part in the description that follows, and in part will be obvious from the description, or may be learned by practice of the subject disclosure. The advantages of the subject disclosure may be realized and attained by means of the elements and combinations particularly pointed out in the appended claims.

[0007] This summary is not intended to identify key or essential features of the present disclosure, but is merely intended to summarize some features and variations thereof. Other details and features will be described in subsequent sections. Further, the foregoing general description and the following detailed description are both illustrative and explanatory and do not limit the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The drawings are a part of the present disclosure and are incorporated into the specification of this subject matter. The drawings illustrate example embodiments of the present disclosure and, in conjunction with the specification and the claims, at least partially explain various principles, elements, or aspects of the present disclosure. The embodiments of the present disclosure will be described more fully hereinafter with reference to the drawings. However, the various elements of the present disclosure may be implemented in many different forms and should not be construed as limited to the embodiments set forth herein. Throughout the text, like reference numerals refer to like elements.

[0009] Figure 1 An example of a computing system in accordance with one or more embodiments of the present disclosure is shown.

[0010] Figure 1A An example workflow in accordance with an embodiment of the present disclosure is shown.

[0011] Figure 1B An example of a reference image and a shifted image of a biological tissue and corresponding normalized images in accordance with an embodiment of the present disclosure is shown.

[0012] Figure 1C An example of a set of reference spot coordinates in a reference slide and a set of shifted spot coordinates in a shifted slide in accordance with aspects described herein is shown.

[0013] Figure 2 Shows an example of an alignment model according to one or more embodiments of the present disclosure.

[0014] Figure 3 Shows another example of a computing system according to one or more embodiments of the present disclosure.

[0015] Figure 4A Shows an example of an input image of biological tissue, one of the exemplified images being a reference image and the other being an affine transformation shifted image.

[0016] Figure 4B Shows an example of a tiled image according to one or more embodiments of the present disclosure.

[0017] Figure 5A Shows an example of four tiled deformation vector fields according to one or more embodiments of the present disclosure.

[0018] Figure 5B Shows an example of a deformation vector field generated from multiple tiled deformation vector fields according to one or more embodiments of the present disclosure.

[0019] Figure 6 Shows an example computing system for training an alignment model using synthetic training data according to one or more embodiments of the present disclosure.

[0020] Figure 6A Schematically depicts an example process flow for generating a reference label map and a shifted label map and for generating synthetic training reference images and synthetic training shifted images according to aspects described herein.

[0021] Figure 7A Shows an example of the alignment performance for a small slide using simulated deformation according to an embodiment of the present disclosure.

[0022] Figure 7B Shows an example of the alignment performance for a small slide using simulated deformation according to an embodiment of the present disclosure.

[0023] Figure 8A and Figure 8B Shows an example of the alignment performance for a whole-slide mouse brain using simulated deformation according to an embodiment of the present disclosure.

[0024] Figure 8A and Figure 8B Shows an example of the alignment performance for a whole-slide mouse brain using simulated deformation according to an embodiment of the present disclosure.

[0025] Figure 9 Shows an example of the alignment performance for a whole-slide human lymph node using simulated deformation according to an embodiment of the present disclosure.

[0026] Figure 10 Shows an example of the alignment performance for more than two slides using simulated deformation according to an embodiment of the present disclosure.

[0027] Figure 11 Shows the application of an alignment model according to an embodiment of the present disclosure to the alignment of two mouse brain slides using real deformation.

[0028] Figure 12 Shows an example of a computer-implemented method for aligning the spatial coordinates of spots within an image and a biological tissue transcriptome map to a common coordinate system according to one or more embodiments of the present disclosure.

[0029] Figure 13 Shows an example of a computer-implemented method for normalizing an image of a biological tissue according to one or more embodiments of the present disclosure.

[0030] Figure 14 Shows an example of a computer-implemented method for normalizing an image of a biological tissue using transcriptome analysis data according to one or more embodiments of the present disclosure.

[0031] Figure 15 Shows an example of a computer-implemented method for training a machine learning (ML) alignment model according to one or more embodiments of the present disclosure.

[0032] Figure 16 Shows an example of a computer-implemented method for generating synthetic training data for training an ML alignment model according to one or more embodiments of the present disclosure.

[0033] Figure 17 Shows an example of a computer-implemented method for obtaining an alignment model and aligning an image of a biological tissue with a reference coordinate system according to one or more embodiments of the present disclosure.

[0034] Figure 18A Shows the results of the alignment performance according to an embodiment of the present disclosure.

[0035] Figure 18B Shows the results of the alignment performance according to an embodiment of the present disclosure.

[0036] Figure 18C Shows the results of the alignment performance according to an embodiment of the present disclosure.

[0037] Figure 18D Shows the results of the alignment performance according to an embodiment of the present disclosure.

[0038] Figure 18EShows the results of the alignment performance according to an embodiment of the present disclosure.

[0039] Figure 18F Shows the results of the alignment performance according to an embodiment of the present disclosure.

[0040] Figure 18G Shows the results of the alignment performance according to an embodiment of the present disclosure.

[0041] Figure 18H Shows the results of the alignment performance according to an embodiment of the present disclosure.

[0042] Figure 18I Shows the results of the alignment performance according to an embodiment of the present disclosure.

[0043] Figure 18J Shows the results of the alignment performance according to an embodiment of the present disclosure.

[0044] Figure 19 Shows an example of a computing system for implementing slide alignment according to one or more embodiments of the present disclosure. Detailed Description

[0045] Among other technical challenges, the present disclosure recognizes and addresses the following problem: aligning the spatial coordinates of spots within the transcriptome maps of an image and a slide of biological tissue relative to each other with respect to a common coordinate system. For example, the biological tissue may include a section of a human organ or other type of animal organ. Embodiments of the present disclosure include computer-implemented methods, computing devices, computing systems, and computer program products that, individually or in combination, allow a set of spatial coordinates of spots within the transcriptome maps of an image and a slide of biological tissue to be aligned relative to each other with respect to a common coordinate system with another set of spatial coordinates of another image and spots within the transcriptome map of another slide of biological tissue. More specifically, embodiments of the present disclosure individually or in combination utilize an alignment model that can be machine-learned. The alignment model may include, for example, a convolutional neural network and may receive both the shifted image and the reference image to be aligned as inputs and learn the diffeomorphic transformation between these images. Such a transformation, simply referred to as a registration transformation, can be easily applied to the alignment of the shifted image and the alignment of the shifted spatial coordinates of the spots within the transcriptome map. In addition, the transcriptome data can be analyzed and transformed in an image-like format and then used as a supplementary input for the alignment model. The alignment model can be trained in a tissue-independent manner using synthetic images. Thus, the alignment model can be widely applicable.

[0046] Compared with the prior art, embodiments of the present disclosure can provide various improvements and efficiencies for automatically aligning the shifted spatial coordinates of spots within the transcriptomic atlas of a slide of biological tissue. For example, embodiments of the present disclosure can utilize a graphics processing unit (GPU) to replace or supplement a central processing unit (CPU). Accordingly, the processing speed can be superior to that of the prior art. Additionally, or as another example, since optical imaging can be combined with spatial transcriptomic analysis to align the image and the spatial coordinates of the spots, the reliability of the alignment can be superior to that of the prior art. Further, as yet another example, by training and applying a machine learning model to serve as an alignment model, the alignment of the image and the spatial coordinates of the spots avoids using landmarks or other types of fiducial markers and also avoids manual alignment. Additionally, or as yet another example, the training of the machine learning model can be based on synthetic images and is tissue-independent, resulting in much greater flexibility and applicability of the image and spot alignment described in the present disclosure. Additionally, by relying on a tiled stitching method, images larger than the image size used in the training of the machine learning model can be easily analyzed.

[0047] Figure 1Shows an example computing system 100 in accordance with one or more embodiments of the present disclosure. The computing system 100 includes a data acquisition platform 104, which may allow for the measurement and collection of various types of data. Specifically, the data acquisition platform 104 may include optical imaging equipment 106 and a transcriptome analyzer device 108. The optical imaging equipment 106 may allow for the use of light (visible light or other light) to generate images of biological tissue slides. The transcriptome analyzer equipment 108 may allow for the generation of spatial transcriptome analysis data of biological tissue slides. The transcriptome analyzer equipment 108 may generate such data with one or more spatial resolutions, including near single-cell resolution, single-cell resolution, and / or subcellular resolution. The computing system 100 also includes one or more memory devices 110 (referred to as data storage 110), which are used to store imaging data indicating images of biological tissue slides. The imaging data may be stored in one or more files within a file system configured in the data storage 110. Each of the one or more files may be formatted according to a suitable format for image storage. In some cases, the image may be a histological stained image. Such images may be obtained using H&E (hematoxylin and eosin solution) staining. Embodiments of the present disclosure are not limited in this regard and may be applied to images obtained using other types of histological staining techniques. In other cases, the image may be an immunohistochemical staining (IHC) image. In still other cases, the image may be an in situ hybridization (ISH) image, including a fluorescence in situ hybridization (FISH) image. In yet other cases, the image may be a single molecule fluorescence in situ hybridization (smFISH) image. Thus, the data generated by the data acquisition platform 104 and stored in the data storage 110 may include various types of data. In some embodiments, single-cell data may include single-cell RNA sequencing data (scRNA-seq) or single-nucleus RNA sequencing data (snRNA-seq). In some embodiments, single-cell data may include single-cell ChIP. In some embodiments, single-cell data may include single-cell ATAC-seq. In some embodiments, single-cell data may include single-cell proteomics. In some embodiments, the spatial data may be coarse-grained. In some embodiments, the spatial data may also include STARmap and / or MERFISH. In some embodiments, the single-cell data is multimodal single-cell data. In some embodiments, the multimodal data is single-cell RNA-seq and chromatin accessibility data (SHARE-seq). In some embodiments, the multimodal data is single-cell RNA-seq and proteomics data (CITE-seq). In some embodiments, the multimodal data is single-cell RNA-seq and patch-clamp electrophysiological recording and morphological analysis of single neurons (Patch-seq).

[0048] The data storage device 110 may also store transcriptome analysis data, including a plurality of data sets corresponding to respective images of slides of biological tissue. Each of the plurality of data sets containing the transcriptome analysis data of a respective slide in the slide containing the biological tissue may be acquired simultaneously with the optical imaging of the respective slide. That is, a first data set containing the transcriptome analysis data of the first slide may be acquired simultaneously with the optical imaging of the first slide of the biological tissue; a second data set containing the transcriptome analysis data of the second slide may be acquired simultaneously with the optical imaging of the second slide of the biological tissue; and so on.

[0049] The computing system 100 further includes a computing device 120 that is functionally coupled to the data storage device 110 via a communication architecture 114 (e.g., a wireless network, a wired network, a wireless link, a wired link, a server device, a router device, a gateway device, a combination thereof, etc.). The computing device 120 may include computing resources (not depicted for clarity), which include, for example, one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), memory, disk space, incoming bandwidth and / or outgoing bandwidth, interfaces (such as I / O interfaces or APIs, or both); controller devices; power supplies; combinations of the foregoing; and / or similar resources.

[0050] The computing device 120 may include an ingestion module 130 that may access images of slides of biological tissue and transcriptome analysis data from the data storage device 110. Accessing the imaging data may include, for example, programmatically downloading the images and / or transcriptome analysis data from the data storage device 110. For example, the ingestion module 130 may download the images and / or transcriptome analysis data via an application programming interface (API). By executing one or more function calls of the API, the ingestion module 130 may receive an image file (or data constituting the file) corresponding to the image and may save the image file (or data constituting the file) in one or more non-volatile memory devices 170 (referred to as memory 170). As another example, the ingestion module 130 may download the images and / or transcriptome analysis data by executing a script to copy the image file from the file system within the data storage device 110 to the memory 170. The memory 170 may be integrated into the computing device 250.

[0051] Now turning to Figure 1AIn the workflow 101 shown, the ingestion module 130 may receive an image of a slide of biological tissue. This image may be referred to as a reference image 103 (or a fixed image), and may be used to establish a reference two-dimensional (2D) coordinate system. The positions of the pixels within the reference image 103 may be defined relative to this 2D coordinate system. The reference image 103 may also be referred to as the "fixed image". In addition, the ingestion module 130 may also receive an image of another slide of biological tissue. Such an image may be referred to as a shifted image 105 because the multiple slides of the biological tissue sections may be shifted relative to each other. The positions of the pixels within the shifted image 105 need not be defined in the reference 2D coordinate system. Thus, the pixels at a particular location within the shifted image 105 need not correspond to the pixels at the same location in the reference image 103. The shifted image 105 may also be referred to as the "moved image". Figure 1B Further examples of the reference image 103 and the shifted image 105 are shown.

[0052] The computing device 120 may align the shifted image 105 and the shifted spatial coordinates of the spots where the transcriptomes are drawn on the shifted slide relative to the 2D reference coordinate system. Before performing this alignment, the computing device 120 may operate on the reference image 103 and the shifted image 105 by applying a normalization process. The normalization process is applied to generate a normalized image representing a tissue mask image. To this end, the computing device 120 may include a normalization module 140 that may apply the normalization process.

[0053] As part of the normalization process, the normalization module 140 may determine whether to apply color normalization. Color normalization may be applied in cases where there is a significant color deviation from an expected color palette and / or color inconsistency across the reference image 103 and the shifted image 105. To make such a determination, the normalization module 140 may apply a color deviation criterion and / or a color inconsistency criterion to both the reference image 103 and the shifted image 105. In some cases, neither the color deviation criterion nor the color inconsistency criterion is met, and thus, the normalization module 140 may determine that color normalization will not be applied. In other cases, in response to meeting at least one of such criteria, the normalization module 140 may determine to apply color normalization to both the reference image 103 and the shifted image 105. Thus, the image component 142 included in the normalization module 140 may receive the reference image 103 and may apply color normalization to the reference image 103. For example, the reference image 103 may be received from the memory 170. In addition, the image component 142 may receive the shifted image 105 and may apply color normalization to the shifted image 105. The shifted image 105 may be received from the memory 170. Applying color normalization may include scaling each RGB color channel such that at each pixel of the input image (reference image or shifted image), the intensity value in each channel is in the interval [0,1].

[0054] Whether or not color normalization is applied, the image component 142 can further apply a normalization process to the reference image 103 and the shifted image 105. Thus, the image component 142 generates a normalized reference image and a normalized shifted image. To apply the normalization process, the image component 142 can receive an input image (e.g., the reference image 101 or the shifted image 103). The input image can be received by the memory 170. The image component 142 can then generate a tissue mask image of the input image by separating tissue foreground pixels and non-tissue background pixels. In the case where the input image is a histological image of biological tissue, the input image can be a red-green-blue (RGB) color image. The image component 142 can operate on the RGB color image to separate tissue foreground pixels and non-tissue background pixels. In some cases, it may be difficult to identify non-tissue background pixels using the RGB color image. In such cases, the image component 142 can transform the RGB color image into a grayscale image. Then, the image component 142 can use the grayscale image to determine a threshold intensity that allows the tissue foreground pixels to be distinguished from the non-tissue background pixels. Specifically, the image component 142 can identify pixels in the grayscale image that have an intensity less than the threshold intensity, and then can configure such pixels as non-tissue background pixels. The image component 142 can assign zero intensity to the non-tissue background pixels in the grayscale image; that is, the non-tissue background pixels can be configured as black pixels in the grayscale image. Thus, the image component 142 can identify the pixels in the RGB color image that correspond to the non-tissue background pixels in the grayscale image. The image component 142 can configure such identified pixels as non-tissue background pixels in the RGB color image. For example, after the non-tissue background pixels in the input image have been identified, the image component 142 can configure the non-tissue background pixels in the tissue mask image as black pixels. The tissue mask image with black pixels constitutes the normalized input image. Figure 1B An example of the normalized reference image 103A and the normalized shifted image 105A is shown.

[0055] The normalization module 140 can also optionally apply one or more shaping operations to the tissue mask image. The shaping operations can include cropping the tissue mask image to remove undesired background portions, and resizing the tissue mask image to achieve a defined resolution suitable for input into the alignment model described herein.

[0056] As mentioned, the transcriptome analysis data can be collected from the same glass slide used to obtain an image of a biological tissue (referred to as a tissue image). Thus, the transcriptome analysis data corresponds to the tissue image and can complement the information contained in the image. In some existing platforms, gene expression measurements can be performed on hundreds to tens of thousands of small tissue regions (referred to as spots). In some cases, a spot can cover an area spanning one cell, and thus the measurement can have single-cell spatial resolution. In other cases, a spot can cover an area spanning several to dozens of cells, and thus the measurement can have near single-cell spatial resolution. In ISH-based platforms, messenger RNA (mRNA) transcripts can be quantified at subcellular resolution. However, gene expression obtained from within a single or a few cells at each spot may be affected by low mRNA transcript detection and may be sensitive to non-spatial confounding factors such as batch effects and cell state changes.

[0057] To mitigate or even completely avoid the limitations imposed by batch effects and cell state changes, the normalization module 140 can transform the transcriptome analysis data into input data formatted as an image. To this end, the ingestion module 130 can receive spatially resolved transcriptome (ST) analysis data corresponding to a first glass slide of a biological tissue and a second glass slide of the biological tissue. The first glass slide can correspond to a reference image, and the second glass slide can correspond to a shifted image. The transcriptome analysis data defines a transcriptome map of corresponding spots within each of the first and second glass slides. The position of a spot in the first glass slide can be defined by the pixel position in the reference image that is below the center of the spot and is thus defined in a reference 2D coordinate system. The position of a spot in the second glass slide is defined by the pixel position in the shifted image that is at the center of the spot and is thus not necessarily defined in the reference 2D coordinate system. Thus, a spot at a particular position within the second glass slide does not have to correspond to a spot at the same position within the first glass slide. Figure 1C An example of a set of reference spot coordinates 107 and a set of shifted spot coordinates 109 is shown. As described herein, the computing device 120 can align the shifted spatial coordinates of spots on the shifted slide relative to the 2D reference coordinate system.

[0058] The ingestion module 130 can pass (or otherwise send) the transcriptome analysis data to the normalization module 140. A transcriptome component 146, which can be part of the normalization module 140, can combine the transcriptome maps of all spots within a particular glass slide into a count matrix, such as Figure 1AThe shown count matrices 111 (for reference image 103) and 113 (for shifted image 104). The columns of the count matrices 111, 113 correspond to respective spot identifiers (IDs), or in some cases to cell IDs. The rows of the count matrices 111, 113 correspond to respective genes. The transcriptome component 146 can normalize the count matrices 111, 113 so that each column has an equal total count. Then, the transcriptome component 146 can perform a natural logarithm transformation on the count matrices 111, 113. The transcriptome component 146 can also center and scale the count matrices 111, 113 to have values in the range of 0 to 1. Subsequently, the transcriptome component 146 can determine the top N variable genes across the spots within a particular slide. The parameter N is configurable. In one example, N = 2000.

[0059] The transcriptome component 146 can integrate the transcriptome data corresponding to the first slide and the second slide. The transcriptome component 146 can scale the integrated data and can achieve dimensionality reduction by applying principal component analysis (PCA). The transcriptome component 146 can perform mutual nearest neighbor-based clustering based on the top M principal components. The resolution parameter is configured to identify the major clusters. The parameter M is configurable. In one example, M = 30.

[0060] After clusters (such as Figure 1A the shown cluster 115) have been identified, the transcriptome component 146 can generate a label map based on the transcriptome analysis data. The label map spans the same defined region as the input image (e.g., reference image 103 or shifted image 105) of the biological tissue associated with the transcriptome analysis data. The label map can include a first label associated with a plurality of first pixels within the input image, a second label associated with a plurality of second pixels within the input image, and so on. To generate the label map, the transcriptome component 146 can assign a cluster label to the pixel located at the centroid of each spot (or each cell in single-cell resolution measurements). In some cases, the centroid pixel can be used as a generation point to divide the pixels of the input image into a Voronoi diagram. The transcriptome component 146 can assign a defined cluster label, such as cluster 1, cluster 2, etc., to the pixels within each partition polygon in the Voronoi diagram, as shown by Figure 1A cluster 115 therein, where the defined cluster label is the same as the cluster label assigned to the centroid pixel of the generation point defining the partition polygon. This label assignment process can be referred to as the inflation of the labels.

[0061] The transcriptome component 146 can generate a contour map based on the label map, which reflects the transcriptome-derived spatial organization of tissue spots (or cells in single-cell resolution measurements). Figure 1AAn example contour map 111A of the reference image 103 and an example contour map 113A of the shifted image 105 are shown - each contour map having three boundaries (e.g., based on clustering 115). Thus far, the boundary between any two regions with different cluster labels can be used to define the contour map. In fact, the first contour in the contour map defines a boundary that separates a subset of a plurality of first pixels having a first cluster label from a subset of a plurality of second pixels having a second cluster label. In some cases, high-complexity regions in the contour map (e.g., 111A and / or 113A) can be merged to avoid overfitting of the model.

[0062] The transcriptome component 146 can overlay the contour map (e.g., 111A and / or 113A) on the normalized input image. For example, as Figure 1A shown, the contour map 111A can be overlaid on the normalized reference image 111C (e.g., the normalized reference image 103A), and the contour map 113A can be overlaid on the normalized shifted image 113C (e.g., the normalized shifted image 105A). The contours in the contour map (e.g., 111A and / or 113A) can be overlaid with the normalized input image at the pixels below the contour. To this end, the transcriptome component 146 can identify specific pixels in the normalized input image corresponding to each contour. The transcriptome component 146 can then update the normalized input image by modifying the corresponding values of the specific pixels. The transcriptome component 146 can determine each value in the corresponding values as a linear combination of a first value from the normalized input image and a second value based on the transcriptome analysis data. The ratio of the second value to the first value in the linear combination can be, for example, 0.3:0.7.

[0063] Transforming the transcriptome data into a contour map and incorporating the contour map into the tissue image can allow minimizing the impact of noise and batch effects on the alignment of a pair of biological tissue images. The contour map derived from the transcriptome data as described herein provides orthogonal complementary information to the tissue image, especially in cases where the tissue image is homogeneous and thus lacks information about the spatial organization / orientation of the slide that is the source of the image. Nevertheless, the normalization of the reference image and the shifted image to overlay the contour map based on the transcriptome data is optional. In fact, in some scenarios, such as when the tissue image contains sufficient regional texture details, the transcriptome data has low quality, etc., it may be undesirable to utilize the transcriptome data.

[0064] The normalization module 140 can pass (or otherwise send) the normalized reference image 111C and the normalized shifted image 113C to the transformation module 150. As Figure 1As shown, transformation module 150 may be included in computing device 120. Transformation module 150 may perform two types of alignment of the normalized shifted image 113C relative to the normalized reference image 111C. To this end, an affine registration component 154, which is part of transformation module 150, may perform a first registration of the normalized shifted image 113C relative to the normalized reference image 111C. The first registration may provide a rough alignment of the normalized shifted image 113C relative to the 2D reference coordinate system associated with reference image 103. Performing the first registration may include determining an affine transformation that may exist between the normalized shifted image 113C and the normalized reference image 111C. Determining the affine transformation includes determining a group of parameters that define the transformation. As a result of performing the first registration, affine registration component 154 may determine a first deformation vector field associated with the affine alignment based on the affine transformation. Such a first deformation vector field may define the new positions to which the corresponding pixels of the normalized shifted image 113C are to be moved so that the normalized shifted image 113C is at least partially aligned with the normalized reference image 111C. Transformation module 150 may save the first deformation vector field in memory 170 (shown in Figure 1A as deformation vector field 151 for speckle alignment).

[0065] As part of applying the first registration, alignment module 160 may apply the first deformation vector field to the normalized shifted image 113C. Accordingly, alignment module 160 may generate a second shifted image (not shown). Alignment module 160 may pass (or otherwise send) the second shifted image to transformation module 150. Transformation module 150 may receive the second shifted image and then may supply the normalized reference image 111C and the second shifted image to dense registration component 156. Dense registration component 156 may perform a second registration of the second shifted image relative to reference image 103 by applying alignment model 158 to the normalized reference image 111C and the second shifted image. Applying the alignment model in this manner may result in a second deformation vector field representing the registration transformation between the reference image and the second shifted image. Dense registration component 156 may be configured (e.g., programmed and constructed) to include alignment model 158. As an alternative, in some embodiments, dense registration component 156 may load or otherwise obtain alignment model 158 from memory 170. To this end, in such embodiments, memory 170 may save alignment model 158.

[0066] As part of applying alignment model 158, dense registration component 156 may determine a corresponding inverse deformation vector field by integrating the negative gradient (also referred to as negative flow) of the deformation vector field. Alignment module 160 may then apply the inverse deformation vector field to the second shifted image, thereby producing an aligned normalized image.

[0067] The alignment model 158 can be a machine learning model that includes a neural network having one or more first type of first layers and one or more second type of second layers. The one or more first layers can be configured to extract features from the normalized reference image and the second shifted image. The one or more second layers can be configured to output a deformation vector field and an inverse deformation vector field corresponding to the deformation vector field. In some cases, as Figure 2 shown, the machine learning model can include a convolutional neural network (CNN) 200 having: an ingestion module 210; an encoder-decoder block 220 including an encoder module for feature extraction, a single bottleneck layer representing the latent space, and a decoder module; and a field construction module 230 configured to output a deformation vector field and an inverse deformation vector field. The ingestion module allows the imaging data defining the reference image 204 and the imaging data defining the shifted image 206 to be received concurrently. Both the reference image and the shifted image can be normalized images according to the aspects described herein. For illustrative purposes only, in the CNN 200, the ingestion module 210 is implemented in a Siamese input processor, and the encoder-decoder module 220 is implemented in a U-Net backbone. Additionally, the CNN 200 is shown configured for images having a size of 256×256 pixels. Of course, the present disclosure is not limited in this regard, and other image sizes are conceivable. The encoder module includes four levels of downsampling (or shrinking), and the decoder module includes four levels of upsampling (or expanding). Each level also includes a skip connection from the encoder module to the decoder module. The last level of the decoder module is connected to the field construction module to output a deformation field, thereby aligning the input shifted image or a set of shifted coordinates of the spot on the shifted slide where the transcriptomic map is obtained with the input reference. Note that the present disclosure is not limited to the U-Net backbone. In fact, other CNNs can also form part of the alignment model, where each of these CNNs includes a module (e.g., a vision transformer) that can perform feature extraction from the imaging data and map the extracted features to a deformation field and a corresponding inverse deformation field.

[0068] For example, as a pre-training step or process, components of the alignment model 158, such as the encoder module of the encoder-decoder block 220, may perform feature extraction from the imaging data. In addition to using the transcriptomic atlas of the input reference, or when the transcriptomic atlas of the input reference is not available, the encoder module may use self-supervised learning (e.g., unsupervised learning) to extract features from the input reference (e.g., raw histological image) and map these extracted features to the deformation field and the corresponding inverse deformation field as described herein. The encoder module may use this self-supervised pre-training to enhance the existing feature extraction capabilities / functions of the encoders described herein. For example, the encoder module may use representation learning, masked image modeling, combinations thereof, etc. to autonomously identify and interpret patterns within the input reference by predicting one or more features using other features (e.g., associated with another part) derived from the one or more features (e.g., associated with the first part). The encoder module using self-supervised learning can improve the alignment performance with higher accuracy. In addition, the encoder module using self-supervised learning for feature extraction combined with subsequent supervised fine-tuning of the synthetic data as further described herein can also improve the model accuracy and precision.

[0069] The alignment model 158 may be configured to receive an image having a defined size (e.g., N p ×N p pixels). In one instance, N p = 256. However, the ML-based alignment described herein is not limited in this regard. In fact, in some embodiments, the computing device 120 may align images larger than the defined size. To this end, as Figure 3 shown, the transformation module 150 may partition the input image into parts having the defined size. To this end, the transformation module 150 may include a tiling component 310 that may generate tiles of the input image having a size equal to M p ×M p (where M p > N p ) prior to registration of the input image.

[0070] More specifically, in the case where each of the reference image and the shifted image has a size of M p ×M p , the tiling component 310 may generate tiles of the normalized reference image. Each tile image of the tiles of the normalized reference image has the same size that matches the defined size N p ×N p . The tile images may be evenly spaced along the width and height of the normalized reference image. As Figure 4AAs shown, each tile image of a block of the normalized reference image spatially partially overlaps with each other tile image adjacent to the tile image. In some cases, there may be at least two adjacent tile images and at most four adjacent tile images. Figure 4B shows an Figure 4A example of an image tile of the reference image (or fixed image) shown in

[0071] Instead of operating on the shifted input image, the mosaicking component 310 can generate blocks of a second shifted image. The blocks of the second shifted image can be generated in the same manner as the blocks of the reference image. Thus, the blocks of the second shifted image have the same structure as the blocks of the reference image. Therefore, there is the same total number of tile images on the tiled second shifted image and the normalized reference image. Each tile image in the blocks of the second shifted image has one and only one matching tile image in the blocks of the normalized reference image, and the tile images respectively cover the same spatial region in the second shifted image and the normalized reference image.

[0072] The tile images of the blocks of the normalized reference image and the tile images of the blocks of the second shifted image can be supplied to the alignment model 158. The tile images can be supplied in sequence, while supplying a pair of tile images, where the pair of tile images consists of a tile image of the normalized reference image and a tile image of the blocks of the second shifted image that matches the tile image from the normalized reference image in terms of placement and size. That is, the two tile images in the pair cover the same spatial region in their respective complete images.

[0073] The transformation module 150 can apply the alignment model 158 to each tile image of the blocks of the normalized reference image, and the tile images of the blocks of the second shifted image that match the tile images in the normalized reference image as described above via the dense registration component 156. Thus, applying the alignment model 158 can generate N T tile deformation vector fields, where N T represents the total number of tiles of the blocks of the normalized reference image, which is equal to the total number of tiles of the blocks of the second shifted image. Figure 5A shows an example of four tile deformation vector fields.

[0074] The alignment module 160 can concatenate N T tile deformation vector fields to form a deformation vector field corresponding to the shifted image with a size of M p × M p . Each tile deformation field among the N T tile deformation fields and each adjacent tile deformation field among its adjacent tile deformation fields have a common definition region. Thus, the alignment module 160 can concatenate N in the following mannerT Patch deformation vector field: For each pixel within a common defined region, determine a weighted average of the patch deformation vector fields that overlap at that pixel, and for each pixel within the common defined region, assign that weighted average to the deformation vector field. In some cases, the numerical weight applied to a patch deformation field at a particular pixel is inversely proportional to the distance between the center of that patch deformation field and that particular pixel. Figure 5B Shows an example of a deformation vector field resulting from connecting multiple patch deformation vector fields according to aspects described herein.

[0075] Alignment model 158 can be trained in a tissue-agnostic manner using synthetic training data. In this way, alignment model 158 may be applicable to any tissue type and / or image acquisition protocol - such as various histological staining techniques, non-histological fluorescence-based IHC imaging, and ISH imaging. Additionally, using synthetic training data can address the problem of limited training data availability common in deep neural network training. In contrast to the prior art, the image synthesis methods described herein are designed for the tissue alignment task to be solved by embodiments of the present disclosure.

[0076] Each instance of the training data record contains a synthetic image quadruple: (a) a color reference image; (b) a first segmentation mask image associated with the color reference image; (c) a color shifted image to be aligned with the color reference image; and (d) a second segmentation mask image associated with the color shifted image. In the present disclosure, the segmentation mask image may be referred to as a label map. Pixels in the segmentation mask image can be classified into a finite number of different classes, where each pixel is assigned a specific class label. These classes can be abstract classes regarding the training data, but may reflect spatially distinct regions of the tissue microenvironment and / or cellular composition in the context of a tissue slide.

[0077] Figure 6 Shows an example computing system 600 for training an alignment model (such as alignment model 158) using synthetic images. Computing system 600 includes an image generator module 610 that can generate multiple synthetic image quadruples to create training data for training the alignment model. Image generator module 610 can generate multiple training images, where each pair of training images includes a training reference image and a training shifted image. The multiple pairs of training images can be generated based on multiple label maps.

[0078] More specifically, for a particular synthetic image quadruple, the image generator module 610 may include a label map generator component 614 (referred to as the label map generator 614), which may configure labels for corresponding pixels across a region of a defined size. Labels may be configured for multiple layers corresponding to the region of the defined size. Specifically, the labels include multiple sets of labels for corresponding layers among the multiple layers. That is, for N L layers, the first set of labels corresponds to the first layer, the second set of labels corresponds to the second layer, and so on until the N L th set of labels corresponding to the N L th layer. In one instance, N L = 5. Thus, the generator module 310 may assign N L labels to each pixel within the region of the defined size, where each of the N L labels corresponds to a respective layer. Each layer may be referred to as a channel.

[0079] The labels within a layer may be configured randomly. To this end, the label map generator 614 may configure multiple arbitrary seeded simplex noise distributions within the respective layer. Thus, for a certain layer, at each pixel within the region of the defined size, the label may have a numerical weight associated with it. The label map generator 614 may determine the numerical weight as a random value according to the simplex noise distribution at that pixel for that layer. Thus, the label map generator 614 may use such simplex noise distributions to determine the respective numerical weights of the multiple defined labels at a particular pixel within the region. Then, the label map generator 614 may assign a first label to the particular pixel, where the first label corresponds to the first numerical weight having the largest magnitude among the respective numerical weights.

[0080] Based on the configured labels, the label map generator 614 may generate a base label map spanning the region of the defined size. Then, the label map generator 614 may generate a reference label map by distorting the base label map using a simplex noise field as a distortion field. This distortion may be referred to as the first simplex noise deformation. Additionally, the label map generator 614 may also generate a shifted label map by distorting the base label map using another simplex noise field as a distortion field. This distortion may be referred to as the second simplex noise deformation.

[0081] The image generator module 610 may include a color image generator component 618 (referred to as the color image generator 618), which may generate a specific training reference image based on a reference label map. To this end, for each label in the first label map, the color image generator 618 may randomly select a color (e.g., an RGB color), and then may configure the pixel group corresponding to the label to have the selected color. Thus, the color image generator 618 generates a color image. In addition, the color image generator 618 may operate on the color image to generate a specific training reference image. Operating on the color image may include: blurring the color image to generate a blurred image; and applying a bias intensity field to the blurred image. The blur applied to the color image may be a Gaussian blur. The bias intensity field spans a region of a defined size.

[0082] The color image generator 618 may also generate a specific training shifted image based on a shifted label map. To this end, for each label in the second label map, the color image generator 618 may randomly select a color (e.g., an RGB color), and then may configure the pixel group corresponding to the label to have the selected color. Thus, the color image generator 618 generates a color image. In addition, the color image generator 618 may operate on the color image to generate a specific training image. As described herein, operating on the color image may include: blurring the color image to generate a blurred image; and applying a bias intensity field to the blurred image. The blur applied to the color image may be a Gaussian blur. The bias intensity field spans a region of a defined size.

[0083] The image generator module 610 may configure the reference label map, the shifted label map, the specific training reference image, and the specific training shifted image to be related to a specific instance of the training data record. The image generator module 610 may save the first label map, the second label map, the specific training reference image, and the specific training shifted image as part of a composite image 634 in one or more memory devices 630 (referred to as the image storage device 630).

[0084] Figure 6A An example process flow 650 for generating a reference label map and a shifted label map and for generating a composite training reference image and a composite training shifted image (e.g., via the image generator module 610) according to aspects described herein is schematically depicted. Noise distribution 651 may include an N-layer 2-D simplex noise with a height of H and a width of W. Noise distribution Each layer n of the noise distribution i may correspond to a class label. As Figure 6A shown, a set of simplex noise distributions having a shape (H, W) corresponding to the noise distribution 651 Applied to the noise distribution For each layer n of 651 i to form a distorted noise distribution 653. Next, the distorted noise distribution 653 can be condensed along dimension N to form a 2-D base label map l655 of shape (H, W). Each pixel of the base label map l655 can be assigned a class label c i (not shown), where layer n i may have the highest intensity at the position corresponding to the pixel. The base label map l655 can be further distorted by separate simplex noise and to create a reference label map l r 657 and a shifted label map l m 659 respectively. Based on the reference label map l r 657 and the shifted label map l m 659, a final reference image 661 (e.g., a synthetic training reference image) and a final shifted image 663 (e.g., a synthetic training shifted image) can be generated. For example, the final reference image 661 and the final shifted image 663 can be generated by assigning an RGB color to each of the class labels associated with the base label map l655. The final reference image 661, the final shifted image 663, the reference label map l r 657 and the shifted label map l m 659 can represent (e.g., as a group) a sample in the training dataset. That is, the final reference image 661, the final shifted image 663, the reference label map l r 657 and the shifted label map l m 659 can represent one of the multiple synthetic image quadruples as described herein.

[0085] The computing system 600 can include a training module 620 that can train an alignment model based on multiple label maps and multiple pairs of training images (e.g., generated via process flow 650) for registering an evaluation shifted image of biological tissue relative to an evaluation reference image of biological tissue. As described herein, the alignment model can be a machine learning model that includes a convolutional neural network (e.g., CNN 200( Figure 2)) The convolutional neural network has: an encoder module for feature extraction; a single bottleneck layer representing the latent space; a decoder module; and a field composition module configured to output a deformation vector field and an inverse deformation vector field corresponding to the deformation vector field. The field composition module may also be referred to as a deformation field formation module. In some instances, as described herein, as a pre-training step or process, the encoder module for feature extraction may perform feature extraction from imaging data (such as an evaluation reference image of biological tissue). In addition to using the transcriptomic atlas of the evaluation reference image, or when the transcriptomic atlas of the evaluation reference image is not available, the encoder module may use self-supervised learning (e.g., unsupervised learning) to autonomously identify and interpret patterns within the evaluation reference image by predicting one or more features using other features (e.g., associated with another part) derived from one or more features (e.g., associated with the first part).

[0086] To train the alignment model, the training module 620 may iteratively determine a solution to the optimization problem regarding the loss function based on a similarity metric of a pair of label maps and the deformation vector field associated with the training reference image and the training shifted image. Such a solution defines the trained machine learning alignment model. More specifically, the loss function may be defined as

[0087]

[0088] where f, s f respectively represent the training reference image and the reference label map associated with this image. Additionally, m and s m respectively represent the shifted image and the shifted label map associated with this image after applying the inferred deformation vector field output by the alignment model . The parameter λ reg is a regularization factor, and is the gradient of the inferred deformation vector field (that is, the magnitude of change)

[0089] In equation (1), the regularization term prevents sudden large deformations. The Dice score Dice(s m , s f ) is a similarity metric that assesses the consistency of the per-pixel class labels between the reference label map and the shifted label map, rather than the per-pixel color and intensity consistency between the training reference image and the training shifted image. The loss function L(m, f) applies to the fact that the tissue slides to be aligned are not expected to be exactly the same in terms of fine-grained details (such as the positions of individual cells / nuclei). Instead, the matching is for cell regions.

[0090] The training module 620 may store the trained alignment model in an alignment model library 644 within one or more memory devices 640 (referred to as the model storage device 640). The training module 620 may also configure an interface, such as an API, to allow access to the stored trained alignment model via function calls.

[0091] In an embodiment of the present disclosure, a trained alignment model (such as the alignment model 158) may infer a conservative deformation in each inference execution (e.g., applying the trained alignment model to a pair of evaluation images). Thus, a computing device (e.g., the computing device 120( Figure 1 )) or, in some cases, a computing device system, may apply the trained alignment model to the pair of evaluation images multiple times until the shift coordinates are stable or a defined number of iterations (e.g., 5 iterations) is reached. The defined number of iterations may be configurable based on user-specified input. The shift coordinates may be considered stable when the mean square error between the current coordinates in the nth iteration and the coordinates in the previous (n - 1)th iteration is less than a defined percentage (e.g., 1%) of the mean square error between the (n - 1)th iteration and the (n - 2)th iteration.

[0092] In an embodiment of the present disclosure, more than two slides may be aligned in a "template-based" mode or a "template-free" mode. The template-based mode includes configuring one of the slides as a fixed reference and aligning the remaining slides relative to the fixed reference. The operations for aligning a pair of slides described herein may be applied to each of the non-reference slides to generate a set of aligned tissue slides.

[0093] The "template-free" mode includes scaling and centering each slide such that the detected spot coordinates on each slide are within the same range along the x-axis and y-axis. For example, assuming a spatial transcriptomics slide size of 512×512, the minimum (x, y) coordinates of the spots in each slide may be at (35, 35), and the maximum (x, y) coordinates of the spots in each slide may not be greater than (477, 477). The "template-free" mode also includes, for each slide s s among a total of N i slides, performing pairwise alignment by applying the trained alignment model to s i using the remaining (N s -1) slides s j (j≠i) as a fixed reference. The resulting (N - 1) sets of aligned coordinates may be averaged with the initial coordinates of s i (assuming it is aligned with itself and thus has no change) and output as the final aligned coordinates of slide s i .

[0094] FIG. 7 to Figure 10Shows the alignment performance in various scenarios with different slide types and slide arrangements according to embodiments of the present disclosure. In some of the execution results, for ease of naming, embodiments of the present disclosure have been generally referred to as "ML aligners". More specifically, Figure 7A and Figure 7B Show examples of the alignment performance of small slides according to embodiments of the present disclosure. Such small slides each have 115 spots. The alignment performance is measured by the mean squared error (MSE). For each simplex noise intensity of 5, 10, and 15, the performance of the ML aligner is better than that of the existing slide alignment techniques. Without wishing to be bound by an interpretation, it should be noted that although GPSA produces a smaller MSE than the MSE produced by the ML aligner for a simplex noise intensity of 20, in this case, the improved performance of GPSA seems to be an artificial factor. More specifically, the number of spots in the small slides seems to mitigate the limited sensitivity of GPSA in distinguishing spots with similar transcriptome profiles for a simplex noise intensity of 20. Figure 8A and Figure 8B Show examples of the alignment performance of full-space slides according to embodiments of the present disclosure. Such full-space slides have approximately 3000 spots. For all the shown simplex noise intensities, the performance of the ML aligner is better than that of the existing slide alignment techniques. Referring again to the simplex noise intensity of 20, it should be noted that the limitation of GPSA in distinguishing spots with similar transcriptome profiles relative to the ML aligner can result in lower performance. Figure 9 Show examples of the alignment performance for a human lymph node full-space slide according to embodiments of the present disclosure. Regardless of the amount of distortion, the performance of the ML aligner is better than that of the existing slide alignment techniques, as shown in bar chart 910, bar chart 920, and bar chart 930. Figure 10 Show examples of the alignment performance for more than two slides. The slides are unordered and correspond to repeated samples. As shown in bar chart 1010, the performance of the ML aligner is better than that of the existing slide alignment techniques.

[0095] Figure 11 Show the application of the alignment model according to embodiments of the present disclosure to the alignment of two mouse brain images. These images correspond to consecutive sections / slides of a mouse brain.

[0096] In view of the aspects described herein, reference can be made, for example, to Figures 12 to 17The flowcharts herein are useful for better understanding example methods that may be implemented in accordance with the present disclosure. For ease of explanation, the example methods disclosed herein are presented and described as a series of boxes (e.g., where each box represents an action or operation in the method). However, the example methods are not limited by the order of the boxes and the associated actions or operations, because some boxes may occur in a different order and / or concurrently with other boxes shown and described herein. Further, not all of the boxes and associated actions shown may be required to implement an example method in accordance with one or more aspects of the present disclosure. Two or more of the example methods (and any other methods disclosed herein) may be combined with each other to implement. Note that the example methods (and any other methods disclosed herein) may alternatively be represented as a series of interrelated states or events, such as in a state diagram.

[0097] The method according to the present disclosure may be stored on an article of manufacture or a computer-readable non-transitory storage medium to allow or facilitate the transfer and conveyance of such method to a computing device or computing device system (such as a desktop computer; a laptop computer; a blade server; or the like) for execution by one or more processors of the computing device and thus implementation, or for storage in one or more memory devices thereof or functionally coupled thereto. In one aspect, one or more processors (such as a processor that implements (e.g., executes) one or more of the disclosed methods) may be employed to execute program code (e.g., processor-executable instructions) stored in a memory device or any computer-readable or machine-readable medium to implement one or more of the disclosed methods. Such program code may provide a computer-executable or machine-executable framework to implement the methods described herein.

[0098] Figure 12 is a flowchart of an example method 1200 for aligning the spatial coordinates of spots within an image and a biological tissue transcriptome map to a common coordinate system in accordance with one or more embodiments of the present disclosure. As described herein, the image may be a histological stained image, an IHC image, or an ISH image (such as a FISH image). A computing device or computing device system may implement the example method 1200 in whole or in part. To that end, each computing device in the computing device includes computing resources that may implement at least one of the boxes included in the example method 1200. Computing resources include, for example, a CPU, a GPU, a tensor processing unit (TPU), a memory, disk space, incoming bandwidth and / or outgoing bandwidth, an interface (such as an I / O interface or an API, or both); a controller device; a power supply; a combination of the foregoing; and / or similar resources. In one example, one or more of the computing systems may include a programming interface; an operating system; software for configuring and / or controlling a virtualized environment; firmware; and similar resources. The computing device system may be referred to as a computing system.

[0099] In some cases, the computing device implementing the example method 1200 may host the ingestion module 130, the normalization module 140, the transformation module 150 (including the tiling component 310 in some cases), and the alignment module 160, as well as other software components / modules. For example, the computing device may implement the example method 1200 by executing one or more instances of the ingestion module 130, the normalization module 140, the transformation module 150 (including the tiling component 310 in some cases), and the alignment module 160. Thus, in response to execution, the ingestion module 130, the normalization module 140, the transformation module 150 (including the tiling component 310 in some cases), and the alignment module 160 may perform operations corresponding to the blocks of the example method 1200, either individually or in combination.

[0100] At block 1210, the computing device may receive a reference image of the biological tissue. The reference image may be obtained via optical imaging of a reference slide of the biological tissue. At block 1215, the computing device may receive a shifted image of the biological tissue. The shifted image may also be obtained via optical imaging of a shifted slide of the biological tissue. For example, the reference slide and the shifted slide may be obtained by successively sectioning an organ of a subject. As described herein, the position of pixels within the reference image may be defined relative to a reference 2D coordinate system, and it is not necessary to define the position of pixels within a particular shifted image within the reference 2D coordinate system. Thus, pixels at a particular position within the shifted image need not correspond to pixels at the same position within the reference image.

[0101] At block 1220, the computing device may apply a normalization process to a particular reference image, thereby producing a normalized reference image. At block 1225, the computing device may apply a normalization process to the shifted image, thereby producing a normalized shifted image. The normalization process may include various operations involving the tissue image and optionally, transcriptomic analysis data collected on the same slide used to obtain the tissue image. To apply the normalization process to the reference image and the shifted image, the computing device may implement the example method 1300 described below ( Figure 13 ) and optionally, the example method 1400 ( Figure 14 ). When implementing these example methods, the reference image and the shifted image are individually used as the input tissue images in the respective applications of the normalization process.

[0102] At block 1230, the computing device may perform a first registration of the normalized shifted image relative to the normalized particular reference image. As described herein, performing the first registration includes determining a rough transformation between the normalized shifted image and the normalized reference image. Performing such a first registration produces a group of parameters defining the rough transformation and also produces a second shifted image. For example, the rough transformation may be an affine transformation.

[0103] At block 1235, the computing device may supply a reference image to the alignment model. At block 1240, the computing device may supply a second specific shifted image to the alignment model. The reference image and the second shifted image may be supplied to the alignment model concurrently. The alignment model may be an ML model that has been trained using synthetic images. Thus, the alignment is tissue-independent. The ML model includes a neural network having one or more layers configured to extract features from the normalized reference image and the second shifted image, and one or more second layers configured to output a deformation vector field and an inverse deformation vector field corresponding to the deformation vector field. In some cases, the ML model includes a CNN (e.g., CNN 200( Figure 2 ))). The deformation vector field represents the registration transformation between the reference image and the second shifted image. In one example, the alignment model is alignment model 158( Figure 1 ).

[0104] At block 1245, the computing device may perform a second registration of the second shifted image relative to the reference image by applying the alignment model to the reference image and the second displacement image, where the application produces a deformation vector field representing the registration transformation between the reference image and the second shifted image. As part of applying the alignment model, the computing device (e.g., via the dense registration component 156( Figure 1 )) may also determine the corresponding inverse deformation vector field by integrating the negative gradient (also known as the negative flow) of the deformation vector field.

[0105] At block 1250, the computing device may receive the reference space coordinates of the spots within the first transcriptomic map of the biological tissue. The first transcriptomic map corresponds to the reference image. In fact, the first transcriptomic map may be obtained from a reference slide (e.g., via the transcriptomic analyzer apparatus 108( Figure 1 ). Each of the spots includes one or more cells.

[0106] At block 1255, the computing device may receive the shifted space coordinates of the spots within the second transcriptomic map of the biological tissue. The second transcriptomic map corresponds to the shifted image. In fact, the second transcriptomic map may be obtained from a shifted slide (e.g., via the transcriptomic analyzer apparatus 108( Figure 1 ). As mentioned, each of the spots includes one or more cells.

[0107] At block 1260, the computing device may perform a first registration of the shifted space coordinates relative to the reference space coordinates based on a rough transformation, thereby producing second shifted space coordinates of the spots within the second transcriptomic map. As part of the first registration, the computing device may apply a rough transformation (e.g., an affine transformation) to the shifted space coordinates to move the spots to the second shifted space coordinates.

[0108] At block 1265, the computing device may perform a second registration of the second shifted spatial coordinates of the spots within the second transcriptomic map based on the registration transformation. As part of the second registration, the computing device may apply the registration transformation to the second shifted spatial coordinates to further move the spots within the second transcriptomic map to a terminal position defined relative to a reference 2D coordinate system.

[0109] By implementing example method 1200, the computing device may align the spots in the reference slide and the shifted slide. To do so, as described above, the computing device relies on the rough transformation and the registration transformation determined by aligning the reference image and the shifted image according to the aspects described herein.

[0110] Figure 13 is a flowchart of an example method 1300 for normalizing a biological tissue image according to one or more embodiments of the present disclosure. Similarly, the image may be a histological stained image, an IHC image, or an ISH image (such as a FISH image). The computing device implementing example method 1200 ( Figure 12 ) or in some cases a computing system may implement example method 1300.

[0111] At block 1310, the computing device may receive an image of a biological tissue. The image may be a histological stained image. Such an image may be obtained using one of various types of histological staining techniques. Thus, the image may be an RGB color image. In some cases, the image of the biological tissue is a reference image of the biological tissue. In other cases, the image of the biological tissue is a shifted image of the biological tissue.

[0112] At block 1320, the computing device may determine whether to apply color normalization to the input image. To do so, the computing device may apply a color deviation criterion and / or a color inconsistency criterion to the input image. In response to an affirmative determination ("yes" branch), the computing device may normalize the color of the image at block 1330 and then may direct the flow of example method 1300 to block 1340. Normalizing the color of the input image may include scaling each RGB color channel such that at each pixel of the image, the intensity value is within the interval [0,1]. In response to a negative determination ("no" branch), the computing device may direct the flow of the example method to block 1340.

[0113] At block 1340, the computing device may generate a tissue mask image for the input image. The tissue mask image may be generated by separating tissue foreground pixels and non-tissue background pixels as described herein.

[0114] At block 1350, the computing device may configure non-tissue background pixels as black pixels. The tissue mask image with black pixels constitutes the normalized input image. Example method 1300 may optionally include block 1360, where the computing device may apply one or more shaping operations to the normalized input image. As described herein, the shaping operations may include cropping and resizing. The normalized input image may be cropped to remove undesired background portions and may be resized such that the normalized input image has a size suitable for input into the ML alignment model mentioned in example method 1200.

[0115] Figure 14 is a flowchart of an example method 1400 for normalizing an image of biological tissue using transcriptome analysis data according to one or more embodiments of the present disclosure. As described herein, the image may be a histological stain image, an IHC image, or an ISH image (such as a FISH image). The computing device implementing example method 1200 ( Figure 12 ) or in some cases a computing system may implement example method 1400.

[0116] At block 1410, the computing device may receive transcriptome analysis data corresponding to an image of biological tissue. The transcriptome analysis data may be obtained from a glass slide of the biological tissue from which the image of the tissue was obtained.

[0117] At block 1420, the computing device may generate a label map based on the transcriptome analysis data, the label map including a first label associated with a plurality of first pixels and a second label associated with a plurality of second pixels.

[0118] At block 1430, the computing device may generate a contour map based on the label map. As described herein, the contour map reflects the transcriptome-derived spatial organization of tissue spots (or cells in single-cell resolution measurements). Thus, a first contour in the contour map defines a boundary that separates a subset of the plurality of first pixels having a first cluster label from a subset of the plurality of second pixels having a second cluster label.

[0119] At block 1440, the computing device may identify a specific set of pixels within the normalized version of the image corresponding to the respective contours in the contour map. Thus, the computing device may identify a first set of pixels corresponding to the first contour in the contour map and may also identify a second set of pixels corresponding to the second contour in the contour map.

[0120] Based on the identified specific set of pixels, the computing device can overlay the contour map on the normalized version of the image at block 1450. To do so, the computing device can update the normalized version of the image by modifying the respective values of the pixels that make up the specific set of pixels. Each of the respective values can be generated by a linear combination of a first value in the normalized version of the image and a second value based on the transcriptomic analysis data.

[0121] As mentioned, converting transcriptomic data into a contour map and incorporating the contour map into an image of biological tissue can allow minimizing the effects of noise and batch effects on the alignment of a pair of images of biological tissue.

[0122] Figure 15 is a flowchart of an example method 1500 for training an ML alignment model (also referred to as an alignment model) according to one or more embodiments of the present disclosure. The computing device or computing device system can implement the example method 1500 in whole or in part. To that end, each computing device in the computing device includes computing resources that can implement at least one of the blocks included in the example method 1500. Computing resources include, for example, a CPU, GPU, TPU, memory, disk space, incoming bandwidth and / or outgoing bandwidth, interfaces (such as I / O interfaces or APIs, or both); controller devices; power supplies; combinations of the foregoing; and / or similar resources. In one example, one or more of the computing devices can include a programming interface; an operating system; software for configuring and / or controlling a virtualization environment; firmware; and similar resources. The computing device system can be referred to as a computing system.

[0123] In some cases, the computing system implementing the example method 1500 can host an image generator module 610 ( Figure 6 ) and a training module 620 ( Figure 6 ) and other software components / modules. For example, the computing system can implement the example method 1500 by executing one or more instances of the image generator module 610 and the training module 620 or a combination thereof. Thus, in response to execution, the image generator module 610 and the training module 620 can perform operations corresponding to the blocks of the example method 1500, either individually or in combination.

[0124] At block 1510, the computing system can generate multiple pairs of training images based on multiple pairs of training label maps. Each pair of the multiple pairs of training images includes a training reference image and a training shifted image. Each pair of the multiple pairs of training label maps includes a training reference label map and a training shifted label map. The computing system can generate the multiple pairs of training images by implementing Figure 16 the example method 1600 shown and described below.

[0125] At block 1520, a computing system may train an ML alignment model based on multiple pairs of label maps and multiple pairs of training images for registering an evaluation shifted image of a biological tissue relative to an evaluation reference image of the biological tissue. The ML alignment model may generate a deformation vector field that represents a registration transformation between the evaluation reference image and the evaluation shifted image. The ML alignment model may also generate an inverse deformation vector field corresponding to the deformation vector field. Training the ML alignment model may include determining a solution to an optimization problem with respect to a loss function based on: (i) a similarity metric of a pair of training label maps; and (ii) the deformation vector field associated with a first training reference image and a first training shifted image in a pair of the multiple pairs of training images. Such a solution defines the trained machine learning alignment model. In one example, the trained ML alignment model may be alignment model 158( Figure 1 ).

[0126] The solution to the optimization problem may be determined iteratively. Thus, determining such a solution may continue iteratively until a termination condition has been satisfied. Determining the solution to the optimization problem may include generating a current deformation field vector by applying a current alignment model to a first pair of training images. The current alignment model is configured in the current iteration of determining the solution to the optimization problem. Additionally, determining the solution to the optimization problem may include applying the current deformation field vector to a shifted label map of the first pair of training label maps, thereby producing a registered shifted label map. Further, determining the solution to the optimization problem may include determining a value of a loss function (e.g., L(m,f) shown in Equation (1)) based on (i) a reference label map of the first pair of training label maps, (ii) the registered shifted label map, and (iii) the current deformation field vector. Additionally, determining the solution to the optimization problem may include generating a next deformation field vector based on the value of the loss function.

[0127] At block 1530, the computing system may supply the trained alignment model. Supplying the trained alignment model may include storing the model in a data storage device (e.g., model storage device 640( Figure 6 )) and, in some cases, configuring an interface (such as an API) to allow access to the stored trained alignment model via one or more function calls.

[0128] Figure 16 is a flow chart of an example method 1600 for generating synthetic training data for training an ML alignment model. A computing system implementing example method 1500( Figure 15 ) or, in some cases, a computing device may implement example method 1600. As described herein, the synthetic training data defines training record instances that include corresponding synthetic image quadruples. Each quadruple consists of a pair of training label maps and a pair of training images. Generating the synthetic training data includes generating multiple such quadruples.

[0129] At block 1610, the computing system can configure tags for corresponding pixels across a region of a defined size. The region can be a square where one side has N pixels and the other side has N pixels. In some configurations N = 2 q where q is a natural number. For example, N can be equal to 256.

[0130] At block 1620, the computing system can generate a base label map based on the configured tags. A base label is a composite image across a region of a defined size. As described herein, the base label map can be used to generate a first label map and a second label map. Specifically, at block 1630, the computing system can generate a reference label map by warping the base label map using a first simplex noise field. Additionally, at block 1640, the computing system can also generate a shifted label map by warping the base label map using a second simplex noise field. Although not shown, the computing system can configure the reference label map and the shifted label map to be associated with a particular synthetic image quadruple.

[0131] At block 1650, the computing system can generate a training reference image based on the reference label map. Additionally, at block 1660, the computing system can generate a training shifted image based on the shifted label map.

[0132] At block 1670, the computing system can configure the training reference image and the training shifted image to be associated with a pair of training images in a particular synthetic image quadruple.

[0133] At block 1680, the computing system can configure the reference label map and the shifted label map to be associated with a pair of training label maps within a particular synthetic image quadruple.

[0134] Figure 17 Illustrated is an example method 1700 for obtaining an alignment model and aligning an image of biological tissue with a common coordinate system in accordance with one or more embodiments of the present disclosure. As described herein, the image can be a histological stain image, an IHC image, or an ISH image (such as a FISH image). A computing device or a computing device system can implement the example method 1700 in whole or in part. To that end, each computing device in the computing device includes computing resources that can implement at least one of the blocks included in the example method 1700. Computing resources include, for example, a CPU, a GPU, a TPU, memory, disk space, incoming bandwidth and / or outgoing bandwidth, an interface (such as an I / O interface or an API, or both); a controller device; a power supply; combinations of the foregoing; and / or similar resources. In one example, one or more of the computing device systems can include a programming interface; an operating system; software for configuring and / or controlling a virtualized environment; firmware; and similar resources. The computing device system can be referred to as a computing system.

[0135] In some cases, the computing system implementing instance method 1700 may host an image generator module 610( Figure 6 ), a training module 620( Figure 6 ), an ingestion module 130, a normalization module 140, a transformation module 150 (including a mosaicking component 310 in some cases), and an alignment module 160, as well as other software components / modules. For example, a computing device may implement instance method 1700 by executing one or more instances of image generator module 610( Figure 6 ), training module 620( Figure 6 ), ingestion module 130, normalization module 140, transformation module 150 (including a mosaicking component 310 in some cases), and alignment module 160. Thus, in response to execution, image generator module 610( Figure 6 ), training module 620( Figure 6 ), ingestion module 130, normalization module 140, transformation module 150 (including a mosaicking component 310 in some cases), and alignment module 160 may perform operations corresponding to the blocks of instance method 1700, either individually or in combination.

[0136] At block 1710, the computing system may generate multiple pairs of training images based on multiple pairs of label maps. Each pair of the multiple pairs of training images includes a training reference image and a training shifted image. Each pair of the multiple pairs of training label maps includes a training reference label map and a training shifted label map.

[0137] At block 1720, the computing system may train an ML alignment model based on the multiple pairs of label maps and the multiple pairs of training images for registering an evaluation shifted image of a biological tissue relative to an evaluation reference image of the biological tissue. The machine learning alignment model may generate a deformation vector field that represents the registration transformation between the evaluation reference image and the evaluation shifted image. Training the ML alignment model may include determining a solution to an optimization problem with respect to a loss function based on: a similarity metric of a pair of label maps; and a deformation vector field associated with a first training reference image and a first training shifted image in a pair of the multiple pairs of training images. Such a solution defines the trained machine learning alignment model.

[0138] At block 1730, the computing system may receive a specific reference image of a biological tissue and a specific shifted image of the biological tissue.

[0139] At block 1740, the computing system may apply a normalization process to a specific reference image and a specific shifted image. As described herein, the normalization process may include various operations involving tissue images and optionally, transcriptomic analysis data collected on the same slide used to obtain the tissue images. To apply the normalization process to the reference image and the shifted image, the computing device may implement the example method 500 (FIG. 5) described below and optionally, the example method 600( Figure 6 ). When implementing these example methods, the reference image and the shifted image are each used as the input tissue images in the respective applications of the normalization process.

[0140] At block 1750, the computing system may perform a first registration of the normalized specific shifted image relative to the normalized specific reference image. As described herein, performing the first registration includes determining an affine transformation between the normalized shifted image and the normalized reference image. Performing such a first registration produces a second shifted image.

[0141] At block 1760, the computing system may supply the specific reference image and the second specific shifted image to a trained ML alignment model.

[0142] At block 1770, the computing system may perform a second registration of the second specific shifted image relative to the specific reference image by applying the machine learning alignment model to the specific reference image and the second specific shifted image.

[0143] Figures 18A to 18J Further alignment performance and / or validation data associated with various scenarios having different slide types and slide arrangements in accordance with embodiments of the present disclosure are shown. In Figures 18A to 18J , for ease of naming, the performance and / or validation statistics associated with embodiments of the present disclosure have generally been referred to as "ML aligner".

[0144] Figure 18A and Figure 18B show the performance of an ML aligner trained using synthetic images as described herein compared to the performance of previously published methods (referred to as "PASTE" and "GPSA" herein and in the figures, respectively). Figure 18A A synthetic / reference image 1801 or a real histological image not used for training is shown in. The image 1801 was generated by adding simplex noise of different amplitudes. Figure 18B A reference image 1803 of a mouse hindbrain coronal section / slide is shown in. The image 1803 was generated using simplex noise wrapping and manual wrapping to produce a set of moving images. As Figure 18A and Figure 18BAs shown in column (A), images 1801 and 1803 are digitally distorted to low, medium, or high levels, or manually distorted to generate a series of moving / shifting images. The original, undistorted images are used as references (1801, 1803). The degree of distortion is quantified as the Figure 18A and Figure 18B "shift" NCC scores in bar graphs 1805 and 1807 shown in respectively. The ML aligner is applied together with the affine method (e.g., affine transformation) and the non-linear alignment method provided by the Advanced Normalization Tool (referred to herein as "ANT") to align each of the moving / shifting images with the corresponding reference image 1801 or 1803. Figure 18A and Figure 18B The aligned images from each method are shown, together with their post-alignment NCC scores shown in bar graphs 1805 and 1807. The NCC values shown for the ML aligner and ANT are the means of 10 repeated runs. The associated error bars are also plotted, but are very small. The statistical significance of the increase in the NCC value of the ML aligner relative to other methods is marked with asterisks and explained in the left-side annotations 1809( Figure 18A ) and 1811( Figure 18B ).

[0145] Figure 18C Shows an evaluation of the ML aligner in aligning Figure 18C digitally distorted spatial transcriptomic slices / slides / images of a mouse brain shown as the reference image 1811 compared to the performance of previously published methods "PASTE" and "GPSA". Specifically, the reference image 1811 shows a mouse sagittal hindbrain slice / slide / image mapped by the 10x Genomics Visium platform ("Visium"). The reference image 1811 is digitally distorted to low (noise amplitude = 5, NCC of the deformed image = 0.606), medium (noise amplitude = 10, NCC of the deformed image = 0.566), or high (noise amplitude = 20, NCC of the deformed image = 0.534) levels using simplex noise to generate Figure 18C a series of moving slices / slides / images shown in column (A) (for all distortions, the noise frequency is kept at 1). As indicated by legend 1813, the grey spots correspond to the spots in the reference image 1811. Legend 1813 also indicates that the blue crosses correspond to Figure 18C the spots in a series of moving slices / slides / images in column (B), while the red crosses correspond to Figure 18CSpots in the aligned images in columns (B)-(D). The difference between the spatial coordinates of the spots in each moving section / slide / image and those in the reference section / slide / image is quantified by the mean squared error (method) shown in bar chart 1815. The ML aligner, together with the previously published methods PASTE and GPSA, is applied to align each of the moving spatial transcriptomic sections / slides / images with the reference image 1811. Figure 18C The coordinates of the spots before alignment (blue crosses, as indicated by legend 1813) and after alignment (red crosses, as indicated by legend 1813) are shown, as well as the spot coordinates of the reference image 1811 (gray spots, as indicated by legend 1813), to aid in visual comparison. The aligned MSE from each method is shown in bar chart 1815. The numerical values from the ML aligner and GPSA are the average of 10 runs and are shown with smaller error bars in bar chart 1815.

[0146] Figure 18D Chart 1819 is shown, which depicts the class labels 1817 of the tissue spots in the reference image 1811 and Figure 18C the spatial consistency of the slides aligned by the ML aligner in. The data presented is taken from Figure 18C a highly distorted example (simplex noise amplitude = 20, NCC of the deformed image = 0.534). The reference image 1811 (shown as a circle in Figure 18C and the moving image (shown as crosses in Figure 18C share the same color code for class 1817. The displayed spots on the moving slide have been aligned by the ML aligner and have the top 10% largest MSEs relative to their reference counterparts. Figure 18C Most of the crosses shown in are correctly located within the region containing the circles of the same color, indicating that their spatial positions are valid for their class labels 1817.

[0147] Figure 18E Chart 1821A - F is shown, which depicts the trajectory of the GPSA loss when aligning the moving slides shown in column (D) of Figure 18C with low distortion (1821A, 1821B), medium distortion (1821C, 1821D), and high distortion (1821E, 1821F). Charts 1821A and 1821B depict the trajectory of the GPSA loss when aligning the moving slides shown in Figure 18D with low distortion. Charts 1821C and 1821D depict the trajectory of the GPSA loss when aligning the moving slides shown in Figure 18D with medium distortion. And charts 1821D and 1821E depict the trajectory of the GPSA loss when aligning the moving slides shown in Figure 18DThe trajectory of GPSA loss when the moving slide shown is aligned with high distortion. For plots 1821A, 1821C, and 1821E, GPSA was performed using the parameters of the Visium platform 1823 (m_X_per_view = 200, m_G = 200, and N_GENES = 10). According to the loss trajectory, GPSA ran for 20,000 epochs or longer until convergence was reached. Plots 1821B, 1821D, and 1821F show the trajectory of GPSA loss when the same moving slide (shown in Figure 18D is aligned) is used with the parameters 1825 (N_GENES = 100, while keeping m_X_per_view = 200 and m_G = 200).

[0148] Figure 18F Plots 1827A - 1827D are shown. Figure 18F Each of the plots shown in Figure 18C is a comparison of the GPSA - aligned spot coordinates (shown as red crosses) with the reference coordinates (shown as gray spots) when the moving slide shown in the (D) column of Figure 18F is aligned. For plot 1821A, the parameters of the Visium platform were set to m_X_per_view = 200, m_G = 200, and N_GENES = 100. For plot 1821B, the parameters of the Visium platform were set to m_X_per_view = 100, m_G = 100, and N_GENES = 100. For plot 1821C, the parameters of the Visium platform were set to m_X_per_view = 50, m_G = 50, and N_GENES = 100. For plot 1821D, the parameters of the Visium platform were set to m_X_per_view = 100, m_G = 100, and N_GENES = 10. In all cases, as Figure 18F shown, GPSA produced aggregated spatial coordinates.

[0149] Figure 18G Shows the performance of the ML aligner in aligning Figure 18G the digitally distorted spatial transcriptomic sections / slides / images of human lymph nodes shown as the reference image 1829 compared to the performance of the previously published methods "PASTE" and "GPSA". Specifically, the reference image 1829 is a section of a human lymph node mapped using the 10x Genomics Visium platform. The reference image 1829 was digitally distorted to low (noise amplitude = 5, NCC of the distorted image = 0.621), medium (noise amplitude = 10, NCC of the distorted image = 0.540), or high (noise amplitude = 20, NCC of the distorted image = 0.504) levels using simplex noise to generate Figure 18GA series of moving sections / slides / images shown in column (A) (for all distortions, the noise frequency is kept at 1). As indicated by legend 1831, the grey spots correspond to the spots in the reference image 1829. Legend 1831 also indicates that the blue crosses correspond to Figure 18G the spots in a series of moving sections / slides / images in column (B) of Figure 18G while the red crosses correspond to the spots in the aligned images in columns (B)-(D) of Figure 18G The difference between the spatial coordinates of the spots in each moving section and those in the reference section / slide / image is quantified by the MSE (method) shown in bar chart 1833. The ML aligner, together with the previously published methods PASTE and GPSA, was applied to align Figure 18G each of the moving spatial transcriptomics sections / slides / images in column (B) with the reference image 1829.

[0150] Figure 18H shows the performance of the ML aligner in de novo alignment of spatial transcriptomics sections / slides / images. Figure 18H The top row of Figure 18H shows four moving spatial transcriptomics sections / slides / images that were independently distorted according to the reference image 1811 taken from the mouse hindbrain described above. Figure 18H For Figure 18H the reference image 1811 was distorted using random seeded simplex noise with an amplitude of 15 and a frequency of 1. The mean pairwise NCC among the tissue images of the moving sections / slides / images was 0.198. The average pairwise MSE among the spot coordinates in the moving sections / slides / images was 0.10. Figure 18H The bottom row of shows the spot coordinates from the four sections / slides / images shown in the top row before alignment (“unaligned coordinates”) and after alignment by the ML aligner, PASTE, and GPSA, respectively, using the same colors and cross symbols as shown in the top row of Figure 18H The post-alignment average MSE for all pairs of sections was 0.046, 0.105, and 0.55 for the ML aligner, PASTE, and GPSA, respectively.

[0151] Figure 18I Shows the application results of the ML aligner in real - space transcriptome sections / slides / images. Figure 18I Two consecutive sagittal sections of a mouse brain (drawn by Visium) are shown in the top row at 1837, where the two sections / slides / images are labeled "Slide 1" and "Slide 2" respectively. Before alignment, the differences between the sections / slides / images shown in 1837 are presented in the superimposed tissue image at 1839. The lower / red part of 1839 corresponds to Slide 1 of 1837, and the upper / green part of 1839 corresponds to Slide 2 of 1837. The spatial consistency of tissue spot clusters between the sections / slides / images shown in 1837 is evaluated in the form of superimposed class - label maps before alignment (1841) and after alignment (1843). The sections / slides / images in 1837 contain the same classes, which are depicted in Figure 18I each of 1841 and 1843 with yellow and blue shading. Each superimposed map shown in 1841 and 1843 is generated by combining the colors of two sections / slides / images (1837) from each RGB channel with a mixing ratio of 0.5 for each, and the regions showing various shades of gray in 1841 and 1843 indicate the consistency between the class labels of the two sections, slides / images in 1837. The incidental Dice score between the unaligned label maps (1841) is 0.794. After alignment by the ML aligner, the Dice score increases to 0.867 (1843).

[0152] In Figure 18I the bottom row at 1845 of Figure 18I four biological replicates of a dissected mouse olfactory bulb are shown as Slides 1 - 4. To illustrate their spatial inconsistency, Figure 18I the bottom row at 1849 of Figure 18Ias shown in 1853).

[0153] Figure 18J Two plots, 1855 and 1857, are shown, which demonstrate batch effects in the mouse olfactory bulb spatial transcriptomics dataset. In Figure 18J Plot 1855 shown on the left, tissue spots from the above four mouse olfactory bulb slides ( Figure 18I slides 1 - 4 shown at 1845) are clustered based on their gene expression profiles and visualized in UMAP. The spots in plot 1855 form four slide - specific clusters, indicating non - negligible batch effects on the slides. As indicated by legend 1855A of plot 1855, the red cluster corresponds to slide 1 of 1845; the green cluster corresponds to slide 2 of 1845; the blue cluster corresponds to slide 3 of 1845; and the purple cluster corresponds to slide 4 of 1845. In Figure 18J Plot 1857 shown on the right, the tissue spot coordinates from Figure 18I slides 1 - 4 shown at 1845 are shown after de novo alignment by GPSA. As indicated by legend 1857A of plot 1855, the red crosses in 1857 correspond to slide 1 of 1845; the green crosses in 1857 correspond to slide 2 of 1845; the blue crosses in 1857 correspond to slide 3 of 1845; and the purple crosses in 1857 correspond to slide 4 of 1845. Plot 1857 shows a severe deviation of the coordinates from the expected square grid, revealing the limitations of GPSA. Additionally, among the aligned slides themselves, the pairwise consistency of the spot coordinates is low (mean pairwise MSE = 24.09), indicating that GPSA is vulnerable to batch effects in transcriptomic data.

[0154] The method and system for image alignment according to aspects described herein can be implemented on Figure 19 the computing system 1900 shown and described below. The computer - implemented methods, devices, and systems disclosed herein can perform one or more functions using one or more computing devices at one or more locations. Figure 19 is a block diagram depicting an example computing system 1900 for performing the disclosed methods and / or implementing the disclosed systems. Computing system 1900 is only an example of an operating environment and is not intended to imply any limitation as to the scope of use or functionality of the operating environment architecture. The operating environment should not be interpreted as having any dependencies or requirements related to any one or combination of the components shown in the exemplary operating environment. Figure 19 The computing environment 1900 shown in Figure 1 or Figure 3)。The computing system 1900 can implement various functions related to the alignment of images of biological tissues relative to a common coordinate system described herein. For example, one or more of the computing devices that form the computing system 1900 can include an ingestion module 130, a normalization module 140, a transformation module 150 (including a mosaicking component 310 in some cases), and an alignment module 160. Additionally, or in some embodiments, as described herein, one or more computing devices that form the computing system 2500 can further include an image generator module 610 and a training module 620.

[0155] The computer-implemented methods, devices, and systems according to the present disclosure can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with the system and method include, but are not limited to, personal computers, server computers, laptop devices, and multiprocessor systems. Other examples include set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0156] The processing of the disclosed computer-implemented methods, devices, and systems can be performed by software components. The disclosed systems, devices, and computer-implemented methods can be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices. Generally, program modules include computer code, routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The disclosed methods can also be practiced in grid-based computing environments and distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media (including memory storage devices).

[0157] Further, the systems, devices, and computer-implemented methods disclosed herein can be implemented via a general-purpose computing device in the form of a computing device 1901. The components of the computing device 1901 can include one or more processors 1903, a main memory 1912, and a system bus 1913 that couples various system components including the one or more processors 1903 to the main memory 1912. The system can employ parallel computing.

[0158] The system bus 1913 represents one or more of several possible types of bus structures, including a memory bus or memory controller, a peripheral bus, an Accelerated Graphics Port, or a local bus using any one of a variety of bus architectures. The system bus 1913 and all of the buses specified in this specification may also be implemented via wired or wireless network connections, and each of the subsystems, including one or more processors 1903, a mass storage device 1904, an operating system 1905, software 1906, data 1907, a network adapter 1908, a system memory 1912, an input / output interface 1910, a display adapter 1909, a display device 1911, and a human machine interface 1902, may be included in one or more remote computing devices 1914a, b, c, located at physically separate locations, connected by this form of bus, thereby effectively implementing a fully distributed system.

[0159] The computing device 1901 generally includes a variety of computer-readable media. Exemplary readable media can be any available media that can be accessed by the computing device 1901 and includes both volatile and nonvolatile media, removable and non-removable media. The main memory 1912 includes computer-readable media in the form of volatile memory, such as random access memory (RAM), and / or nonvolatile memory, such as read-only memory (ROM). The main memory 1912 typically contains data such as data 1907 and / or program modules such as the operating system 1905 and software 1906, which can be immediately accessed by and / or currently operated on by one or more processors 1903. For example, the software 1906 can include an ingestion module 130, a normalization module 140, a transformation module 150 (including a mosaicking component 310 in some cases), and an alignment module 160. Additionally, or in other embodiments, the software 1906 can also include an image generator module 610 and a training module 620. For example, the operating system 1905 can be embodied in one of the Windows operating system, Unix, or Linux.

[0160] In another aspect, the computing device 1901 may also include other removable / non-removable, volatile / nonvolatile computer storage media. For example, Figure 19 a mass storage device 1904 is shown, which can provide nonvolatile storage of computer code, computer-readable instructions, data structures, program modules, and other data for the computing device 1901. For example and without intending to be limiting, the mass storage device 1904 can be a hard disk, a removable magnetic disk, a removable optical disk, a magnetic tape cartridge, or other magnetic storage device, a flash memory card, a CD-ROM, a digital versatile disk (DVD), or other optical storage, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0161] Any number of program modules can be stored on the mass storage device 1904, such as including an operating system 1905 and software 1906. Each of the operating system 1905 and the software 1906 (or some combination thereof) can include elements of programming and software 1906. Data 1907 can also be stored on the mass storage device 1904. The data 1907 can be stored in any one of one or more databases known in the art. Examples of such databases include Access, SQL Server, mySQL, PostgreSQL, SQLite, etc. The database can be centralized or distributed across multiple systems. The mass storage device 1904 and the main memory 1912 can, individually or in combination, embody or include the memory 170. Additionally, or in some embodiments, the mass storage device 1904 can embody or include an image storage device 630 and a model storage device 640. Further, or in still other embodiments, a combination of mass storage devices 1904 that may be present in remote computing devices 1914a, b, c can embody or include a data storage device 110.

[0162] On the other hand, a user can input commands and information into the computing device 1901 via an input device (not shown). Examples of such input devices include, but are not limited to, a keyboard, a pointing device (e.g., a “mouse”), a microphone, a joystick, a scanner, a haptic input device (such as gloves and other body coverings), etc. These and other input devices can be connected to one or more processors 1903 via a human-machine interface 1902 coupled to the system bus 1913, but can also be connected through other interfaces and bus structures, such as a parallel port, a game port, an IEEE 1394 port (also known as a FireWire port), a serial port, or a universal serial bus (USB).

[0163] In yet another aspect, the display device 1911 may also be connected to the system bus 1913 via an interface such as a display adapter 1909. It is contemplated that the computing device 1901 may have more than one display adapter 1909 and the computing device 1901 may have more than one display device 1911. For example, the display device 1911 may be a monitor, an LCD (Liquid Crystal Display), or a projector. In addition to the display device 1911, other output peripheral devices may include components that can be connected to the computing device 1901 via the input / output interface 1910, such as speakers (not shown) and printers (not shown). Any operation and / or result of the method may be output to the output device in any form. Such output may be any form of visual representation, including but not limited to text, graphics, animation, audio, tactile, etc. The display device 1911 and the computing device 1901 may be part of one device or separate devices.

[0164] The computing device 1901 may operate in a network environment using logical connections to one or more remote computing devices 1914a, b, c. For example, the remote computing device may be a personal computer, a portable computer, a smartphone, a server device, a router device, a network computer, a peer device, or other common network nodes, etc. The logical connection between the computing device 1901 and the remote computing devices 1914a, b, c may be made via a network 1915 such as a LAN and / or a general WAN. Such network connections may be through a network adapter 1908. The network adapter 1908 may be implemented in both wired and wireless environments.

[0165] For illustrative purposes, application programs and other executable program components such as operating system 1905 are shown herein as discrete blocks, but it should be recognized that such programs and components reside at different times in different storage components of computing device 1901 and are executed by one or more processors 1903 of the computer. Implementations of software 1906 may be stored on or transmitted via some form of computer-readable medium. Any of the disclosed methods may be performed by computer-readable instructions embodied on a computer-readable medium. A computer-readable medium may be any available medium that can be accessed by a computer. By way of example and not limitation, a computer-readable medium may include "computer storage media" and "communication media". Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Exemplary computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer.

[0166] It should be understood that the methods and systems described herein are not limited to the specific operations, processes, components or structures described, nor to the order or particular combination of such operations or components. It should also be understood that the terms used herein are for the purpose of describing example embodiments only and are not intended to be limiting or restrictive.

[0167] As used herein, unless the context clearly dictates otherwise, the singular forms "a", "an" and "the" include both single referents and plural referents. In reference to numerical values expressed as approximations using antecedents such as "about" or "approximately", a numerical value so expressed should include a reasonable deviation from the reference value. If such approximations are included within a range, not only the endpoints are considered approximations, but the extent of the range should also be considered an approximation. Unless the context clearly dictates otherwise, lists should be considered exemplary and not limited to the elements or the order of listing of the elements comprising the list.

[0168] Throughout the specification and claims of the present disclosure, the following terms have the meanings set forth: "comprise" and variations thereof, such as "comprising" and "comprises", mean including but not limited to, and are not intended to exclude, for example, other additives, components, integers, or operations. "Include" and variations thereof, such as "including", are not intended to mean that the content is limited or restricted to the content indicated as being included, or to exclude content not indicated. "May" means permitted but not limited or restricted content. "Optional" or "optionally" means content that may or may not be included without changing the result or the content described. "Preferred" and variations thereof, such as "preferred" or "preferably", mean exemplary and more desirable but not essential content. "Such as" means content that is only an example.

[0169] Unless the context clearly dictates otherwise, the operations and components described herein for performing the disclosed methods and constructing the disclosed systems are illustrative. It should be understood that when combinations, subsets, interactions, groups, etc. of these operations and components are disclosed, although specific references to each individual and collective combination and arrangement of these components are not explicitly disclosed, each combination and arrangement is specifically contemplated and described herein for all methods and systems. This applies to all aspects of the present application, including but not limited to the operations in the disclosed methods and / or the disclosed components in the systems. Thus, if there are multiple additional operations that can be performed or components that can be added, it should be understood that each of these additional operations and each of these components can be performed and added in combination with any particular embodiment or embodiments of the disclosed systems and methods.

[0170] Embodiments of the present disclosure may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware aspects. Additionally, the methods and systems may take the form of a computer program product on a computer-readable storage medium, in which computer-readable program instructions (e.g., computer software) are embodied. Any suitable computer-readable storage medium can be utilized, including a hard disk, CD-ROM, optical storage device, magnetic storage device, memristor, non-volatile random access memory (NVRAM), flash memory, or a combination thereof, whether internal, networked, or cloud-based.

[0171] Embodiments of the present disclosure have been described with reference to the drawings, flowcharts, and other illustrations of computer-implemented methods, systems, apparatuses, and computer program products. Each block of the block diagrams and flowcharts, and combinations of blocks in the block diagrams and flowcharts, can be implemented by processor-accessible instructions. Such instructions can include, for example, computer program instructions (e.g., processor-readable and / or processor-executable instructions). The processor-accessible instructions can be constructed (e.g., linked and compiled) and stored in a processor-executable form in one or more memory devices or one or more other processor-accessible non-transitory storage media. These computer program instructions (constructed or otherwise) can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine. The loaded computer program instructions can be accessed and executed by one or more processors or other types of processing circuitry. In response to execution, the loaded computer program instructions provide the functions described in connection with the flowchart blocks (individually or in a particular combination) or the blocks in the block diagrams (individually or in a particular combination). Thus, such instructions executed on a computer or other programmable data processing apparatus create means for implementing the functions specified in the flowchart blocks (individually or in a particular combination) or the blocks in the block diagrams (individually or in a particular combination).

[0172] These computer program instructions can also be stored in a computer-readable memory, which can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture for implementing the functions specified in the flowchart blocks (individually or in a particular combination) or the blocks in the block diagrams (individually or in a particular combination), including processor-accessible instructions (e.g., processor-readable instructions and / or processor-executable instructions). The computer program instructions (constructed or otherwise) can also be loaded onto a computer or other programmable data processing apparatus such that a series of operations are performed on the computer or other programmable device to produce a computer-implemented process. The series of operations can be performed in response to the execution by one or more processors or other types of processing circuitry. Thus, such instructions executed on a computer or other programmable device provide operations for implementing the functions specified in the flowchart blocks (individually or in a particular combination) or the blocks in the block diagrams (individually or in a particular combination).

[0173] Thus, the blocks of the block diagrams and flowcharts support combinations of means for performing the specified functions associated with such illustrations and / or flowcharts, combinations of operations for performing the specified functions, and program instruction means for performing the specified functions. Each block of the block diagrams and flowcharts, and combinations of blocks in the block diagrams and flowcharts, can be implemented by a dedicated hardware-based computer system that performs the specified functions or operations or a combination of dedicated hardware and computer instructions.

[0174] The method and system may employ artificial intelligence techniques, such as machine learning and iterative learning. Examples of such techniques include, but are not limited to, expert systems, case-based reasoning, Bayesian networks, behavior-based AI, neural networks, fuzzy systems, evolutionary computing (e.g., genetic algorithms), swarm intelligence (e.g., ant algorithms), and hybrid intelligent systems (e.g., expert inference rules generated by neural networks or rules generated by statistical learning).

[0175] Although the computer-implemented methods, apparatus, devices, and systems have been described in conjunction with preferred embodiments and specific examples, it is not intended to limit the scope to the specific embodiments set forth, as the embodiments herein are intended in all respects to be illustrative rather than restrictive.

[0176] Unless otherwise expressly stated, it is not intended that any method described herein be construed as requiring that its operations be performed in a particular order. Therefore, in the event that a method claim does not actually recite the order in which its operations are to be performed or the operations are not otherwise specifically stated in the claims or specification to be limited to a particular order, no order is intended to be inferred in any respect. This applies to any possible non-expressive basis for interpretation, including: logical issues regarding the arrangement of operations or operational flows; plain meaning derived from grammatical organization or punctuation; the number or type of embodiments described in the specification.

[0177] It will be apparent to those skilled in the art that various modifications and variations may be made without departing from the scope or spirit. Other embodiments will be apparent to those skilled in the art from considerations of the specification and practice disclosed herein. This specification and examples are intended to be considered exemplary only, with the true scope and spirit being indicated by the appended claims.

[0178] Example embodiments:

[0179] Embodiment 1. A computer-implemented method, comprising: receiving a reference image of a biological tissue; receiving a shifted image of the biological tissue; applying a normalization process to the reference image to produce a normalized reference image; applying a normalization process to the shifted image to produce a normalized shifted image; performing a first registration of the normalized shifted image relative to the normalized reference image to produce a group of parameters defining a coarse transformation and further producing a second shifted image; supplying the normalized reference image to a machine learning alignment model; supplying the second shifted image to the machine learning alignment model; and performing a second registration of the second shifted image relative to the reference image by applying the machine learning alignment model to the reference image and the second shifted image, wherein the application produces a deformation vector field representing the registration transformation between the reference image and the second shifted image.

[0180] Example 2. The computer-implemented method according to Example 1 further includes: receiving reference spatial coordinates of spots within a first transcriptomic map of the biological tissue, the first transcriptomic map corresponding to the reference image, wherein each spot includes one or more cells; receiving shifted spatial coordinates of spots within a second transcriptomic map of the biological tissue, the second transcriptomic map corresponding to the shifted image; performing a first registration of the shifted spatial coordinates relative to the reference spatial coordinates based on the rough transformation, thereby generating second shifted spatial coordinates of the spots within the second transcriptomic map; and performing a second registration of the second shifted spatial coordinates of the spots within the second transcriptomic map based on the registration transformation.

[0181] Example 3. The computer-implemented method according to any one of Examples 1 or 2, wherein supplying the normalized reference image includes: generating patches of the normalized reference image, wherein the patches of the normalized reference image are composed of a first defined number of tile images; and sequentially supplying each tile image in the patches of the normalized reference image to the machine learning alignment model.

[0182] Example 4. The computer-implemented method according to Example 3, wherein supplying the second shifted image includes: generating patches of the second shifted image, wherein the patches of the second shifted image are composed of a second defined number of tile images, the second defined number being equal to the first defined number; and sequentially supplying each tile image in the patches of the second shifted image to the machine learning alignment model, wherein the first tile image in the patches of the second shifted image is supplied concurrently with the first tile image in the patches of the normalized reference image; wherein the first tile image in the patches of the second shifted image spans a portion of the second shifted image, and wherein the first tile image in the patches of the normalized reference image spans a portion of the patches of the normalized reference image, and wherein the portion of the second shifted image and the portion of the normalized reference image are identical to each other in placement and size.

[0183] Example 5. The computer-implemented method according to Example 4, wherein applying the machine learning alignment model includes: applying the machine learning alignment model to the first tile image in the patches of the normalized reference image and the first tile image in the patches of the second shifted image.

[0184] Example 6. The computer-implemented method according to Example 5, wherein applying the machine learning alignment model generates a first tile deformation vector field and a second tile deformation vector field, and the method further includes: concatenating the first tile deformation vector field and the second tile deformation vector field to at least partially form the deformation vector field.

[0185] Example 7. The computer-implemented method according to Example 6, wherein each of the first tile deformation vector field, the second tile deformation vector field, and the corresponding adjacent tile deformation vector fields has a common defined region, and the concatenating includes: for each pixel within the common defined region, determining a weighted average of one of the first tile deformation vector field or the second tile deformation vector field and one of the corresponding adjacent tile deformation vector fields that overlaps at the pixel; and for each pixel within the common defined region, assigning the weighted average to the deformation vector field.

[0186] Example 8. The computer-implemented method according to any one of the preceding examples, wherein the rough transformation is an affine transformation, and wherein performing the first registration includes: determining the affine transformation between the normalized shifted image and the normalized reference image.

[0187] Example 9. The computer-implemented method according to any one of the preceding examples, wherein the normalization process includes: receiving an input image of the biological tissue; generating a tissue mask image of the input image; and configuring non-tissue pixels in the tissue mask image as black pixels; wherein the tissue mask image having black pixels constitutes the normalized input image.

[0188] Example 10. The computer-implemented method according to any one of the preceding examples, wherein the normalization process further includes: receiving transcriptome analysis data corresponding to the input image; generating a label map based on the transcriptome analysis data, the label map including a first label associated with a plurality of first pixels and a second label associated with a plurality of second pixels; generating a contour map based on the label map, wherein a first contour in the contour map defines a boundary separating a subset of the plurality of first pixels and a subset of the plurality of second pixels; identifying specific pixels corresponding to the first contour within the normalized input image; and updating the normalized input image by modifying corresponding values of the specific pixels, each of the corresponding values being generated by a linear combination of a first value from the normalized input image and a second value based on the transcriptome analysis data.

[0189] Example 11. The computer-implemented method according to any one of the preceding examples, wherein the machine learning alignment model comprises a convolutional neural network (CNN), the CNN having an encoder module, a decoder module, and a field construction module, the field construction module being configured to output the deformation vector field.

[0190] Example 12. The computer-implemented method according to any one of the preceding examples, wherein the machine learning alignment model comprises a neural network, the neural network comprising: one or more first layers configured to extract features from the normalized reference image and the second shifted image, the one or more first layers being of a first type; and one or more second layers configured to output the deformation vector field, the one or more second layers being of a second type.

[0191] Example 13. A computer-implemented method, comprising: generating a plurality of pairs of training images based on a plurality of pairs of training label maps, wherein each pair of the plurality of pairs of training images comprises a training reference image and a training shifted image; determining a solution to an optimization problem with respect to a loss function based on a similarity measure of a pair of training label maps and a deformation vector field representing a registration transformation between a first training reference image and a first training shifted image in a pair of the plurality of pairs of training images, wherein the solution defines an alignment model for registering an evaluation shifted image of a biological tissue with respect to an evaluation reference image of the biological tissue.

[0192] Example 14. The computer-implemented method according to Example 12, wherein determining the solution to the optimization problem comprises: generating a current deformation field vector by applying a current alignment model to a first pair of training images, wherein the current alignment model is configured in a current iteration of determining the solution to the optimization problem; applying the current deformation field vector to a shifted label map of the first pair of training label maps, thereby producing a registered shifted label map; determining a value of the loss function based on (i) a reference label map of the first pair of training label maps, (ii) the registered shifted label map, and (iii) the current deformation field vector; and generating a next deformation field vector based on the value of the loss function.

[0193] Example 15. The computer-implemented method according to Example 13, wherein the loss function comprises a Dice similarity coefficient and a gradient of the deformation vector field, wherein the gradient is weighted by a regularization factor.

[0194] Example 16. The computer-implemented method according to any one of Examples 13 to 15, wherein the alignment model includes a convolutional neural network (CNN), the CNN includes an encoder module, a decoder module, and a field construction module, and the field construction module is configured to output the deformation vector field.

[0195] Example 17. The computer-implemented method according to any one of Examples 13 to 16, wherein the machine learning alignment model includes a neural network, the neural network includes: one or more first layers configured to extract features from the normalized reference image and the second shifted image, the one or more first layers being of a first type; and one or more second layers configured to output the deformation vector field, the one or more second layers being of a second type.

[0196] Example 18. The computer-implemented method according to any one of Examples 13 to 17, wherein generating the multiple pairs of training images includes: configuring labels for corresponding pixels across a region of a defined size; and generating a base label map based on the configured labels, the base label map spanning the region of the defined size.

[0197] Example 19. The computer-implemented method according to Example 18, wherein the configuring includes: configuring multiple sets of simplex noise distributions within corresponding layers, each of the layers corresponding to the region of the defined size, wherein a first set of the multiple sets includes a first simplex noise distribution centered at a corresponding position within a first layer of the corresponding layer, and wherein a second set of the multiple sets includes a second simplex noise distribution centered at a corresponding position within a second layer; configuring multiple defined labels for each pixel within the region, each of the multiple defined labels corresponding to a particular layer within the corresponding layer; using the multiple sets of simplex noise distributions to determine corresponding numerical weights of the multiple defined labels at a particular pixel within the region; and assigning a first label to the particular pixel, the first label corresponding to the first numerical weight having the largest magnitude among the corresponding numerical weights.

[0198] Example 20. The computer-implemented method according to Example 19, wherein generating the base label map includes: merging the corresponding layers having labeled pixels into a single defined layer defining the base label map.

[0199] Example 21. The computer-implemented method according to Example 18, wherein generating the multiple pairs of training images further comprises: generating a reference label map by distorting the base label map using a first simplex noise field; generating a specific training reference image based on the reference label map by: randomly configuring the colors of the corresponding labels in the reference label map to produce a color image; blurring the color image to produce a blurred image; and applying a bias field to the blurred image, the bias intensity field spanning the region of the defined size.

[0200] Example 22. The computer-implemented method according to Example 21, wherein generating the multiple pairs of training images further comprises: generating a shifted label map by distorting the base label map using a second simplex noise field; generating a specific training shifted image based on the shifted label map by: randomly configuring the colors of the corresponding labels in the second label map to produce a second color image; blurring the second color image to produce a second blurred image; and applying the bias intensity field to the second blurred image.

[0201] Example 23. The computer-implemented method according to Example 22, wherein generating the multiple pairs of training images further comprises: configuring the specific training reference image and the specific training shifted image to be associated with a specific pair in the multiple pairs of images.

[0202] Example 24. The computer-implemented method according to Example 23, further comprising: configuring the reference label map and the shifted label map to be associated with a specific pair in the multiple pairs of training label maps.

[0203] Example 25. A computer-implemented method includes: generating a plurality of pairs of training images based on a plurality of pairs of training label maps, where each pair of the plurality of pairs of training images includes a training reference image and a training shifted image; training a machine learning alignment model based on the plurality of pairs of label maps and the plurality of pairs of training images for registering an evaluation shifted image of a biological tissue relative to an evaluation reference image of the biological tissue, where the alignment model generates a deformation vector field representing a registration transformation between the evaluation reference image and the evaluation shifted image; receiving a specific reference image of the biological tissue and a specific shifted image of the biological tissue; applying a normalization process to the specific reference image and the specific shifted image; performing a rough registration of the normalized specific shifted image relative to the normalized specific reference image, thereby generating a group of parameters defining a rough transformation and further generating a second specific shifted image; supplying the specific reference image and the second specific shifted image to the trained machine learning alignment model; and performing a fine registration of the second specific shifted image relative to the specific reference image by applying the machine learning alignment model to the specific reference image and the second specific shifted image.

[0204] Example 26. The computer-implemented method according to Example 25 further includes: receiving specific reference spatial coordinates of spots within a first transcriptomic map of the biological tissue, the first transcriptomic map corresponding to the reference image, where each spot includes one or more cells; receiving specific shifted spatial coordinates of spots within a second transcriptomic map of the biological tissue, the second transcriptomic map corresponding to the shifted image; performing a rough registration of the specific shifted spatial coordinates relative to the specific reference spatial coordinates based on the rough transformation, thereby generating second specific shifted spatial coordinates of the spots within the second transcriptomic map; and performing a fine registration of the second specific shifted spatial coordinates of the spots within the second transcriptomic map based on the registration transformation.

[0205] Example 27. The computer-implemented method according to any one of Examples 25 or 26, where the training includes: determining a solution to an optimization problem regarding a loss function based on a similarity metric of a first pair of training label maps among the plurality of pairs of training label maps and a deformation vector field associated with a first training reference image and a first training shifted image in a pair of the plurality of pairs of training images, where the solution defines the trained machine learning alignment model.

Claims

1. A computer-implemented method, comprising: Receiving a reference image of a biological tissue; Receiving a shifted image of the biological tissue; Applying a normalization process to the reference image to produce a normalized reference image; Applying the normalization process to the shifted image to produce a normalized shifted image; Performing a first registration of the normalized shifted image relative to the normalized reference image to produce a group of parameters defining a rough transformation and further producing a second shifted image; Supplying the normalized reference image to a machine learning alignment model; Supplying the second shifted image to the machine learning alignment model; Performing a second registration of the second shifted image relative to the reference image by applying the machine learning alignment model to the reference image and the second shifted image, wherein the application produces a deformation vector field representing a registration transformation between the reference image and the second shifted image.

2. The computer-implemented method according to claim 1, further comprising, Receiving reference spatial coordinates of spots within a first transcriptomic atlas of the biological tissue, the first transcriptomic atlas corresponding to the reference image, wherein each spot comprises one or more cells; Receiving shifted spatial coordinates of spots within a second transcriptomic atlas of the biological tissue, the second transcriptomic atlas corresponding to the shifted image; Performing a first registration of the shifted spatial coordinates relative to the reference spatial coordinates based on the rough transformation to produce second shifted spatial coordinates of the spots within the second transcriptomic atlas; and Performing a second registration of the second shifted spatial coordinates of the spots within the second transcriptomic atlas based on the registration transformation.

3. The computer-implemented method according to any one of claims 1 or 2, wherein supplying the normalized reference image comprises: Generating patches of the normalized reference image, wherein the patches of the normalized reference image are composed of a first defined number of tile images; And Sequentially supplying each tile image in the patches of the normalized reference image to the machine learning alignment model.

4. The computer-implemented method according to claim 3, wherein supplying the second shifted image comprises: Generating patches of the second shifted image, wherein the patches of the second shifted image are composed of a second defined number of tile images, the second defined number being equal to the first defined number; And Sequentially supplying each tile image in the patches of the second shifted image to the machine learning alignment model, wherein the first tile image in the patches of the second shifted image is supplied concurrently with the first tile image in the patches of the normalized reference image; Wherein the first tile image in the patches of the second shifted image spans a portion of the second shifted image, and wherein the first tile image in the patches of the normalized reference image spans a portion of the patches of the normalized reference image, and The portion of the second shifted image and the portion of the normalized reference image are identical to each other in terms of placement and size.

5. The computer-implemented method according to claim 4, wherein applying the machine learning alignment model comprises: Apply the machine learning alignment model to the first tile image of the chunk of the normalized reference image and the first tile image of the chunk of the second shifted image.

6. The computer-implemented method according to claim 5, wherein applying the machine learning alignment model generates a first tile deformation vector field and a second tile deformation vector field, and the method further comprises: Connect the first tile deformation vector field and the second tile deformation vector field to at least partially form the deformation vector field.

7. The computer-implemented method according to claim 6, wherein each of the first tile deformation vector field, the second tile deformation vector field, and the corresponding adjacent tile deformation vector fields has a common defined region, The connection includes, For each pixel within the common defined region, determine a weighted average of one of the first tile deformation vector field or the second tile deformation vector field and one of the corresponding adjacent tile deformation vector fields that overlaps at the pixel; and For each pixel within the common defined region, assign the weighted average to the deformation vector field.

8. The computer-implemented method according to any one of the preceding claims, wherein the coarse transformation is an affine transformation, and wherein performing the first registration comprises: Determine the affine transformation between the normalized shifted image and the normalized reference image.

9. The computer-implemented method according to any one of the preceding claims, wherein the normalization process includes: Receive an input image of the biological tissue; Generate a tissue mask image of the input image; And Configure non-tissue pixels in the tissue mask image as black pixels; Wherein the tissue mask image having black pixels constitutes the normalized input image.

10. The computer-implemented method according to any one of the preceding claims, wherein the normalization process further includes: Receive transcriptome analysis data corresponding to the input image; Generate a label map based on the transcriptome analysis data, the label map including a first label associated with a plurality of first pixels and a second label associated with a plurality of second pixels; Generate a contour map based on the label map, wherein a first contour in the contour map defines a boundary separating a subset of the plurality of first pixels and a subset of the plurality of second pixels; Identify specific pixels corresponding to the first contour within the normalized input image; Update the normalized input image by modifying corresponding values of the specific pixels, each of the corresponding values being generated by a linear combination of a first value from the normalized input image and a second value based on the transcriptome analysis data.

11. The computer-implemented method according to any one of the preceding claims, wherein the machine learning alignment model includes a convolutional neural network (CNN), the CNN having an encoder module, a decoder module, and a field construction module, the field construction module being configured to output the deformation vector field.

12. The computer-implemented method according to any one of the preceding claims, wherein the machine learning alignment model includes a neural network, the neural network including, One or more first layers configured to extract features from the normalized reference image and the second shifted image, the one or more first layers being of a first type; and One or more second layers configured to output the deformation vector field, the one or more second layers being of a second type.

13. A computer-implemented method comprising: Generating a plurality of pairs of training images based on a plurality of pairs of training label maps, wherein each of the plurality of pairs of training images includes a training reference image and a training shifted image; Determining a solution to an optimization problem regarding a loss function based on a similarity metric of a pair of training label maps and a deformation vector field representing a registration transformation between a first training reference image and a first training shifted image in a pair of the plurality of pairs of training images, Wherein the solution defines an alignment model for registering an evaluation shifted image of a biological tissue relative to an evaluation reference image of the biological tissue.

14. The computer-implemented method according to claim 12, wherein determining the solution to the optimization problem comprises: Generating a current deformation field vector by applying a current alignment model to a first pair of training images, wherein the current alignment model is configured in a current iteration of determining the solution to the optimization problem; Applying the current deformation field vector to a shifted label map of the first pair of training label maps, thereby producing a registered shifted label map; Determining a value of the loss function based on (i) a reference label map of the first pair of training label maps, (ii) the registered shifted label map, and (iii) the current deformation field vector; And Generating a next deformation field vector based on the value of the loss function.

15. The computer-implemented method according to claim 13, wherein the loss function includes a Dice similarity coefficient and a gradient of the deformation vector field, wherein the gradient is weighted by a regularization factor.

16. The computer-implemented method according to any one of claims 13 to 15, wherein the alignment model includes a convolutional neural network (CNN), the CNN including an encoder module, a decoder module, and a field construction module configured to output the deformation vector field.

17. The computer-implemented method according to any one of claims 13 to 16, wherein the machine learning alignment model includes a neural network, the neural network including, One or more first layers configured to extract features from the normalized reference image and the second shifted image, the one or more first layers being of a first type; and One or more second layers configured to output the deformation vector field, the one or more second layers being of a second type.

18. The computer-implemented method according to any one of claims 13 to 17, wherein generating the plurality of pairs of training images includes: Configuring labels for corresponding pixels spanning a region of a defined size; And Generating a base label map based on the configured labels, the base label map spanning the region of the defined size.

19. The computer-implemented method according to claim 18, wherein the configuring comprises: Configuring multiple sets of simplex noise distributions within respective layers, each of the layers corresponding to the region of the defined size, wherein a first set of the multiple sets includes a first simplex noise distribution centered at a corresponding location within a first layer of the respective layer, and wherein a second set of the multiple sets includes a second simplex noise distribution centered at a corresponding location within a second layer; Configuring multiple defined labels for each pixel within the region, each of the multiple defined labels corresponding to a specific layer within the respective layer; Using the multiple sets of simplex noise distributions to determine respective numerical weights of the multiple defined labels at a specific pixel within the region; And Assigning a first label to the specific pixel, the first label corresponding to a first numerical weight having the largest magnitude among the respective numerical weights.

20. The computer-implemented method according to claim 19, wherein said generating the base tag graph comprises: Merging the respective layers having labeled pixels into a single defined layer defining the base label map.

21. The computer-implemented method according to claim 18, wherein the generating the multiple pairs of training images further comprises: Generating a reference label map by warping the base label map using a first simplex noise field; Generating a specific training reference image based on the reference label map by: Randomly configuring colors of the respective labels in the reference label map to produce a color image; Blurring the color image to produce a blurred image; And Applying a bias field to the blurred image, the bias intensity field spanning the region of the defined size.

22. The computer-implemented method according to claim 21, wherein the generating the multiple pairs of training images further comprises: Generating a shifted label map by warping the base label map using a second simplex noise field; Generating a specific training shifted image based on the shifted label map by: Randomly configuring colors of the respective labels in the second label map to produce a second color image; Blurring the second color image to produce a second blurred image; And Applying the bias intensity field to the second blurred image.

23. The computer-implemented method according to claim 22, wherein said generating said plurality of pairs of training images further comprises: Configuring the specific training reference image and the specific training shifted image to be associated with a specific pair among the multiple pairs of images.

24. The computer-implemented method according to claim 23, further comprising: Configuring the reference label map and the shifted label map to be associated with a specific pair among the multiple pairs of training label maps.

25. A computer-implemented method, comprising: Generating multiple pairs of training images based on multiple pairs of training label maps, wherein each pair of the multiple pairs of training images includes a training reference image and a training shifted image; Training a machine learning alignment model based on the multiple pairs of label maps and the multiple pairs of training images for registering an evaluation shifted image of a biological tissue relative to an evaluation reference image of the biological tissue, wherein the alignment model produces a deformation vector field representing a registration transformation between the evaluation reference image and the evaluation shifted image; Receiving a specific reference image of the biological tissue and a specific shifted image of the biological tissue; Apply a normalization process to the specific reference image and the specific shifted image; Perform a coarse registration of the normalized specific shifted image relative to the normalized specific reference image, thereby generating a group of parameters defining a coarse transformation and further generating a second specific shifted image; Supply the specific reference image and the second specific shifted image to a trained machine learning alignment model; Perform a fine registration of the second specific shifted image relative to the specific reference image by applying the machine learning alignment model to the specific reference image and the second specific shifted image.

26. The computer-implemented method according to claim 25, further comprising, Receiving specific reference spatial coordinates of spots within a first transcriptomic map of the biological tissue, the first transcriptomic map corresponding to the reference image, wherein each spot comprises one or more cells; Receiving specific shifted spatial coordinates of spots within a second transcriptomic map of the biological tissue, the second transcriptomic map corresponding to the shifted image; Performing a coarse registration of the specific shifted spatial coordinates relative to the specific reference spatial coordinates based on the coarse transformation, thereby generating second specific shifted spatial coordinates of the spots within the second transcriptomic map; and Performing a fine registration of the second specific shifted spatial coordinates of the spots within the second transcriptomic map based on the registration transformation.

27. The computer-implemented method according to any one of claims 25 or 26, wherein the training comprises: Determine a solution to an optimization problem regarding a loss function based on a similarity measure of a first pair of training label maps among a plurality of pairs of training label maps and a deformation vector field associated with a first training reference image and a first training shifted image in a pair among the plurality of pairs of training images, wherein the solution defines the trained machine learning alignment model.

Images