Method for performing image registration

Abstract

A computer-implemented method for performing image registration on a plurality of images of a subject, the method comprising: receiving a first image of the subject and a second image of the subject; performing first image registration to register the first image and the second image; receiving a parameter identifying a boundary shape of a region of interest of the registered first image or the registered second image; generating a first cutting image of the registered first image and a second cutting image of the registered second image based on the parameters of the boundary shape; and performing second image registration to register the first cropped image and the second cropped image.

Description

Technical Field

[0001] This specification relates to methods and systems for performing image registration on multiple images of a subject. The methods and systems can be used to register medical images, such as images of consecutive sections of a tissue sample obtained from an animal subject. However, the methods and systems can also be used to register other types of images of a subject. Background Art

[0002] Image registration can be used to superimpose two or more images of the same subject taken at different times, from different viewpoints, and / or by different sensors. In the field of medical imaging, multiple images can be generated from a tissue sample of a subject (e.g., a human) to obtain complementary information about the biology or medicine of the subject. For example, a first image can be an immunohistochemistry (IHC) image of a first section of a tissue sample, while a second image can be an IHC image of a second section of the tissue sample. In some cases, the second section may be consecutive with the first section (i.e., in the tissue sample, the second section is parallel and adjacent to the first section). For example, the first image may be an IHC image stained with CD8 (cluster of differentiation 8), while the second IHC image may be an IHC image stained with pan-cytokeratin (Pan-CK). The CD8 IHC image and the Pan-CK IHC image can be used to classify a patient into a predefined immunophenotype.

[0003] To facilitate the analysis of the images, it may be necessary to match these images pixel-to-pixel through the process of image registration. Image registration can allow subsequent performance of various image analysis techniques on the registered images, including but not limited to virtual staining, semantic image segmentation, cancer immunophenotype analysis, or creation of a patient model for simulation.

[0004] Currently, human experts such as pathologists can analyze the first image and the second image and attempt to perform the registration of these images manually, but this can be a very time-consuming process and requires a high level of skill. Alternatively, known computer-implemented registration methods can be used, but they may result in poor image alignment. Existing image registration methods may cause misalignment of images that represent the same subject but are captured by different imaging modalities or are different at the local level. For example, existing image registration methods do not work well for IHC images of consecutive sections of a tumor biopsy sample. Therefore, there is a need to provide an improved method for performing image registration on multiple images of a subject, so that the time consumption can be reduced and improved alignment can be provided. Summary of the Invention

[0005] According to a first aspect of the present disclosure, there is provided a computer-implemented method for performing image registration on a plurality of images of a subject, the method comprising: receiving a first image of the subject and a second image of the subject; performing a first image registration to register the first image and the second image; receiving parameters identifying a boundary shape of a region of interest of the registered first image or the registered second image; generating a first cropped image of the registered first image and a second cropped image of the registered second image based on the parameters of the boundary shape; and performing a second image registration to register the first cropped image and the second cropped image.

[0006] Performing the first image registration may include: segmenting the first image to generate a first segmented image, and segmenting the second image to generate a second segmented image; generating a first distance map based on the first segmented image, and generating a second distance map based on the second segmented image; registering the first distance map and the second distance map; and registering the first image and the second image based on the registration of the first distance map and the second distance map.

[0007] The first segmented image and the second segmented image may be binary images.

[0008] The first image registration may include a rigid body transformation.

[0009] Performing the second image registration may include performing normalized cross-correlation based on the first cropped image and the second cropped image.

[0010] Performing the second image registration may further include transforming the first cropped image and the second cropped image from the spatial domain to the frequency domain using the fast Fourier transform (FFT), wherein the normalized cross-correlation may be performed using the first cropped image in the frequency domain and the second cropped image in the frequency domain.

[0011] The subject may be an organism, and the plurality of images may be a plurality of images of tissues of the organism.

[0012] The first image may be an image of a first slice of tissue of the organism, and the second image may be an image of a second slice of tissue of the organism.

[0013] The first image registration may be performed based on the tissue-level morphology of the first image and the second image.

[0014] The second image registration may be performed based on the cell-level morphology of the first cropped image and the second cropped image.

[0015] At least one of the first image or the second image may be an immunohistochemical image or a hematoxylin and eosin stained image.

[0016] At least one of the following: the first cropped image may have a higher image resolution than the first image; or the second cropped image may have a higher image resolution than the second image.

[0017] The method may further include outputting the registered first cropped image and the second cropped image for display.

[0018] According to a second aspect of the present disclosure, there is provided a computer-implemented method for performing image registration on a plurality of images of a subject, the method including: receiving a first image of the subject and a second image of the subject; and performing a first image registration to register the first image and the second image; wherein performing the first image registration may include: segmenting the first image to generate a first segmented image, and segmenting the second image to generate a second segmented image; generating a first distance map based on the first segmented image, and generating a second distance map based on the second segmented image; registering the first distance map and the second distance map; and registering the first image and the second image based on the registration of the first distance map and the second distance map.

[0019] The method may further include receiving parameters identifying the boundary shape of an area of interest of the registered first image or the registered second image; generating a first cropped image of the registered first image and a second cropped image of the registered second image based on the parameters of the boundary shape; and performing a second image registration to register the first cropped image and the second cropped image.

[0020] The first segmented image and the second segmented image may be binary images.

[0021] The first image registration may include a rigid body transformation.

[0022] Performing the second image registration may include performing normalized cross-correlation based on the first cropped image and the second cropped image.

[0023] Performing the second image registration may further include transforming the first cropped image and the second cropped image from the spatial domain to the frequency domain using a fast Fourier transform (FFT), wherein the normalized cross-correlation may be performed using the first cropped image in the frequency domain and the second cropped image in the frequency domain.

[0024] The subject may be an organism, and the plurality of images may be a plurality of images of tissues of the organism.

[0025] The first image may be an image of a first section of a tissue of the organism, and the second image may be an image of a second section of the tissue of the organism.

[0026] The first image registration may be performed based on the tissue-level morphology of the first image and the second image.

[0027] The second image registration may be performed based on the cellular level morphology of the first cropped image and the second cropped image.

[0028] At least one of the first image or the second image may be an immunohistochemistry image or a hematoxylin and eosin staining image.

[0029] At least one of the following: the first cropped image may have a higher image resolution than the first image; or the second cropped image may have a higher image resolution than the second image.

[0030] The method may further include outputting the registered first cropped image and second cropped image for display.

[0031] According to a third aspect of the present disclosure, there is provided a computer program product comprising computer readable code, which, when executed by a computing system, causes the computing system to perform a method according to any of the aforementioned methods.

[0032] According to a fourth aspect of the present disclosure, there is provided a system comprising one or more processors and a memory storing computer-readable instructions which, when executed by the one or more processors, cause the system to perform any of the aforementioned methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Embodiments will now be described, by way of non-limiting examples, with reference to the accompanying drawings, in which:

[0034] Figure 1 A schematic overview of an example known method for performing image registration is shown;

[0035] Figure 2 Shows from Figure 1 The image extracted from the image;

[0036] Figure 3 shows a schematic overview of an example method for performing image registration on multiple images of a subject in accordance with aspects of the present disclosure;

[0037] Figure 4 A flow chart illustrating an example method of performing image registration on multiple images of a subject according to aspects of the present disclosure;

[0038] Figure 5 A flow chart illustrating another example method of performing image registration on multiple images of a subject according to aspects of the present disclosure; and

[0039] Figure 6 Schematic examples of systems / apparatus for performing any of the methods described herein are shown. DETAILED DESCRIPTION

[0040] As discussed above, known computer-implemented registration methods may lead to poor image alignment, especially for medical images obtained from tissue samples. For example, existing registration methods do not work well for IHC images of consecutive sections from tumor biopsy samples.

[0041] Figure 1 A schematic overview of an example known method for performing image registration is shown. Figure 1 A first IHC image 102 and a second IHC image 104 prior to image registration by a known registration method are shown. The first IHC image 102 in this example is an IHC image stained with CD8, while the second IHC image 104 is an IHC image stained with Pan-CK.

[0042] The first IHC image 102 and the second IHC image 104 are consecutive sections of a tumor biopsy sample. The first IHC image 102 in this example includes at least an image representation of a first tissue structure 106a and an image representation of a second tissue structure 108a. As an image of a consecutive section of a tumor biopsy sample, the second IHC image 104 in this example also contains image representations of the first and second tissue structures. However, as can be seen Figure 1 from it, the image representation of the first tissue structure 106b and the image representation of the second tissue structure 108b in the second IHC image 104 are both visually different from the corresponding image representations of the first tissue structure 106a and the second tissue structure 108a in the first IHC image 102. For example, the image representation of the first tissue structure 106b and the image representation of the second tissue structure 108b shown in the second IHC image 104 are located at different positions relative to the image representation of the first tissue structure 106a and the image representation of the second tissue structure 108a in the first IHC image 102. In addition, although the tissue structures of the first IHC image 102 and the second image 104 may appear similar at the scale of the global tissue level, the similarity may not be obvious at the local (e.g., cellular level) scale.

[0043] For a variety of different reasons, the image representation of the first tissue structure 106a and the image representation of the second tissue structure 108a in the first IHC image 102 may be different from the image representation of the first tissue structure 106b and the image representation of the second tissue structure 108b in the second IHC image 104. For example, using different staining techniques (CD8 vs. Pan-CK) to generate the first IHC image 102 and the second IHC image 104 may cause some differences. In addition, when the first IHC image 102 and the second IHC image 104 are obtained from different sections of the same tissue sample, since each section contains different cells, each section will be similar to the previous or next consecutive section, but not identical.

[0044] Thus, consecutive slices may have very similar morphology at the tissue level, but their morphology may be less similar at the cellular level. Figure 2 This is illustrated, showing a first scaled portion 202 of a first IHC image 102 and a second scaled portion 204 of a second IHC image 104. The first scaled portion 202 contains a part of the image representation of a second tissue structure 108a obtained from the first IHC image 102, while the second scaled portion 204 contains a part of the image representation of a second tissue structure 108b obtained from the second IHC image 104. Although the first scaled portion 202 and the second scaled portion 204 appear similar at the global level (e.g., the outlines of the tissue structures are similar), upon closer inspection, they are different at the cellular level (indicated by the dots in the first scaled portion 202 and the second scaled portion 204). This may be due to differences in cell count and cell morphology between the tissue sample slices used for the first IHC image 102 and the tissue sample slices used for the second IHC image 104.

[0045] Known computer-implemented image registration methods for registering the first IHC image 102 and the second IHC image 104 may produce suboptimal registration, as Figure 1 shown in the synthetic IHC image 110. After registering the first IHC image 102 and the second IHC image 104 using a known image registration method, a synthetic IHC image 110 is generated. The synthetic IHC image 110 can be generated by aligning the first IHC image 102 and the second IHC image 104 using a known image registration method and then superimposing them.

[0046] The synthetic IHC image 110 shows the image representations 106a, 106b, 108a, 108b of the first tissue structure and the second tissue structure previously discussed with respect to the first IHC image 103 and the second IHC image 104. However, it can be seen from the synthetic IHC image 110 that the image representations 106a, 106b, 108a, 108b of the first tissue structure and the second tissue structure are poorly aligned, and thus the first IHC image 102 and the second IHC image 104 are poorly aligned overall. This poor alignment may be due to the inability of the known registration method to successfully handle the differences between the first IHC image 102 and the second IHC image 104 at the cellular level. This poor alignment makes it difficult for medical professionals to analyze the synthetic IHC image 110.

[0047] Although Figure 1The discussion is about IHC images, but the discussion can equally apply to other types of medical images, or non-medical image types. For example, at least one of the first IHC image 102 or the second IHC image 104 can be a different type of IHC image, or can be not an IHC image, but a hematoxylin and eosin (H&E) stained image. In other examples, one of the images can be an image obtained by magnetic resonance imaging (MRI), and the other image can be an image obtained by computed tomography (CT) scan.

[0048] Aspects of the present disclosure can provide an improved image registration method. Aspects of the present disclosure can be particularly relevant to registering multiple images that have greater similarity at a global scale compared to a local scale.

[0049] Figure 3 A schematic overview of a computer-implemented method 300 for performing image registration on multiple images of a subject in accordance with aspects of the present invention is shown. The method 300 can be executed by a computing system, such as the system described with respect to Figure 6 the system.

[0050] Receive a first image 302 of a subject and a second image 304 of the subject. Figure 3 The first image 302 and the second image 304 shown in are the same as the first IHC image 102 and the second IHC image 104 discussed with respect to Figure 1 However, in other examples, different types of images can be received, such as H&E stained images.

[0051] In this example, the subject is a human patient, and the first image 302 and the second image 304 are images of consecutive sections of a tumor biopsy sample taken from the patient. The first image 302 and the second image 304 contain corresponding features, such as tissue structures to be analyzed after image registration. However, the techniques described herein apply to many other scenarios and to a wide range of organisms, such as plants, animals, fungi, bacteria, and viruses.

[0052] Perform a first image registration process using the first image 302 and the second image 304 to register (i.e., align) the first image 302 and the second image 304. Image registration can be used to estimate the transformation between the first image 302 and the second image 304. The estimated transformation can be applied to the first image 302 or the second image 304 to align each image.

[0053] The first image registration is global registration and aligns the first image 302 and the second image 304 based on the tissue-level morphology of the first image 302 and the second image 304. The first image registration can be performed using any suitable image registration method disclosed in the following literature: for example, Barbara Zitová, Jan Flusser, Image registration methods: a survey [Image registration methods: survey], Image and Vision Computing [Image and Vision Computing], Vol. 21, No. 11, 2003, pp. 977-1000, ISSN 0262-8856, https: / / doi.org / 10.1016 / S0262-8856(03)00137-9.

[0054] In Figure 3 the example of, performing the first image registration includes segmenting the first image 302 to generate a first segmented image 306, and segmenting the second image 304 to generate a second segmented image 308. The segmentation can be performed using a suitable threshold segmentation algorithm known in the art, but other types of segmentation algorithms known in the art can also be used instead. The segmentation can enable the creation of a binary image.

[0055] An example of a suitable segmentation algorithm is the Otsu algorithm, as disclosed, for example, in N. Otsu, "A Threshold Selection Method from Gray-Level Histograms", IEEE Transactions on Systems, Man, and Cybernetics, Vol. 9, No. 1, pp. 62 - 66, January 1979, doi:10.1109 / TSMC.1979.4310076 (https: / / ieeexplore.ieee.org / document / 4310076). Another example of a segmentation algorithm can use a deep learning-based method, such as using the UNET convolutional neural network, as disclosed, for example, in: Ronneberger, O., Fischer, P., Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation, in: Navab, N., Hornegger, J., Wells, W., Frangi, A. (Eds.), Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015. MICCAI 2015.

[0056] Then, the first segmented image 306 (which can be a binary image) is converted into a first distance map 310, and the second segmented image 308 (which can also be a binary image) is converted into a second distance map 312. The distance map is also known as distance transform. A suitable algorithm is used to perform the conversion to the distance map. For example, as disclosed in the following reference, C.R. Maurer, Jr., R. Qi, and V. Raghavan, "A Linear Time Algorithm for Computing Exact Euclidean Distance Transforms of Binary Images in Arbitrary Dimensions", IEEE-Transactions on Pattern Analysis and Machine Intelligence, 25(2):265-270, 2003.

[0057] The value of each pixel in the distance maps 310, 312 corresponds to the distance to the nearest contour point in the corresponding segmented images 306, 308. Binary images do not provide strong gradients, which is not helpful for optimization when aligning images during image registration. The distance map representation provides a better gradient signal and thus can bring improved optimization results when aligning images.

[0058] Then, the first image 302 can be registered to the second image 304 by performing image registration on the first distance map 310 and the second distance map 312 to obtain an estimated transform between the first distance map 310 and the second distance map 312. The first image 302 can be registered to the second image 304 by applying the estimated transform.

[0059] The first image registration can include rigid body image registration involving rigid body transformation (i.e., transformation by rotation and / or translation). Based on the assumption that the tissue sections shown in the first image 302 and the second image 304 are very similar in morphology at the tissue level, the rigid body transformation should be sufficient for the first image registration. However, in other examples, different transformations, such as affine transformation or non-rigid transformation, can be used.

[0060] In this example, the gradient descent algorithm is used for the first image registration, which finds the optimal parameters given a rigid body transformation. However, in other examples, any suitable algorithm known in the art can be used to perform the first image registration. To use the gradient descent algorithm, a cost function is defined, which is the value to be optimized. In this case, the mean squared error is used as the cost function, but in other examples, other metrics such as mutual information or L1 difference can be used.

[0061] Figure 3 A composite image 314 generated from the registered first image 302 and second image 304 is shown. The composite image 314 can be generated by registering the first image 302 and the second image 304 using the first image registration and then superimposing them. Figure 3 The shown composite image 314 is compared with Figure 1 the shown composite IHC image 110. It can be seen that, compared with previous image registration methods (e.g., previous image registration methods that do not involve segmenting the images and converting them into distance maps before performing the registration), the global registration method described so far may have provided improved image registration.

[0062] Image 316 shows a portion of the composite image 314, which generally corresponds to the second tissue structure previously discussed with respect to Figure 1 and Figure 2 . It can be seen from image 316 that the first image 302 and the second image 304 have been well aligned.

[0063] Note that in some examples, the composite image 314 and / or image 316 are not actually generated. They are provided in Figure 3 for illustrative purposes to show the degree of alignment of the first image 302 and the second image 304 by the first image registration.

[0064] The first global image registration can allow the alignment of the first image 302 and the second image 304 to a sufficient accuracy. However, in some examples, further registration accuracy of the first image 302 and the second image 304 can also be achieved by performing a second local image registration, as discussed below.

[0065] Once the first image 302 and the second image 304 are aligned using the first registration, a region of interest (ROI) 318 is identified in at least one of the first image 302 or the second image 304, or in the composite image 314 in some examples. The ROI can be identified by a human user such as a pathologist, but in other examples, it can be identified, for example, through a computer-implemented process using a machine learning algorithm trained to identify specific ROIs.

[0066] The ROI can include features in the first image 302 and / or the second image 304 that a pathologist is interested in further analyzing.

[0067] Receive parameters that identify the boundary shape 320 of the ROI of the first image 302 or the second image 304. These parameters can include the coordinates of the boundary shape 320, such as the coordinates corresponding to the position of the boundary shape in the first image 302 or the second image 304. These parameters may be determined based on user-provided input. For example, the user can use a user input interface (such as a touch screen or a mouse) coupled to the computing system described with respect to Figure 6 to draw the boundary shape 320 on a display version of the first image 302, the second image 304, or the composite image 314.

[0068] The parameters (such as coordinates) of the boundary shape 320 can be used to generate a pair of cropped images, specifically, a first cropped image 322 of the first image 302 and a second cropped image 324 of the second image 304. More specifically, the parameters and transformations estimated by the first image registration process are used to identify the boundary shape 320 (and thus, the ROI 318) in both the registered first image 302 and the second image 304. For example, if the parameters are associated with the boundary shape 320 identified at the first image 302, the corresponding boundary shape 320 can be determined for the second image 304 by applying the transformation to the parameters. Similarly, if the parameters are associated with the boundary shape 320 identified at the second image 304, the corresponding boundary shape 320 can be determined for the first image 302 by applying the transformation to the parameters.

[0069] The first cropped image 322 can be cropped from the first image 302 based on the parameters to at least include the ROI 318 identified by the boundary shape 320. The first cropped image 322 can be generated to include a margin around the boundary shape 320 such that the first cropped image 322 is larger than and includes the ROI 318 and the boundary shape 320. For example, the width and height of the first cropped image 322 can be larger than the maximum width and height of the ROI 318 in the first image 302, and in some examples, can be larger than the maximum width and height by a predetermined amount.

[0070] In a manner similar to the first cropped image 322, a second cropped image 324 can be cropped from the second image 304 based on parameters to at least include the ROI 318 identified by the boundary shape 320. The second cropped image 324 can also be generated to include a margin around the boundary shape 320 such that the second cropped image 324 is larger than and includes the ROI 318 and the boundary shape 320. For example, the width and height of the second cropped image 324 can also be greater than the maximum width and height of the ROI 318 in the second image 304 and, in some examples, can be greater than the maximum width and height by a predetermined amount.

[0071] Figure 3 The ROI 318 and the boundary shape 320 located on the second cropped image 324 are shown.

[0072] Then, a second local image registration process is performed to register the first cropped image 322 and the second cropped image 324. The second image registration can involve registering the first cropped image 322 and the second cropped image 324 at a finer level than the first registration from the first image 302 to the second image 304. That is, the second image registration may involve registering the first cropped image 322 and the second cropped image 324 based on the cellular-level morphology shown in the images rather than the tissue-level morphology in the images.

[0073] Performing the second image registration can include performing normalized cross-correlation (NCC) to register the first cropped image 322 and the second cropped image 324. As an example, the NCC method disclosed in the following document can be used: D. Padfield, “Masked object registration in the Fourier domain,” IEEE Transactions on Image Processing (2012). DOI: 10.1109 / TIP.2011.2181402, which is incorporated herein by reference. The advantage of NCC is that it has a very fast computational speed in the Fourier domain. However, in other examples, different suitable registration algorithms can be used to perform the second image registration, such as the algorithms described for the first image registration.

[0074] To perform NCC, the first cropped image 322 and the second cropped image 324 can be transformed from the spatial domain to the frequency domain using the fast Fourier transform (FFT), and then NCC is performed on the transformed first cropped image 322 and second cropped image 324 in the frequency domain (Fourier). The translation transformation for aligning the first cropped image 322 and the second cropped image 324 can be determined based on the result of the NCC.

[0075] Then, a translation transformation can be used to register the first cropped image 322 and the second cropped image 324. Additionally or alternatively, a translation transformation can be used to register the first image 302 and the second image 304.

[0076] The registered first cropped image 322 and second cropped image 324 can be output for display, for example, by a display coupled to the computing system described with respect to Figure 6 the described computing system.

[0077] As Figure 3 shown, a composite image 326 can be generated based on the registered first cropped image 322 and second cropped image 324 and output for display. The composite image 326 can include the registered first cropped image 322 superimposed on the registered second cropped image 324. Figure 3 Also shown is an ROI 318 highlighted on the composite image 326, however this is optional.

[0078] In some embodiments, the cropped images 322, 324 are resized / rescaled to a predetermined size before performing the second image registration. Interpolation techniques can be used to perform the rescaling to fill in any missing pixel data.

[0079] The first cropped image 322 can have a higher resolution than the first image 302. For example, the first image 302 can be a relatively low-resolution version of a certain image, while the first cropped image 322 can be a cropped portion of a relatively high-resolution version of the same image. Similarly, the second cropped image 324 can have a higher resolution than the second image 304. By performing global registration on the larger but relatively lower-resolution image and then local registration on the relatively smaller but higher-resolution cropped images, this can improve the efficiency of the overall image registration method.

[0080] Figure 4 A flowchart of an example method 400 for performing image registration on multiple images of a subject in accordance with aspects of the present invention is shown. The method 400 can be performed by a computing system, such as the system described with respect to Figure 6 The method 400 can share one or more aspects with the methods described previously with respect to Figure 3 the described methods.

[0081] In step 402, a first image 302 of a subject and a second image 304 of the subject are received. As an example, the subject can be an organism, such as an animal, particularly a human. The first image 302 can be an image of a first tissue section of the organism, and the second image 304 can be an image of a second tissue section. The first section and the second section can be consecutive or approximately consecutive sections of the same tissue. The first image 302 and the second image 304 can be obtained using different image preparation methods. For example, at least one of the first image 302 or the second image 304 can be an immunohistochemistry (IHC) image or a hematoxylin and eosin (H&E) stained image.

[0082] In step 404, a first image registration process is performed to register the first image 302 and the second image 304. The first image registration can be global registration and can be performed based on the tissue-level morphology of the first image 302 and the second image 304. The first image registration can involve a rigid body transformation, but in other examples, an affine or non-rigid body transformation can be used.

[0083] The first image registration can be performed using a gradient descent algorithm, where the mean squared error is selected as the cost function to be optimized. However, in other examples, different cost functions (such as mutual information or L1 difference) can be used, and / or an algorithm different from gradient descent can be used, as previously discussed.

[0084] In step 406, parameters identifying the boundary shape 320 of the region of interest 318 of the first image 302 or the second image 304 are received. These parameters can include coordinates and can be input by a user, for example, by drawing the boundary shape 320 on a representation of the first image or the second image output on a display.

[0085] In step 408, a first cropped image 322 of the first image 302 and a second cropped image 324 of the second image 304 are generated based on the parameters of the boundary shape 320. Both the first image 302 and the second image 304 are cropped to include the ROI 318 identified by the boundary shape 320. The first image 302 and the second image 304 have been registered in step 404 using a first image registration process. Thus, based on the rigid body transformation estimated during the first image registration, the position of the boundary shape 320 in the first image 302 will have a corresponding relationship with the position of the boundary shape 320 in the second image 304. Therefore, the parameters (e.g., coordinates) of the boundary shape 320 can be applied to both the first image 302 and the second image 304 to produce the first cropped image 322 and the second cropped image 324. This step limits the regions of the first image 302 and the second image 304 that undergo the second image registration process. The first cropped image 322 and / or the second cropped image 324 can be generated to include a margin around the boundary shape 320. The first cropped image 322 and / or the second cropped image 324 can have a higher resolution than the corresponding first image 302 or second image 304.

[0086] In step 410, a second image registration process is performed to register the first cropped image 322 and the second cropped image 324. The second image registration can involve registering the first cropped image 322 and the second cropped image 324 based on cell-level morphology rather than tissue-level morphology.

[0087] Performing the second image registration can include performing normalized cross-correlation (NCC) on the first cropped image 322 and the second cropped image 324 to register the images, as previously discussed, but in other examples, different suitable registration algorithms can be used to perform the second image registration.

[0088] Once the first cropped image 322 and the second cropped image 324 are registered, the registered first cropped image 322 and second cropped image 324 can be output for display, such as on a computer monitor. A composite image 326 including the registered first cropped image 322 and second cropped image 324 (or at least a portion of the registered first cropped image and second cropped image) can be output for display.

[0089] Figure 5 A flowchart of another example method 500 for performing image registration on multiple images of a subject is shown. The method 500 can be performed by a computing system, such as the system described with respect to Figure 6 the system. Figure 5 The method 500 shown is similar to the method shown in Figure 5 but has additional steps.

[0090] In step 502, a first image 302 of the subject and a second image 304 of the subject are received. In some examples, the manner employed is similar to that described in step 402 with respect to Figure 4 previously.

[0091] In step 504, a first image registration process is performed to register the first image 302 and the second image 304. This can be done in a manner similar to that described in step 404 with respect to Figure 4 previously, however, step 504 includes a number of additional steps 506, 508, 510.

[0092] In step 506, the first image 302 is segmented to generate a first segmented image 306, and the second image 304 is segmented to generate a second segmented image 308. This can be performed, for example, in the manner described previously with respect to Figure 3 that.

[0093] In step 508, a first distance map 310 is generated based on the first segmented image 306, and a second distance map 312 is generated based on the second segmented image 308. This can be performed, for example, in the manner described previously with respect to Figure 3 that.

[0094] In step 510, the first distance map 310 is registered to the second distance map 312. This can be performed using a registration algorithm, for example, as discussed previously with respect to Figure 3 that. Then, the first image 302 and the second image 304 can be registered based on the registration of the first distance map 310 and the second distance map 312. For example, the transformation estimated based on the registration of the first distance map 310 and the second distance map 312 can be applied to the first image 302 and / or the second image 304 in order to align the first image 302 and the second image 304.

[0095] In step 512, parameters identifying the boundary shape 320 of the region of interest 318 of the first image 302 or the second image 304 are received. Step 512 can correspond to one or more aspects of step 406, which was discussed previously and will not be repeated here for the sake of brevity.

[0096] In step 514, a first cropped image 322 of the first image 302 and a second cropped image 324 of the second image 304 are generated based on the parameters of the boundary shape 320. Step 514 can correspond to one or more aspects of step 408, which was discussed previously and will not be repeated here for the sake of brevity.

[0097] In step 516, a second image registration process is performed to register the first cropped image 322 and the second cropped image 324. Step 516 may correspond to one or more aspects of step 410, which has been discussed previously and will not be repeated for the sake of brevity.

[0098] Once the first cropped image 322 and the second cropped image 324 are registered, the registered first cropped image 322 and second cropped image 324 can be output for display, such as on a computer monitor. A composite image 326 including the registered first cropped image 322 and second cropped image 324 (or at least a portion of the registered first and second cropped images) can be output for display.

[0099] Figure 6 A schematic example of a system / device 600 for performing any of the methods described herein is shown. The illustrated system / device is an example of a computing device. Those skilled in the art will understand that other types of computing devices / systems may alternatively be used to implement the methods described herein, such as a distributed computing system.

[0100] The device (or system) 600 includes one or more processors 602. The one or more processors control the operation of the other components of the system / device 600. For example, the one or more processors 602 may include a general-purpose processor. The one or more processors 602 may be a single-core device or a multi-core device. The one or more processors 602 may include a central processing unit (CPU) or a graphics processing unit (GPU). Alternatively, the one or more processors 702 may include specialized processing hardware, such as a RISC processor or programmable hardware with embedded firmware. Multiple processors may be included.

[0101] The system / device includes working or volatile memory 604. The one or more processors may access the volatile memory 704 to process data and may control the storage of data in the memory. The volatile memory 604 may include any type of RAM, such as static RAM (SRAM), dynamic RAM (DRAM), or it may include flash memory, such as an SD card.

[0102] The system / device includes non-volatile memory 606. The non-volatile memory 606 stores an operation instruction set 608 for controlling the operation of the processor 602 in the form of computer-readable instructions. The non-volatile memory 606 may be any type of memory, such as read-only memory (ROM), flash memory, or magnetic drive memory.

[0103] One or more processors 602 are configured to execute operation instructions 608 to cause the system / device to perform any of the methods described herein. The operation instructions 608 may include code related to the hardware components of the system / device 600 (i.e., drivers), as well as code related to the basic operations of the system / device 600. Generally, one or more processors 602 execute one or more of the operation instructions 608 permanently or semi-permanently stored in the non-volatile memory 606, and use the volatile memory 604 to temporarily store data generated during the execution of the operation instructions 608.

[0104] Embodiments of the methods described herein may be implemented in digital electronic circuitry, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These may include computer program products (e.g., software stored on, for example, a disk, an optical disk, a memory, a programmable logic device) that include computer readable instructions that, when executed by a computer, cause the computer to perform one or more of the methods described herein, as Figure 6 described, such that the computer performs one or more of the methods described herein.

[0105] Any system feature described herein may also be provided as a method feature, and vice versa. As used herein, a means-plus-function feature may alternatively be expressed in terms of its corresponding structure. Specifically, method aspects may be applied to system aspects, and vice versa.

[0106] Furthermore, any, some, and / or all features in one aspect may be applied in any suitable combination to any, some, and / or all features in any other aspect. It should also be understood that particular combinations of the various features described and defined in any aspect of the present invention may be implemented and / or provided and / or used independently.

[0107] Although several embodiments have been shown and described, those skilled in the art should understand that changes may be made in these embodiments without departing from the principles of the disclosure, the scope of which is defined in the claims and their equivalents.

[0108] The terms “drug” or “medicament” are used synonymously herein and describe a pharmaceutical preparation that contains one or more active pharmaceutical ingredients or pharmaceutically acceptable salts or solvates thereof and optionally a pharmaceutically acceptable carrier. In the broadest sense, an active pharmaceutical ingredient (“API”) is a chemical structure that has a biological effect on a human or an animal. In pharmacology, a drug or medicament is used to treat, cure, prevent, or diagnose a disease or to otherwise enhance physical or mental health. A drug or medicament may be used for a limited duration or regularly for a chronic disorder.

[0109] As described below, a medicament or pharmaceutical agent may include at least one API or a combination thereof in different types of formulations for treating one or more diseases. Examples of APIs may include small molecules (having a molecular weight of 500 Da or less); polypeptides, peptides, and proteins (e.g., hormones, growth factors, antibodies, antibody fragments, and enzymes); carbohydrates and polysaccharides; and nucleic acids, double-stranded or single-stranded DNA (including naked and cDNA), RNA, antisense nucleic acids (such as antisense DNA and RNA), small interfering RNA (siRNA), ribozymes, genes, and oligonucleotides. Nucleic acids can be incorporated into molecular delivery systems (such as vectors, plasmids, or liposomes). Mixtures of one or more drugs are also contemplated.

[0110] A medicament or pharmaceutical agent can be contained in a primary package or "drug container" suitable for use with a drug delivery device. The drug container can be, for example, a cartridge, syringe barrel, reservoir, or other rigid or flexible vessel configured to provide a suitable chamber for storing one or more drugs (e.g., short-term or long-term storage). For example, in some cases, the chamber can be designed to store the drug for at least one day (e.g., 1 day to at least 30 days). In some cases, the chamber can be designed to store the drug for about 1 month to about 2 years. Storage can be at room temperature (e.g., about 20 °C) or refrigerated temperature (e.g., about -4 °C to about 4 °C). In some cases, the drug container can be or can include a dual-chamber cartridge configured to separately store two or more components of a pharmaceutical formulation to be administered (e.g., an API and a diluent, or two different drugs), with one component stored in each chamber. In such a case, the two chambers of the dual-chamber cartridge can be configured to allow mixing between the two or more components before and / or during dispensing into a human or animal body. For example, the two chambers can be configured such that they are in fluid communication with each other (e.g., through a conduit between the two chambers) and allow the user to mix the two components when needed before dispensing. Alternatively or additionally, the two chambers can be configured to allow mixing when the components are dispensed into a human or animal body.

[0111] The medicaments or agents comprised in a drug delivery device as described herein can be used for treating and / or preventing many different types of medical disorders. Examples of disorders include, for example, diabetes or diabetes-related complications (such as diabetic retinopathy), thromboembolic disorders (such as deep vein or pulmonary thromboembolism). Further examples of disorders are acute coronary syndrome (ACS), angina pectoris, myocardial infarction, tumors, macular degeneration, inflammation, hay fever, atherosclerosis and / or rheumatoid arthritis. Examples of APIs and medicaments are those described in the following compendia: such as the Rote Liste 2014 (for example but not limited to, main group 12 (antidiabetic medicaments) or 86 (oncology medicaments)), and the Merck Index (15th edition).

[0112] Examples of APIs for treating and / or preventing type 1 or type 2 diabetes or diabetes-related complications associated with type 1 or type 2 diabetes include insulin (such as human insulin, or human insulin analogues or derivatives); glucagon-like peptide (GLP-1), GLP-1 analogues or GLP-1 receptor agonists, or their analogues or derivatives; dipeptidyl peptidase-4 (DPP4) inhibitors, or their pharmaceutically acceptable salts or solvates; or any mixture of the above. As used herein, the terms “analogue” and “derivative” refer to a polypeptide having a molecular structure that can be formally derived from the structure of a naturally occurring peptide (such as the structure of human insulin) by deletion and / or exchange of at least one amino acid residue present in the naturally occurring peptide and / or by addition of at least one amino acid residue. The added and / or exchanged amino acid residues can be encoded amino acid residues or other naturally occurring residues or purely synthetic amino acid residues. Insulin analogues are also referred to as “insulin receptor ligands”. In particular, the term “derivative” refers to a polypeptide having a molecular structure that can be formally derived from the structure of a naturally occurring peptide (such as the structure of human insulin), wherein one or more organic substituents (such as fatty acids) are bound to one or more amino acids. Optionally, one or more amino acids present in the naturally occurring peptide may have been deleted and / or replaced by other amino acids (including non-encoded amino acids), or amino acids (including non-encoded amino acids) have been added to the naturally occurring peptide.

[0113] Examples of insulin analogs are Gly(A21), Arg(B31), Arg(B32) human insulin (insulin glargine); Lys(B3), Glu(B29) human insulin (insulin glulisine); Lys(B28), Pro(B29) human insulin (insulin lispro); Asp(B28) human insulin (insulin aspart); human insulin in which proline at position B28 is replaced by Asp, Lys, Leu, Val or Ala and in which Lys at position B29 can be replaced by Pro; Ala(B26) human insulin; Des(B28 - B30) human insulin; Des(B27) human insulin and Des(B30) human insulin.

[0114] Examples of insulin derivatives are, for example, B29 - N - myristoyl - des(B30) human insulin, Lys(B29)(N - tetradecanoyl) - des(B30) human insulin (insulin detemir, ); B29 - N - palmitoyl - des(B30) human insulin; B29 - N - myristoyl human insulin; B29 - N - palmitoyl human insulin; B28 - N - myristoyl LysB28ProB29 human insulin; B28 - N - palmitoyl - LysB28ProB29 human insulin; B30 - N - myristoyl - ThrB29LysB30 human insulin; B30 - N - palmitoyl - ThrB29LysB30 human insulin; B29 - N - (N - palmitoyl - γ - glutamyl) - des(B30) human insulin, B29 - N - ω - carboxypentadecanoyl - γ - L - glutamyl - des(B30) human insulin (insulin degludec, ); B29 - N - (N - lithocholyl - γ - glutamyl) - des(B30) human insulin; B29 - N - (ω - carboxyheptadecanoyl) - des(B30) human insulin and B29 - N - (ω - carboxyheptadecanoyl) human insulin.

[0115] Examples of GLP - 1, GLP - 1 analogs and GLP - 1 receptor agonists are, for example, lixisenatide exenatide (Exendin - 4, a 39 - amino acid peptide produced by the salivary gland of the Gila monster), liraglutide semaglutide, taspoglutide, albiglutide dulaglutide rExendin-4, CJC-1134-PC, PB-1023, TTP-054, Langlenatide / HM-11260C (Efpeglenatide), HM-15211, CM-3, GLP-1Eligen, ORMD-0901, NN-9423, NN-9709, NN-9924, NN-9926, NN-9927, Nodexen, Viador-GLP-1, CVX-096, ZYOG-1, ZYD-1, GSK-2374697, DA-3091, MAR-701, MAR709, ZP-2929, ZP-3022, ZP-DI-70, TT-401 (Pegapamodtide), BHM-034. MOD-6030, CAM-2036, DA-15864, ARI-2651, ARI-2255, Tirzepatide (LY3298176), Bamadutide (SAR425899), Exenatide-XTEN, and Glucagon-Xten.

[0116] Examples of oligonucleotides are, for example: Mipomersen sodium A cholesterol-reducing antisense therapeutic agent for treating familial hypercholesterolemia or RG012 for treating Alport syndrome.

[0117] Examples of DPP4 inhibitors are Linagliptin, Vildagliptin, Sitagliptin, Deglibitin, Saxagliptin, Berberine.

[0118] Examples of hormones include pituitary hormones or hypothalamic hormones or regulatory active peptides and their antagonists, such as gonadotropins (follicle-stimulating hormone, luteinizing hormone, chorionic gonadotropin, gonadotrophin), somatotropin (growth hormone), desmopressin, terlipressin, gonadorelin, triptorelin, leuprorelin, buserelin, nafarelin, and goserelin.

[0119] Examples of polysaccharides include glycosaminoglycans, hyaluronic acid, heparin, low molecular weight heparin or ultra-low molecular weight heparin or their derivatives, or sulfated polysaccharides (e.g., the polysulfated forms of the above polysaccharides), and / or their pharmaceutically acceptable salts. An example of a pharmaceutically acceptable salt of polysulfated low molecular weight heparin is Enoxaparin sodium. An example of a hyaluronic acid derivative is Hylan G-F 20 A sodium hyaluronate.

[0120] As used herein, the term "antibody" refers to an immunoglobulin molecule or an antigen-binding portion thereof. Examples of antigen-binding portions of immunoglobulin molecules include F(ab) and F(ab')2 fragments, which retain the ability to bind antigen. Antibodies can be polyclonal antibodies, monoclonal antibodies, recombinant antibodies, chimeric antibodies, deimmunized antibodies or humanized antibodies, fully human antibodies, non-human (e.g., murine) antibodies, or single-chain antibodies. In some embodiments, the antibody has effector function and can fix complement. In some embodiments, the antibody has reduced or no ability to bind to Fc receptors. For example, the antibody can be an isotype or subtype, an antibody fragment, or a mutant that does not support binding to Fc receptors, e.g., its Fc receptor-binding region has been mutagenized or deleted. The term "antibody" also includes antigen-binding molecules based on tetravalent bispecific tandem immunoglobulins (TBTIs) and / or dual variable domain antibody-like binding proteins having a cross-over binding domain orientation (CODV).

[0121] The term "fragment" or "antibody fragment" refers to a polypeptide (e.g., an antibody heavy chain and / or light chain polypeptide) derived from an antibody polypeptide molecule that does not contain the full-length antibody polypeptide but still contains at least a portion of the full-length antibody polypeptide capable of binding to an antigen. Antibody fragments can include cleaved portions of the full-length antibody polypeptide, but the term is not limited to such cleaved fragments. Antibody fragments useful in the present invention include, for example, Fab fragments, F(ab')2 fragments, scFv (single-chain Fv) fragments, linear antibodies, monospecific or multispecific antibody fragments (such as bispecific, trispecific, tetra-specific, and multispecific antibodies (e.g., diabodies, triabodies, tetra-bodies)), monovalent or multivalent antibody fragments (such as bivalent antibodies, trivalent antibodies, tetravalent antibodies, and multivalent antibodies), minibodies, chelating recombinant antibodies, triabodies or diabodies, intracellular antibodies, nanobodies, small modular immunopharmaceuticals (SMIPs), binding domain immunoglobulin fusion proteins, camelized antibodies, and antibodies containing VHHs. Additional examples of antigen-binding antibody fragments are known in the art.

[0122] The term "complementary determining region" or "CDR" refers to short polypeptide sequences within the variable regions of both heavy chain polypeptides and light chain polypeptides that are primarily responsible for mediating specific antigen recognition. The term "framework region" refers to the amino acid sequences within the variable regions of both heavy chain polypeptides and light chain polypeptides that are not CDR sequences and are primarily responsible for maintaining the correct positioning of the CDR sequences to allow antigen binding. Although framework regions typically do not directly participate in antigen binding as known in the art, certain residues within the framework regions of some antibodies can directly participate in antigen binding or can affect the ability of one or more amino acids in the CDRs to interact with an antigen.

[0123] Examples of antibodies are anti-PCSK-9 mAb (e.g., alirocumab), anti-IL-6 mAb (e.g., sarilumab), and anti-IL-4 mAb (e.g., dupilumab).

[0124] Also contemplated is the use of pharmaceutically acceptable salts of any API described herein in a drug or medicament in a drug delivery device. Pharmaceutically acceptable salts are, for example, acid addition salts and basic salts.

[0125] Those skilled in the art will understand that modifications (additions and / or removals) can be made to the different components, formulations, devices, methods, systems, and embodiments of the API described herein without departing from the full scope and spirit of the invention, and the invention encompasses such modifications and any and all equivalents thereof.

[0126] Example drug delivery devices can relate to needle-based injection systems as described in Table 1 of Section 5.2 of ISO 11608-1:2014(E). As described in ISO 11608-1:2014(E), needle-based injection systems can be broadly classified into multi-dose container systems and single-dose (partially or fully emptied) container systems. The container can be a replaceable container or an integral non-replaceable container.

[0127] As further described in ISO 11608-1:2014(E), multi-dose container systems can relate to needle-based injection devices having a replaceable container. In such a system, each container holds multiple doses, and the size of these doses can be fixed or variable (predetermined by the user). Another multi-dose container system can relate to needle-based injection devices having an integral non-replaceable container. In such a system, each container holds multiple doses, and the size of these doses can be fixed or variable (predetermined by the user).

[0128] As further described in ISO 11608-1:2014(E), single-dose container systems can relate to needle-based injection devices having a replaceable container. In one example of such a system, each container holds a single dose, where the entire deliverable volume is expelled (fully emptied). In additional examples, each container holds a single dose, where a portion of the deliverable volume is expelled (partially emptied). Also as described in ISO 11608-1:2014(E), single-dose container systems can relate to needle-based injection devices having an integral non-replaceable container. In one example of such a system, each container holds a single dose, where the entire deliverable volume is expelled (fully emptied). In additional examples, each container holds a single dose, where a portion of the deliverable volume is expelled (partially emptied).

Claims

1. A computer-implemented method (300, 400, 500) for performing image registration on multiple images of a subject, the method comprising: Receiving (402, 502) a first image (302) of the subject and a second image (304) of the subject; Performing (404, 504) a first image registration to register the first image and the second image; Receiving (406, 512) parameters identifying a boundary shape (320) of a region of interest (318) of the registered first image or the registered second image; Generating (408, 514) a first cropped image (322) of the registered first image and a second cropped image (324) of the registered second image based on the parameters of the boundary shape; And Performing (410, 516) a second image registration to register the first cropped image and the second cropped image, wherein performing the first image registration comprises: Segmenting (506) the first image to generate a first segmented image and segmenting the second image to generate a second segmented image; Generating (508) a first distance map based on the first segmented image and generating a second distance map based on the second segmented image; Registering (510) the first distance map and the second distance map; and Registering the first image and the second image based on the registration of the first distance map and the second distance map.

2. The method according to claim 1, wherein, The first segmented image and the second segmented image are binary images.

3. The method according to claim 1 or 2, wherein The first image registration includes a rigid body transformation.

4. The method according to any one of the preceding claims, wherein, Performing the second image registration comprises performing normalized cross-correlation based on the first cropped image and the second cropped image.

5. The method according to claim 4, wherein, Performing the second image registration further comprises transforming the first cropped image and the second cropped image from the spatial domain to the frequency domain using a fast Fourier transform (FFT), wherein the normalized cross-correlation is performed using the first cropped image in the frequency domain and the second cropped image in the frequency domain.

6. The method according to any one of the preceding claims, wherein, The subject is an organism, and the multiple images are multiple images of tissues of the organism.

7. The method according to claim 6, wherein, The first image is an image of a first slice of the tissue of the organism, and the second image is an image of a second slice of the tissue of the organism.

8. The method according to claim 6 or 7, wherein, The first image registration is performed based on the tissue-level morphology of the first image and the second image.

9. The method according to claim 6, 7 or 8, wherein The second image registration is performed based on the cell-level morphology of the first cropped image and the second cropped image.

10. As described in any of the preceding claims, wherein, At least one of the first image or the second image is an immunohistochemical image or a hematoxylin and eosin stained image.

11. The method according to any one of the preceding claims, wherein, At least one of the following: The first cropped image has a higher image resolution than the first image; or The second cropped image has a higher image resolution than the second image.

12. The method according to any one of the preceding claims, further comprising outputting the registered first cropped image and second cropped image for display.

13. A computer program product comprising computer-readable code that, when executed by a computing system, causes the computing system to perform the method according to any one of the preceding claims.

14. A system (600) includes one or more processors (602) and a memory (606) that stores computer-readable instructions (608) that, when executed by the one or more processors, cause the system to perform the method according to any one of claims 1 to 12.

Images