Method for performing image registration

Abstract

1. A computer-implemented method for performing image registration of multiple images of an object, the method comprising: receiving a first image of the object and a second image of the object; performing a first image registration to register the first image and the second image; receiving boundary shape parameters that identify a region of interest in the registered first image or the registered second image; generating a first cropped image of the registered first image and a second cropped image of the registered second image based on the boundary shape parameters; and performing a second image registration to register the first cropped image and the second cropped image.

Description

[Technical Field]

[0001] This specification relates to methods and systems for performing image registration of multiple images of an object. The methods and systems may be used to register medical images, such as images of successive slices of a tissue sample obtained from an animal subject. However, the methods and systems may also be used to register other types of images of an object. [Background technology]

[0002] Image registration can be used to overlay two or more images of the same subject taken at different times, from different viewpoints, and / or with different sensors. In the field of medical imaging, multiple images can be generated from a tissue sample of a subject, such as a human, to obtain complementary information regarding biological or medical insights about the subject. For example, a first image can be an immunohistochemistry (IHC) image of a first slice of the tissue sample, while a second image can be an IHC image of a second slice of the tissue sample. In some cases, the second slice can be contiguous to the first slice (i.e., the second slice was parallel to and adjacent to the first slice in the tissue sample). The first image can be a CD8 (cluster of differentiation 8)-stained IHC image, while the second IHC image can be, for example, a pancytokeratin (Pan-CK)-stained IHC image. The CD8 IHC image and the Pan-CK IHC image can be used to classify patients into a predetermined immunophenotype.

[0003] To facilitate analysis of images, they may need to be matched pixel-by-pixel through a process of image registration, which may allow the registered images to be used to subsequently perform multiple image analysis techniques, including, but not limited to, virtual staining, semantic image segmentation, cancer immunophenotyping, or creating patient models for simulation.

[0004] Currently, a human expert, such as a pathologist, may analyze the first and second images and attempt to manually register them, which can be a very time-consuming process and require a high level of skill. Alternatively, known computer-implemented registration methods may be used, but they may result in poor image registration. Images representing the same object but captured by different imaging modalities or that differ at a local level may not be registered by existing image registration methods. For example, existing image registration methods do not perform well for IHC images from consecutive slices of a tumor biopsy sample. Therefore, there is a need to provide an improved method of performing image registration of multiple images of an object that may be less time-consuming and provide improved registration. Summary of the Invention [Means for solving the problem]

[0005] According to a first aspect of the present disclosure, there is provided a computer-implemented method for performing image registration of multiple images of an object, the method including: receiving a first image of the object and a second image of the object; performing a first image registration to register the first image and the second image; receiving boundary shape parameters that identify a region of interest in the registered first image or the registered second image; generating a first cropped image of the registered first image and a second cropped image of the registered second image based on the boundary shape parameters; and performing a second image registration to register the first cropped image and the second cropped image.

[0006] Performing the first image registration may include segmenting the first image to generate a first segmented image, segmenting the second image to generate a second segmented image, generating a first distance map based on the first segmented image, generating a second distance map based on the second segmented image, registering the first distance map and the second distance map, and registering the first image and the second image based on the registration of the first distance map and the second distance map.

[0007] The first segmented image and the second segmented image may be binary images.

[0008] The first image registration may include a rigid transformation.

[0009] Performing the second image registration may include performing a normalized cross-correlation based on the first cropped image and the second cropped image.

[0010] Performing the second image registration may further include transforming the first cropped image and the second cropped image from the spatial domain to the frequency domain using a Fast Fourier Transform (FFT), and a normalized cross-correlation may be performed using the first cropped image in the frequency domain and the second cropped image in the frequency domain.

[0011] The object may be a living organism, and the multiple images may be multiple images of tissue of the organism.

[0012] The first image may be an image of a first slice of tissue of the organism, and the second image may be an image of a second slice of tissue of the organism.

[0013] The first image registration may be performed based on tissue-level morphology of the first and second images.

[0014] A second image registration can be performed based on the cellular level morphology of the first cropped image and the second cropped image.

[0015] At least one of the first image or the second image can be an immunohistochemistry image or a hematoxylin and eosin stained image.

[0016] At least one of the first cropped image may have a higher image resolution than the first image, or the second cropped image may have a higher image resolution than the second image.

[0017] The method may further include outputting the registered first cropped image and second cropped image for display.

[0018] According to a second aspect of the present disclosure, there is provided a computer-implemented method for performing image registration of multiple images of an object, the method including receiving a first image of the object and a second image of the object; and performing first image registration to register the first image and the second image, wherein performing the first image registration may include segmenting the first image to generate a first segmented image, segmenting the second image to generate a second segmented image, generating a first distance map based on the first segmented image, generating a second distance map based on the second segmented image, registering the first distance map and the second distance map; and registering the first image and the second image based on the registration of the first distance map and the second distance map.

[0019] The method may further include receiving boundary shape parameters that identify a region of interest in the registered first image or the registered second image, generating a first cropped image of the registered first image and a second cropped image of the registered second image based on the boundary shape parameters, and performing a second image registration to register the first cropped image and the second cropped image.

[0020] The first segmented image and the second segmented image may be binary images.

[0021] The first image registration may include a rigid transformation.

[0022] Performing the second image registration may include performing a normalized cross-correlation based on the first cropped image and the second cropped image.

[0023] Performing the second image registration may further include transforming the first cropped image and the second cropped image from the spatial domain to the frequency domain using a Fast Fourier Transform (FFT), and a normalized cross-correlation may be performed using the first cropped image in the frequency domain and the second cropped image in the frequency domain.

[0024] The object may be a living organism, and the multiple images may be multiple images of tissue of the organism.

[0025] The first image may be an image of a first slice of tissue of the organism, and the second image may be an image of a second slice of tissue of the organism.

[0026] The first image registration may be performed based on tissue-level morphology of the first and second images.

[0027] A second image registration can be performed based on the cellular level morphology of the first cropped image and the second cropped image.

[0028] At least one of the first image or the second image can be an immunohistochemistry image or a hematoxylin and eosin stained image.

[0029] At least one of the first cropped image may have a higher image resolution than the first image, or the second cropped image may have a higher image resolution than the second image.

[0030] The method may further include outputting the registered first cropped image and second cropped image for display.

[0031] According to a third aspect of the present specification, there is provided a computer program product comprising computer readable code which, when executed by a computing system, causes the computing system to perform any of the methods set forth above.

[0032] According to a fourth aspect of the present specification, there is provided a system comprising one or more processors and a memory, the memory storing computer-readable instructions that, when executed by the one or more processors, cause the system to perform any of the methods described above.

[0033] Embodiments will now be described, by way of non-limiting example, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0034] [Figure 1] 1 is a schematic diagram of an exemplary known method for performing image registration. [Figure 2] FIG. 2 shows an image extracted from the image of FIG. 1. [Figure 3] 1 is a schematic diagram of an exemplary method for performing image registration of multiple images of an object, according to aspects of the present disclosure. [Figure 4] 1 is a flowchart of an exemplary method for performing image registration of multiple images of an object, according to aspects of the present disclosure. [Figure 5] 10 is a flowchart of another exemplary method for performing image registration of multiple images of an object, according to aspects of the present disclosure. [Figure 6] FIG. 1 shows a schematic example of a system / apparatus for carrying out any of the methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0035] As discussed, known computer-implemented registration methods can result in poor image alignment, particularly for medical images obtained from tissue samples. For example, existing registration methods perform poorly on IHC images from successive slices of a tumor biopsy sample.

[0036] Figure 1 shows a schematic diagram of an exemplary known method for performing image registration. Figure 1 shows a first IHC image 102 and a second IHC image 104 before registration of the images by known registration methods. In this example, the first IHC image 102 is a CD8-stained IHC image, and the second IHC image 104 is a Pan-CK-stained IHC image.

[0037] First IHC image 102 and second IHC image 104 are of successive slices of a tumor biopsy sample. First IHC image 102 in this example includes at least an image representation of a first tissue structure 106a and an image representation of a second tissue structure 108a. Second IHC image 104 in this example, which is an image of a successive slice of the tumor biopsy sample, also includes image representations of the first and second tissue structures, although it can be seen in FIG. 1 that both the image representation of first tissue structure 106b and the image representation of second tissue structure 108b in second IHC image 104 are visually different from the corresponding image representation of first tissue structure 106a and the image representation of second tissue structure 108a in first IHC image 102. For example, the image representation of the first tissue structure 106b and the image representation of the second tissue structure 108b shown in the second IHC image 104 are in different positions relative to the image representation of the first tissue structure 106a and the image representation of the second tissue structure 108a in the first IHC image 102. Furthermore, while the tissue structures in the first IHC image 102 and the second image 104 may appear similar on a global tissue-level scale, the similarity may be less apparent on a local (e.g., cellular-level) scale.

[0038] The image representation of the first tissue structure 106a and the image representation of the second tissue structure 108a in the first IHC image 102 may differ from the image representation of the first tissue structure 106b and the image representation of the second tissue structure 108b in the second IHC image 104 for several different reasons. For example, the use of different staining techniques (CD8 vs. Pan-CK) to generate the first IHC image 102 and the second IHC image 104 may explain some of the differences. Furthermore, if the first IHC image 102 and the second IHC image 104 are obtained from different slices of the same tissue sample, each slice will be similar but not identical to the previous or next successive slice because each slice contains different cells.

[0039] Thus, consecutive slices may have very similar morphologies at the tissue level, but the morphologies may be less similar at the cellular level. This is illustrated by FIG. 2, which shows a first zoom portion 202 of a first IHC image 102 and a second zoom portion 204 of a second IHC image 104. The first zoom portion 202 includes a portion of an image representation of a second tissue structure 108a obtained from the first IHC image 102, and the second zoom portion 204 includes a portion of an image representation of a second tissue structure 108b obtained from the second IHC image 104. While the first zoom portion 202 and the second zoom portion 204 appear similar at a gross level (e.g., have similar contours of the tissue structures), closer inspection may reveal that they differ at the cellular level (indicated by the dots in the first zoom portion 202 and the second zoom portion 204). This may be due to different cell numbers and cell morphologies between the slice of tissue sample used for the first IHC image 102 and the slice of tissue used for the second IHC image 104 .

[0040] Known computer-implemented image registration methods for registering first IHC image 102 and second IHC image 104 may produce a suboptimal registration, as illustrated by composite IHC image 110 shown in Figure 1. After registering first IHC image 102 and second IHC image 104 using known image registration methods, composite IHC image 110 has been generated. Composite IHC image 110 may be generated by overlaying first IHC image 102 and second IHC image 104 after they have been aligned using known image registration methods.

[0041] The composite IHC image 110 shows the image representations 106a, 106b, 108a, 108b of the first and second tissue structures previously discussed in connection with the first and second IHC images 103, 104. However, it can be seen from the composite IHC image 110 that the image representations 106a, 106b, 108a, 108b of the first and second tissue structures are not aligned, and thus the first and second IHC images 102, 104 are generally poorly aligned. This poor alignment may result from the inability of known registration methods to successfully handle differences between the first and second IHC images 102, 104 at the cellular level. Such poor alignment may make analysis of the composite IHC image 110 difficult for medical professionals.

[0042] 1 has been discussed with respect to IHC images, the discussion may equally apply to other types of medical images, or indeed types of non-medical images. For example, at least one of the first IHC image 102 or the second IHC image 104 may be a different type of IHC image, or may be a hematoxylin and eosin (H&E) stained image rather than an IHC image. In another example, one of the images may be an image obtained by magnetic resonance imaging (MRI), and the other image may be obtained from a computed tomography (CT) scan.

[0043] Aspects of the present disclosure may provide improved methods of image registration, which may be particularly relevant to registering multiple images that have greater similarity on a global scale than they do on a local scale.

[0044] 3 shows a schematic diagram of a computer-implemented method 300 for performing image registration of multiple images of an object, according to an embodiment of the present invention. Method 300 may be performed by a computing system, such as the system described in connection with FIG.

[0045] A first image 302 of the subject and a second image 304 of the subject are received. The first image 302 and second image 304 shown in Figure 3 are the same as the first IHC image 102 and second IHC image 104 discussed in connection with Figure 1, although in other examples, different types of images may be received, such as H&E stained images.

[0046] In this example, the subject is a human patient, and the first image 302 and the second image 304 are images of successive slices of a tumor biopsy sample taken from the patient. The first image 302 and the second image 304 contain corresponding features, such as tissue structures, that are to be analyzed after the images are registered. However, the techniques described herein are applicable to many other scenarios and to a wide range of organisms, such as plants, animals, fungi, bacteria, and viruses.

[0047] A first image registration process is performed to register (i.e., align) the first image 302 and the second image 304 using the first image 302 and the second image 304. The image registration may be used to estimate a transformation between the first image 302 and the second image 304. The estimated transformation may be applied to the first image 302 or the second image 304 to align the images.

[0048] The first image registration is a global registration and aligns the first image 302 and the second image 304 based on tissue-level morphology of the first image 302 and the second image 304. The first image registration can be performed using any suitable image registration method, for example, as disclosed in Barbara Zitova, Jan Flusser, Image registration methods: a survey, Image and Vision Computing, Volume 21, Issue 11, 2003, pp. 977-1000, ISSN 0262-8856, https: / / doi.org / 10.1016 / S0262-8856(03)00137-9.

[0049] 3, performing the first image registration includes segmenting the first image 302 to generate a first segmented image 306 and segmenting the second image 304 to generate a second segmented image 308. The segmentation may be performed using a suitable threshold segmentation algorithm known in the art, although other types of segmentation algorithms known in the art may alternatively be used. The segmentation may result in the generation of a binary image.

[0050] An example of a suitable segmentation algorithm is the Otsu algorithm, which is disclosed, for example, in N. Otsu, "A Threshold Selection Method from Gray-Level Histograms," IEEE Transactions on Systems, Man, and Cybernetics, vol. 9, no. 1, pp. 62-66, Jan. 1979, doi:10.1109 / TSMC.1979.4310076 (https: / / ieeexplore.ieee.org / document / 4310076). Another example of a segmentation algorithm may use a deep learning based method using the UNET convolutional neural network, such as that disclosed by Ronneberger, O., Fischer, P., Brox, T (2015), U-Net: Convolutional Networks for Biomedical Image Segmentation, In: Navab, N., Hornegger, J., Wells, W., Frangi, A. (eds) Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015, MICCAI 2015.

[0051] The first segmented image 306 (which may be a binary image) is then converted into a first distance map 310, and the second segmented image 308 (which may also be a binary image) is converted into a second distance map 312. Distance maps are also known as distance transforms. The conversion to a distance map may be performed using a suitable algorithm, such as that disclosed in C.R. Maurer, Jr., R. Qi, and V. Raghavan, "A Linear Time Algorithm for Computing Exact Euclidean Distance Transforms of Binary Images in Arbitrary Dimensions," IEEE - Transactions on Pattern Analysis and Machine Intelligence, 25(2):265-270, 2003.

[0052] The value of each pixel in the distance maps 310, 312 corresponds to the distance to the nearest contour point in the corresponding segmented image 306, 308. Binary images do not provide strong gradients and are therefore not useful for optimization when aligning images during image registration. The distance map representation provides a better gradient signal and may therefore lead to improved optimization results when aligning images.

[0053] The first image 302 may then be registered to the second image 304 by performing image registration on the first distance map 310 and the second distance map 312 to obtain an estimated transformation between the first distance map 310 and the second distance map 312. The first image 302 may be registered to the second image 304 by applying the estimated transformation.

[0054] The first image registration may include a rigid image registration involving a rigid transformation, which is a transformation by rotation and / or translation. Based on the assumption that the slices of tissue shown in the first image 302 and the second image 304 will be very similar morphologically at the tissue level, a rigid transformation should be sufficient for the first image registration. However, in other examples, a different transformation, such as an affine transformation or a non-rigid transformation, may be used.

[0055] In this example, a gradient descent algorithm is used for the first image registration, which finds the best parameters given the rigid transformation, but in other examples, the first image registration can be performed using any suitable algorithm known in the art. To use the gradient descent algorithm, a cost function is defined, the value of which will be optimized. In this case, mean square error is used as the cost function, but in other examples, other metrics such as mutual information or L1 difference can be used.

[0056] 3 shows a composite image 314 generated from registered first image 302 and second image 304. Comparing composite image 314 shown in FIG. 3 with composite IHC image 110 shown in FIG. 1, it can be seen that the holistic registration method described thus far can already provide improved image registration compared to previous image registration methods, for example, compared to previous image registration methods that do not involve segmenting images and converting them into distance maps before performing registration.

[0057] Diagram 316 shows a portion of composite image 314 that generally corresponds to the second tissue structure described above in connection with Figures 1 and 2. From diagram 316, it can be seen that first image 302 and second image 304 are already well registered.

[0058] Note that in some examples, composite image 314 and / or image 316 are not actually generated; they are provided in FIG. 3 to illustrate the degree of alignment of first image 302 and second image 304 due to first image registration.

[0059] The first global image registration may allow for sufficient accuracy in aligning the first image 302 and the second image 304. Nevertheless, in some examples, further accuracy in registering the first image 302 and the second image 304 may be achieved by also performing a second local image registration, as discussed below.

[0060] Once the first image 302 and the second image 304 are aligned using the first registration, a region of interest (ROI) 318 is identified in at least one of the first image 302 or the second image 304, or in some examples, in the composite image 314. The ROI may be identified by a human user, such as a pathologist, but in other examples, may be identified by a computer-implemented process, for example, using a machine learning algorithm trained to identify specific ROIs. The ROI may include features in the first image 302 and / or the second image 304 that are of interest to the pathologist for further analysis.

[0061] Parameters of a boundary shape 320 that identifies an ROI in the first image 302 or the second image 304 are received. The parameters may include coordinates of the boundary shape 320, for example, coordinates corresponding to the location of the boundary shape in the first image 302 or the second image 304. The parameters may be determined based on input provided by a user. For example, a user may draw the boundary shape 320 on a displayed version of the first image 302, the second image 304, or the composite image 314 using a user input interface, such as a touchscreen or mouse coupled to a computing system as described in connection with FIG. 6 .

[0062] The parameters (e.g., coordinates) of the boundary shape 320 may be used to generate a pair of cropped images, specifically a first cropped image 322 of the first image 302 and a second cropped image 324 of the second image 304. More specifically, the parameters and transformation estimated by the first image registration process are used to identify the boundary shape 320 (and thus the ROI 318) in both the registered first image 302 and second image 304. For example, if parameters are associated with the boundary shape 320 identified in the first image 302, a corresponding boundary shape 320 may be determined for the second image 304 by applying a transformation to the parameters. Similarly, if parameters are associated with the boundary shape 320 identified in the second image 304, a corresponding boundary shape 320 may be determined for the first image 302 by applying a transformation to the parameters.

[0063] The first cropped image 322 may be cropped from the first image 302 based on the parameters to include at least the ROI 318 identified by the bounding shape 320. The first cropped image 322 may be generated to include a margin around the bounding shape 320 such that the first cropped image 322 is larger than and includes the ROI 318 and the bounding shape 320. For example, the first cropped image 322 may have a width and height that are, in some examples, greater than the maximum width and height of the ROI 318 in the first image 302 by a predetermined amount.

[0064] In a similar manner to the first cropped image 322, the second cropped image 324 may be cropped from the second image 304 based on the parameters to include at least the ROI 318 identified by the bounding shape 320. The second cropped image 324 may also be generated to include a margin around the bounding shape 320 such that the second cropped image 324 is larger than and includes the ROI 318 and the bounding shape 320. For example, the second cropped image 324 may also have a width and height that, in some examples, are greater than the maximum width and height of the ROI 318 in the second image 304 by a predetermined amount.

[0065] FIG. 3 shows the ROI 318 and boundary shape 320 located on the second cropped image 324 .

[0066] A second local image registration process is then performed to register the first cropped image 322 and the second cropped image 324. The second image registration may involve registering the first cropped image 322 and the second cropped image 324 at a smaller level than the first registration of the first image 302 to the second image 304. That is, the second image registration may involve registering the first cropped image 322 and the second cropped image 324 based on the cellular-level morphology shown in the images rather than the tissue-level morphology within the images.

[0067] Performing the second image registration may include performing normalized cross-correlation (NCC) to register the first cropped image 322 and the second cropped image 324. As an example, the NCC method disclosed in D. Padfield, "Masked object registration in the Fourier domain," IEEE Transactions on Image Processing (2012), DOI: 10.1109 / TIP.2011.2181402, which is incorporated herein by reference, may be used. NCC is advantageous in that it is very fast to compute in the Fourier domain. However, in other examples, the second image registration may be performed using a different suitable registration algorithm, such as the algorithm described in connection with the first image registration.

[0068] A fast Fourier transform (FFT) may be used to transform the first cropped image 322 and the second cropped image 324 from the spatial domain to the frequency domain so that NCC may be performed, and then NCC may be performed on the transformed first cropped image 322 and the second cropped image 324 in the frequency (Fourier) domain. A translational transformation to align the first cropped image 322 and the second cropped image 324 may be determined based on the results of the NCC.

[0069] A translational transformation may then be used to register the first cropped image 322 and the second cropped image 324. Additionally or alternatively, a translational transformation may be used to register the first image 302 and the second image 304.

[0070] The registered first cropped image 322 and second cropped image 324 may be output for display, for example, by a display coupled to the computing system described in connection with FIG.

[0071] 3, a composite image 326 based on the registered first cropped image 322 and second cropped image 324 may be generated and output for display. The composite image 326 may include the registered first cropped image 322 overlaid with the registered second cropped image 324. FIG. 3 also shows the ROI 318 highlighted on the composite image 326, although this is optional.

[0072] In some implementations, the cropped images 322, 324 are resized / rescaled to a predetermined size before performing the second image registration. Interpolation techniques may be used to perform the rescaling to fill in the missing pixel data.

[0073] The first cropped image 322 may have a higher resolution than the first image 302. For example, the first image 302 may be a relatively lower resolution version of an image, while the first cropped image 322 may be a cropped portion of a relatively higher resolution version of the same image. Similarly, the second cropped image 324 may have a higher resolution than the second image 304. This may improve the efficiency of the overall image registration method by performing global registration on a larger, relatively lower resolution image, and then performing local registration on a smaller, relatively higher resolution cropped image.

[0074] 4 shows a flowchart of an exemplary method 400 for performing image registration of multiple images of an object, according to an embodiment of the present invention. Method 400 may be performed by a computing system, such as the system described in connection with FIG. 6. Method 400 may share one or more aspects with the method described above in connection with FIG. 3.

[0075] In step 402, a first image 302 of an object and a second image 304 of the object are received. As an example, the object may be a living organism, such as an animal, particularly a human. The first image 302 may be an image of a first slice of tissue of the organism, and the second image 304 may be an image of a second slice of tissue. The first slice and the second slice may be consecutive or nearly consecutive slices of the same tissue. The first image 302 and the second image 304 may be obtained using different image preparation methods. For example, at least one of the first image 302 or the second image 304 may be an immunohistochemistry (IHC) image or a hematoxylin and eosin (H&E) stained image.

[0076] In step 404, a first image registration process is performed to register the first image 302 and the second image 304. The first image registration may be a global registration and may be performed based on tissue-level morphology of the first image 302 and the second image 304. The first image registration may involve a rigid transformation, although in other examples, an affine transformation or a non-rigid transformation may be used.

[0077] The first image registration may be performed using a gradient descent algorithm, with mean squared error selected as the cost function to optimize, however, in other examples, a different cost function may be used, such as mutual information or L1 difference, as previously discussed, and / or a different algorithm for gradient descent may be used.

[0078] In step 406, parameters of a bounding shape 320 that identifies the region of interest 318 in the first image 302 or second image 304 are received. The parameters may include coordinates and may be input by a user, for example, by drawing the bounding shape 320 on a representation of the first image or second image output on a display.

[0079] In step 408, a first cropped image 322 of the first image 302 and a second cropped image 324 of the second image 304 are generated based on the parameters of the boundary shape 320. Both the first image 302 and the second image 304 are cropped to include the ROI 318 identified by the boundary shape 320. Because the first image 302 and the second image 304 were registered using the first image registration process in step 404, the position of the boundary shape 320 in the first image 302 will correspond to the position of the boundary shape 320 in the second image 304 based on the rigid transformation estimated during the first image registration. Thus, the parameters (e.g., coordinates) of the boundary shape 320 may be applied to both the first image 302 and the second image 304 to generate the first cropped image 322 and the second cropped image 324. This step limits the regions of the first image 302 and the second image 304 that are subjected to the second image registration process. The first cropped image 322 and / or the second cropped image 324 may be generated to include a margin around the bounding shape 320. The first cropped image 322 and / or the second cropped image 324 may have a higher resolution than the corresponding first image 302 or second image 304.

[0080] At step 410, a second image registration process is performed to register the first cropped image 322 and the second cropped image 324. The second image registration may involve registering the first cropped image 322 and the second cropped image 324 based on cellular-level morphology rather than tissue-level morphology.

[0081] Performing the second image registration may include performing a normalized cross-correlation (NCC) on the first cropped image 322 and the second cropped image 324 to register the images, as described above, although in other examples, the second image registration may be performed using a different suitable registration algorithm.

[0082] Once the first cropped image 322 and the second cropped image 324 are registered, the registered first cropped image 322 and second cropped image 324 may be output for display, for example, on a computer display. A composite image 326 including the registered first cropped image 322 and second cropped image 324 (or at least a portion of the registered first cropped image and second cropped image) may be output for display.

[0083] Figure 5 shows a flowchart of another exemplary method 500 for performing image registration of multiple images of an object. Method 500 may be performed by a computing system, such as the system described in connection with Figure 6. Method 500 shown in Figure 5 is similar to the method shown in Figure 5, but has additional steps.

[0084] In step 502, a first image of an object 302 and a second image of an object 304 are received, in some examples in a manner similar to that described above in connection with step 402 of FIG.

[0085] In step 504, a first image registration process is performed to register the first image 302 and the second image 304. This may be in a similar manner as described above in connection with step 404 of Figure 4, except that step 504 includes several additional steps 506, 508, 510.

[0086] In step 506, the first image 302 is segmented to generate a first segmented image 306, and the second image 304 is segmented to generate a second segmented image 308. This may be performed, for example, as described above in connection with FIG.

[0087] In step 508, a first distance map 310 is generated based on the first segmented image 306, and a second distance map 312 is generated based on the second segmented image 308. This may be performed, for example, as described above in connection with FIG.

[0088] In step 510, the first distance map 310 is registered to the second distance map 312. This may be performed using a registration algorithm, for example, as described above in connection with FIG. 3. The first image 302 and the second image 304 may then be registered based on the registration of the first distance map 310 and the second distance map 312. For example, to align the first image 302 and the second image 304, a transformation estimated from the registration of the first distance map 310 and the second distance map 312 may be applied to the first image 302 and / or the second image 304.

[0089] In step 512, parameters of a boundary shape 320 that identifies a region of interest 318 in the first image 302 or the second image 304 are received. Step 512 may correspond to one or more aspects of step 406 as previously discussed and will not be repeated for the sake of brevity.

[0090] In step 514, a first cropped image 322 of the first image 302 and a second cropped image 324 of the second image 304 are generated based on the parameters of the bounding shape 320. Step 514 may correspond to one or more aspects of step 408 as previously discussed and will not be repeated for the sake of brevity.

[0091] In step 516, a second image registration process is performed to register the first cropped image 322 and the second cropped image 324. Step 516 may correspond to one or more aspects of step 410 as previously discussed and will not be repeated for the sake of brevity.

[0092] Once the first cropped image 322 and the second cropped image 324 are registered, the registered first cropped image 322 and second cropped image 324 may be output for display, for example, on a computer display. A composite image 326 including the registered first cropped image 322 and second cropped image 324 (or at least a portion of the registered first cropped image and second cropped image) may be output for display.

[0093] 6 shows a schematic example of a system / apparatus 600 for performing any of the methods described herein. The illustrated system / apparatus is an example of a computing device. Those skilled in the art will appreciate that other types of computing devices / systems, such as, for example, distributed computing systems, may alternatively be used to perform the methods described herein.

[0094] The device (or system) 600 includes one or more processors 602. The one or more processors control the operation of the other components of the system / device 600. The one or more processors 602 may include, for example, a general-purpose processor. The one or more processors 602 may be single-core or multi-core devices. The one or more processors 602 may include a central processing unit (CPU) or a graphical processing unit (GPU). Alternatively, the one or more processors 702 may include specialized processing hardware, for example, a RISC processor or programmable hardware with embedded firmware. Multiple processors may be included.

[0095] The system / device includes a working or volatile memory 604. One or more processors may access the volatile memory 704 to process data and may control data storage in the memory. The volatile memory 604 may include any type of RAM, such as static RAM (SRAM), dynamic RAM (DRAM), or it may include flash memory, such as an SD-card.

[0096] The system / device includes a non-volatile memory 606. The non-volatile memory 606 stores a set of operating instructions 608 in the form of computer-readable instructions for controlling the operation of the processor 602. The non-volatile memory 606 can be any type of memory, such as read-only memory (ROM), flash memory, or magnetic drive memory.

[0097] The one or more processors 602 are configured to execute operational instructions 608 to cause the system / device to perform any of the methods described herein. The operational instructions 608 may include code for the basic operation of the system / device 600, as well as code (i.e., drivers) for hardware components of the system / device 600. Generally, the one or more processors 602 execute one or more instructions of the operational instructions 608, which are stored permanently or semi-permanently in non-volatile memory 606, while the volatile memory 604 is used to temporarily store data generated during execution of the operational instructions 608.

[0098] Implementations of the methods described herein may be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof, which may include a computer program product (e.g., software stored on a magnetic or optical disk, memory, programmable logic device, etc.) containing computer-readable instructions that, when executed by a computer, cause the computer to perform one or more of the methods described herein, such as those described in connection with FIG.

[0099] System features described herein may also be provided as method features, and vice versa. As used herein, "means-plus-function" features may be expressed in terms of their corresponding structure. In particular, method aspects may also apply to system aspects, and vice versa.

[0100] Furthermore, any, some, and / or all features of one aspect may be applied to any, some, and / or all features of any other aspect in any suitable combination. It should also be understood that specific combinations of various features described and defined in any aspect of the invention may be implemented and / or provided and / or used separately.

[0101] While several embodiments have been shown and described, it will be understood by those skilled in the art that changes can be made to these embodiments without departing from the principles of the present disclosure, the scope of which is defined in the claims or equivalents thereof.

[0102] As used herein, the terms "drug" or "medicament" are used interchangeably to refer to a pharmaceutical formulation containing one or more active pharmaceutical ingredients or pharmaceutically acceptable salts or solvates thereof and, optionally, a pharmaceutically acceptable carrier. An active pharmaceutical ingredient ("API"), in its broadest sense, is a chemical structure that has a biological effect on humans or animals. In pharmacology, drugs or medications are used to treat, cure, prevent, or diagnose disease, or otherwise improve physical or mental well-being. Drugs or medications may be used for a limited period of time or periodically for chronic conditions.

[0103] As described below, drugs or pharmaceutical agents can include at least one API or a combination thereof in various types of formulations for the treatment of one or more diseases. Examples of APIs include small molecules having a molecular weight of 500 Da or less, polypeptides, peptides, and proteins (e.g., hormones, growth factors, antibodies, antibody fragments, and enzymes), carbohydrates and polysaccharides, as well as nucleic acids, double-stranded or single-stranded DNA (including naked and cDNA), RNA, antisense nucleic acids such as antisense DNA and RNA, small interfering RNA (siRNA), ribozymes, genes, and oligonucleotides. Nucleic acids can be incorporated into molecular delivery systems such as vectors, plasmids, or liposomes. Mixtures of one or more drugs are also contemplated.

[0104] The drug or agent may be contained within a primary package or "drug container" adapted for use with a drug delivery device. The drug container may be, for example, a cartridge, syringe, reservoir, or other sturdy or flexible vessel configured to provide a chamber suitable for storage (e.g., short-term or long-term storage) of one or more drugs. For example, in some cases, the chamber may be designed to store the drug for at least one day (e.g., from one day to at least 30 days). In some cases, the chamber may be designed to store the drug for about one month to about two years. Storage may occur at room temperature (e.g., about 20°C) or at refrigerated temperatures (e.g., from about -4°C to about 4°C). In some cases, the drug container may be or include a dual-chamber cartridge configured to separately store two or more components of a pharmaceutical formulation to be administered (e.g., an API and a diluent or two different drugs), one in each chamber. In such cases, the two chambers of the dual-chamber cartridge may be configured to allow mixing of the two or more components prior to and / or during administration to a human or animal body. For example, the two chambers may be configured so that they are in fluid communication with one another (e.g., by a conduit between the two chambers), allowing the user to mix the two components if desired prior to administration. Alternatively, or additionally, the two chambers may be configured to allow mixing upon administration of the components into the human or animal body.

[0105] Drugs or agents contained in drug delivery devices as described herein can be used to treat and / or prevent many different types of medical disorders. Examples of disorders include, for example, diabetes mellitus or complications associated with diabetes mellitus, such as diabetic retinopathy, and thromboembolic disorders, such as deep vein thromboembolism or pulmonary thromboembolism. Further examples of disorders include acute coronary syndrome (ACS), angina pectoris, myocardial infarction, cancer, macular degeneration, inflammation, hay fever, atherosclerosis, and / or rheumatoid arthritis. Examples of APIs and drugs are those listed in handbooks such as the Rote Liste 2014, including, but not limited to, Main Group 12 (antidiabetic drugs) or 86 (oncology drugs), and the Merck Index, 15th edition.

[0106] Examples of APIs for the treatment and / or prevention of type 1 or type 2 diabetes mellitus or complications of type 1 or type 2 diabetes mellitus include insulin, e.g., human insulin, or an insulin analog or derivative, glucagon-like peptide (GLP-1), a GLP-1 analog or GLP-1 receptor agonist, or an analog or derivative thereof, a dipeptidyl peptidase-4 (DPP4) inhibitor, or a pharmaceutically acceptable salt or solvate thereof, or any mixture thereof. As used herein, the terms "analog" and "derivative" refer to a polypeptide having a molecular structure that is formally derivable from the structure of a naturally occurring peptide, e.g., the structure of human insulin, by deleting and / or replacing at least one amino acid residue occurring in the naturally occurring peptide and / or by adding at least one amino acid residue. The added and / or replaced amino acid residue can be either a codable amino acid residue or other naturally occurring residue, or a purely synthetic amino acid residue. Insulin analogs are also referred to as "insulin receptor ligands." In particular, the term "derivative" refers to a polypeptide having a molecular structure formally derivable from that of a naturally occurring peptide, e.g., the structure of human insulin, in which one or more organic substituents (e.g., fatty acids) are attached to one or more of the amino acids. Optionally, one or more amino acids present in the naturally occurring peptide may be deleted and / or substituted by other amino acids, including non-codable amino acids, or amino acids, including non-codable amino acids, may be added to the naturally occurring peptide.

[0107] Examples of insulin analogues are Gly(A21), Arg(B31), Arg(B32) human insulin (insulin glargine), Lys(B3), Glu(B29) human insulin (insulin glulisine), Lys(B28), Pro(B29) human insulin (insulin lispro), Asp(B28) human insulin (insulin aspart), human insulin in which the proline in position B28 may be substituted with Asp, Lys, Leu, Val or Ala and the Lys in position B29 may be substituted with Pro, Ala(B26) human insulin, Des(B28-B30) human insulin, Des(B27) human insulin and Des(B30) human insulin. Examples of insulin derivatives include B29-N-myristoyl-des(B30) human insulin, Lys(B29)(N-tetradecanoyl)-des(B30) human insulin (insulin detemir, Levemir®), B29-N-palmitoyl-des(B30) human insulin, B29-N-myristoyl human insulin, B29-N-palmitoyl human insulin, B28-N-myristoylLysB28ProB29 human insulin, B28-N-palmitoyl-LysB28ProB29 human insulin, and B30-N-myristoyl-ThrB29LysB30 human insulin. Examples of insulin include B29-N-palmitoyl-ThrB29LysB30 human insulin, B29-N-(N-palmitoyl-gamma-glutamyl)-des(B30) human insulin, B29-N-omega-carboxypentadecanoyl-gamma-L-glutamyl-des(B30) human insulin (insulin degludec, Tresiba®), B29-N-(N-lithocholyl-gamma-glutamyl)-des(B30) human insulin, B29-N-(ω-carboxyheptadecanoyl)-des(B30) human insulin, and B29-N-(ω-carboxyheptadecanoyl) human insulin.

[0108] Examples of GLP-1, GLP-1 analogs and GLP-1 receptor agonists include, for example, lixisenatide (Lyxumia®), exenatide (exendin-4, Byetta®, Bydureon®, a 39 amino acid peptide produced by the salivary glands of the flathead monster), liraglutide (Victoza®), semaglutide, taspoglutide, albiglutide (Syncria®), dulaglutide (Trulicity®), rexendin-4, CJC-1134-PC, PB-1023, TTP-054, langrenatide / HM-11260C (efpegrenatide), HM-15211, CM-3, GLP-1 Erigen, ORMD-0901, NN-9423, NN-9709, NN-9924, NN-9926, NN-9927, Nodexene, Viador-GLP-1, CVX-096, ZYOG-1, ZYD-1, GSK-2374697, DA-3091, MAR-701, MAR709, ZP-2929, These include ZP-3022, ZP-DI-70, TT-401 (pegapamodtide), BHM-034, MOD-6030, CAM-2036, DA-15864, ARI-2651, ARI-2255, tirzepatide (LY3298176), bamadutide (SAR425899), exenatide-XTEN, and glucagon-Xten.

[0109] An example of an oligonucleotide is mipomersen sodium (Kynamro®), a cholesterol-lowering antisense therapeutic agent for the treatment of, for example, familial hypercholesterolemia, or RG012 for the treatment of Alport syndrome. Examples of DPP4 inhibitors are linagliptin, vildagliptin, sitagliptin, denagliptin, saxagliptin, berberine.

[0110] Examples of hormones include pituitary or hypothalamic hormones or regulatory active peptides such as gonadotropins (follitropin, lutropin, chorion gonadotropin, menotropin), somatropin (somatropin), desmopressin, terlipressin, gonadorelin, triptorelin, leuprorelin, buserelin, nafarelin, and goserelin, and their antagonists.

[0111] Examples of polysaccharides include glycosaminoglycans, hyaluronic acid, heparin, low molecular weight heparin, or ultra-low molecular weight heparin, or derivatives thereof, or sulfated forms of the above polysaccharides, such as polysulfated forms, and / or pharmaceutically acceptable salts thereof. An example of a pharmaceutically acceptable salt of polysulfated low molecular weight heparin is enoxaparin sodium. An example of a hyaluronic acid derivative is Hylan G-F20 (Synvisc®), which is derived from sodium hyaluronate.

[0112] As used herein, the term "antibody" refers to an immunoglobulin molecule or an antigen-binding portion thereof. Examples of antigen-binding portions of immunoglobulin molecules include F(ab) and F(ab')2 fragments that retain antigen-binding ability. An antibody can be a polyclonal antibody, a monoclonal antibody, a recombinant antibody, a chimeric antibody, a deimmunized or humanized antibody, a fully human antibody, a non-human (e.g., murine) antibody, or a single-chain antibody. In some embodiments, an antibody has effector function and is capable of fixing complement. In some embodiments, an antibody has reduced or no binding ability to Fc receptors. For example, an antibody can be an isotype or subtype, antibody fragment, or mutant that does not support Fc receptor binding, e.g., in which the Fc receptor binding region has been mutated or deleted. The term antibody also includes antigen-binding molecules based on tetravalent bispecific tandem immunoglobulins (TBTIs) and / or dual variable region antibody-like binding proteins with a crossover binding region orientation (CODV).

[0113] The term "fragment" or "antibody fragment" refers to a polypeptide derived from an antibody polypeptide molecule (e.g., an antibody heavy and / or light chain polypeptide) that does not include the full-length antibody polypeptide but comprises at least a portion of the full-length antibody polypeptide that is still capable of binding to antigen. Antibody fragments can include truncations of a full-length antibody polypeptide, but the term is not limited to such truncated fragments. Antibody fragments useful in the present invention include Fab fragments, F(ab')2 fragments, scFv (single-chain Fv) fragments, linear antibodies, monospecific or multispecific antibody fragments, such as bispecific, trispecific, tetraspecific, and multispecific antibodies (e.g., diabodies, triabodies, and tetrabodies), monovalent or multivalent antibody fragments, such as bivalent, trivalent, tetravalent, and multivalent antibodies, minibodies, chelating recombinant antibodies, tribodies or bibodies, intrabodies, nanobodies, small modular immunopharmaceuticals (SMIPs), binding domain immunoglobulin fusion proteins, camelized antibodies, and VHH-containing antibodies. Additional examples of antigen-binding antibody fragments are known in the art.

[0114] The term "complementarity determining region" or "CDR" refers to short polypeptide sequences within the variable regions of both heavy and light chain polypeptides that are primarily responsible for mediating specific antigen recognition. The term "framework region" refers to amino acid sequences within the variable regions of both heavy and light chain polypeptides that are not CDR sequences but are primarily responsible for maintaining the proper orientation of the CDR sequences to enable antigen binding. Although the framework regions themselves typically do not directly participate in antigen binding, as is known in the art, certain residues within the framework regions of a particular antibody may be directly involved in antigen binding or may affect the ability of one or more amino acids within the CDRs to interact with the antigen. Examples of antibodies are anti-PCSK-9 mAb (e.g., alirocumab), anti-IL-6 mAb (e.g., sarilumab), and anti-IL-4 mAb (e.g., dupilumab).

[0115] Pharmaceutically acceptable salts of any of the APIs described herein are contemplated for use in the drug or drugs in the drug delivery device. Pharmaceutically acceptable salts include, for example, acid addition salts and base salts. Those skilled in the art will understand that modifications (addition and / or removal) of the various components of the APIs, formulations, devices, methods, systems, and embodiments described herein may be made without departing from the full scope and spirit of the invention, which encompasses such modifications and all equivalents thereof.

[0116] An exemplary drug delivery device may include a needle-based injection system as described in Table 1 of Section 5.2 of ISO 11608-1:2014(E). As described in ISO 11608-1:2014(E), needle-based injection systems can be broadly divided into multi-dose container systems and single-dose (with partial or complete evacuation) container systems. The container may be a replaceable container or a one-piece, non-replaceable container.

[0117] As further described in ISO 11608-1:2014(E), a multi-dose container system may include a needle-based injection device with replaceable containers. In such a system, each container holds multiple doses and may be fixed or variable in size (pre-set by the user). Another multi-dose container system may include a needle-based injection device integrated with a non-replaceable container. In such a system, each container holds multiple doses and may be fixed or variable in size (pre-set by the user).

[0118] As further described in ISO 11608-1:2014(E), a single-dose container system may include a needle-based injection device with replaceable containers. In one example of such a system, each container holds a single dose, thereby dispensing the entire deliverable amount (full discharge). In a further example, each container holds a single dose, thereby dispensing a portion of the deliverable amount (partial discharge). Also as described in ISO 11608-1:2014(E), a single-dose container system may include a needle-based injection device integrated with a non-replaceable container. In one example of such a system, each container holds a single dose, thereby dispensing the entire deliverable amount (full discharge). In a further example, each container holds a single dose, thereby dispensing a portion of the deliverable amount (partial discharge).

Claims

1. 1. A computer-implemented method (300, 400, 500) for performing image registration of multiple images of an object, comprising: receiving (402, 502) a first image (302) of an object and a second image (304) of the object; performing a first image registration to register the first image and the second image (404, 504); receiving (406, 512) parameters of a boundary shape (320) that identifies a region of interest (318) in the registered first image or the registered second image; generating (408, 514) a first cropped image (322) of the registered first image and a second cropped image (324) of the registered second image based on the parameters of the boundary shape; performing a second image registration to register the first cropped image and the second cropped image (410, 516); Including, performing the first image registration; Segmenting the first image to generate a first segmented image and segmenting the second image to generate a second segmented image (506); generating a first distance map based on the first segmented image and a second distance map based on the second segmented image (508); registering the first distance map and the second distance map (510); registering the first image and the second image based on the registration of the first distance map and the second distance map; A computer-implemented method (300, 400, 500) comprising:

2. The method of claim 1 , wherein the first segmented image and the second segmented image are binary images.

3. The method of claim 1 or 2, wherein the first image registration comprises a rigid transformation.

4. 4. The method of claim 1, wherein performing the second image registration comprises performing a normalized cross-correlation based on the first cropped image and the second cropped image.

5. 5. The method of claim 4, wherein performing the second image registration further comprises transforming the first cropped image and the second cropped image from a spatial domain to a frequency domain using a Fast Fourier Transform (FFT), and wherein the normalized cross-correlation is performed using the first cropped image in the frequency domain and the second cropped image in the frequency domain.

6. The method of any one of claims 1 to 5, wherein the object is a living organism and the plurality of images is a plurality of images of tissues of the organism.

7. The method of claim 6 , wherein the first image is an image of a first slice of the tissue of the organism and the second image is an image of a second slice of the tissue of the organism.

8. The method of claim 6 or 7, wherein the first image registration is performed based on tissue-level morphology of the first image and the second image.

9. The method of claim 6, 7, or 8, wherein the second image registration is performed based on cellular-level morphology of the first cropped image and the second cropped image.

10. 10. The method of claim 1, wherein at least one of the first image or the second image is an immunohistochemistry image or a hematoxylin and eosin stained image.

11. the first cropped image has a higher image resolution than the first image; or the second cropped image has a higher image resolution than the second image; The method according to any one of claims 1 to 10, wherein the method is at least one of

12. The method of any one of claims 1 to 11, further comprising outputting the registered first and second cropped images for display.

13. A computer program product comprising computer readable code which, when executed by a computing system, causes said computing system to perform the method of any one of claims 1 to 12.

14. 13. A system (600) comprising one or more processors (602) and a memory (606), the memory storing computer readable instructions (608) that, when executed by the one or more processors, cause the system to perform the method of any one of claims 1 to 12.

Images