Method and system for image registration

Abstract

This invention provides various methods and systems for automatically registering and stitching images. In one example, one method includes: taking a first image (224) and a second image (225) of an object as input, training a model (260) to output a transformation matrix (270) based on the first image (224) and the second image (225), wherein the model (260) is trained with multiple training datasets (242), each training dataset (242) including image pairs (244, 245), masks indicating regions of interest (ROI) (246, 247), and associated ground truth (248); automatically stitching the first image (224) and the second image (225) together based on the transformation matrix (270) to form a stitched image; and outputting the stitched image for display on a display device (134) and / or storing the stitched image in a memory (106).

Description

Technical Field

[0001] The implementation schemes of the subject matter disclosed in this article relate to medical imaging. Background Technology

[0002] Imaging techniques such as X-ray imaging allow for the non-invasive acquisition of images of internal structures or features of subjects, such as patients. Digital X-ray imaging systems produce digital data that can be processed into radiographic images. In a digital X-ray imaging system, radiation from a source is directed towards the subject. A portion of the radiation passes through the subject and impacts a detector. The detector comprises discrete image elements or an array of detector pixels and generates an output signal based on the amount or intensity of radiation impacting each pixel region. The output signal is then processed to generate images that can be displayed for viewing. These images are used to identify and / or examine internal structures and organs within the patient's body. In some cases, multiple images can be used to capture a single structure or region of interest. Multiple images can be captured under the same or different conditions (e.g., patient position, radiation dose, etc.). Summary of the Invention

[0003] This paper provides an implementation scheme for image registration. In one example, a method includes training a model based on a transformation matrix output from a first image and a second image of an object as input. The model is trained on multiple training datasets, each including image pairs, masks indicating regions of interest (ROIs), and associated ground truth values. The method also includes automatically stitching the first and second images together based on the transformation matrix to form a stitched image, and outputting the stitched image for display on a display device and / or storing the stitched image in memory.

[0004] It should be understood that the above brief description is provided to introduce selected concepts further described in the detailed embodiments in a simplified form. This is not intended to identify key or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims following the detailed embodiments. Furthermore, the claimed subject matter is not limited to embodiments that address any shortcomings mentioned above or in any part of this disclosure. Attached Figure Description

[0005] The invention will be better understood by referring to the following description of non-limiting embodiments, in which:

[0006] Figure 1 A block diagram of an exemplary embodiment of an image processing system is shown;

[0007] Figure 2 A block diagram of an exemplary implementation of a Transform Matrix Prediction Network (TMPN) training and implementation system is shown;

[0008] Figure 3A A first exemplary workflow for training TMPN is shown;

[0009] Figure 3B A second exemplary workflow for training TMPN is shown;

[0010] Figure 4 A flowchart is shown as an exemplary method for training a TMPN to generate a transformation matrix based on input data;

[0011] Figure 5 A flowchart of an exemplary method for implementing a trained TMPN is shown;

[0012] Figure 6A Exemplary images comparing automatic and manual pasting of X-ray image pairs with lateral motion are shown;

[0013] Figure 6B Exemplary images comparing automatic and manual pasting for X-ray image pairs including non-rigid artifacts are shown;

[0014] Figure 6C Exemplary images comparing automatic and manual pasting of X-ray image pairs taken using an incorrect scheme are shown;

[0015] Figure 6D An exemplary X-ray image including a foreign object is shown; and

[0016] Figure 7 An exemplary image is shown that compares the input image pair and the resulting stitched image according to the method of this disclosure. Detailed Implementation

[0017] The following description relates to various implementations of methods for training and implementing deep learning models such as convolutional neural networks (CNNs) to output transformation matrices based on the correspondence of image pairs and to automatically stitch together images in the image pairs based on the transformation matrices to form a stitched image.

[0018] Some imaging protocols, such as those used for imaging the spine or one or more leg bones (e.g., the femur), involve acquiring two overlapping images and then stitching them together to form a stitched image that shows the full range of desired anatomical features that cannot be imaged in a single image. Conventional mechanisms for automatically stitching two overlapping images together may include search-based methods where common anatomical features between the overlapping images are identified and used as the basis for the stitched image. However, these methods often result in poor image registration (e.g., causing anatomical features in one image to be laterally offset from those in another) when lateral movement occurs between image acquisitions. Furthermore, these methods can struggle to register and stitch images when there is minimal overlap between images (e.g., less than 5 cm), when non-rigid artifacts such as gastric gas are present, or when images are acquired at different X-ray dose levels.

[0019] Therefore, according to the embodiments disclosed herein, a deep learning model such as a CNN as described above can be trained to output a transformation matrix that can be used to stitch two images together, wherein the deep learning model can be trained to output an accurate transformation matrix even in the presence of large lateral or vertical motion between images, non-rigid or rigid artifacts in the images, varying dose levels, and low overlap. In some embodiments, the deep learning model is trained with multiple training datasets, each training dataset including two training pairs and associated ground truth values. The training pairs may together include an image pair consisting of a first image and a second image, wherein the first image has a first field of view (FOV) and the second image has a second FOV. Each of the first and second images may capture a region of interest (ROI) with at least partial overlap of the FOVs. The ROI may be a rigid structure, such as a rigid anatomical feature (e.g., spine, tibia) or implant. The training pairs may also include one or more masks, wherein a first mask is generated based on the first image and / or a second mask is generated based on the second image. Each mask may be configured to identify the ROI applied to the registration image. Thus, a transformation matrix can be generated by the model. The transformation matrix can include one or more of the following functions: horizontal offset, vertical offset, rotation, skew, and scaling. The resulting transformation matrix can then be compared to the associated ground truth, and two or more loss functions can be used to compute the loss. The computed loss can then be used to improve the deep learning model. Enhanced training datasets can be used to further improve the deep learning model. By training the deep learning model using multiple training datasets, enhanced training datasets, and multiple loss functions, the trained deep learning model can tolerate lateral motion, non-rigid artifacts, foreign objects, dose invariance, etc.

[0020] A trained deep learning model can then be implemented to output a transformation matrix for stitching together image pairs (e.g., a first and a second image showing a common ROI and a partially overlapping FOV) into a single image that includes the complete ROI. Because the ROI is a rigid structure, generating a transformation matrix based on the ROI is more accurate than if the ROI is soft tissue (which might shift its position between the captures of the first and second images).

[0021] Thus, a universal image stitching system can be obtained, which allows and therefore can stitch together images including at least one of lateral motion, non-rigid artifacts, foreign objects, dose invariance, etc., so that the ROI and other elements of the first and second images are not distorted in the stitched image.

[0022] Figure 1 A block diagram of an exemplary embodiment of an image processing system is shown. Figure 1 The image processing system includes a neural network module configured with a neural network such as a CNN or a Siamese twin network. This neural network model can be trained and deployed by other modules of the image processing system to output a transformation matrix based on a first image and a second image of the object. The neural network can therefore be called a Transform Matrix Prediction Network (TMPN). The transformation matrix can be used to automatically stitch the first and second images together, where each of the first and second images captures a different view of the region of interest (ROI), such that the resulting stitched image includes the complete ROI. Figure 2 A high-level workflow describing the methods used to train and implement TMPN is shown. Figure 3A A first exemplary method for generating training pairs for training a TMPN is shown. The training pairs can be input into the TMPN to generate a transformation matrix, where the transformation matrix includes offset vectors (e.g., horizontal offset, vertical offset, rotation, skew, and scaling) generated using multiple training pairs. Figure 2 As shown, the loss can then be calculated by comparing the stitched image generated using the transformation matrix with the ground truth transformation matrix. A second exemplary method for training TMPN is shown in Figure 3B In this process, the image is input into the TMPN to generate a predicted offset vector. The predicted offset vector is compared to the ground truth offset vector, and a corresponding mask of the image is applied to both the predicted and ground truth offset vectors to scale the computed loss to the ROI. Figures 3A to 3B The loss calculated by the method can be used to... Figure 2 The verifier shown is used to improve TMPN. Figure 4 A flowchart is shown of an exemplary method for training a TMPN to generate transformation matrices based on training data, the method including training pairs optionally comprising corresponding masks, such as... Figures 3A to 3B As stated above. Figure 5 A flowchart is shown as an exemplary method for implementing a trained TMPN to generate a transformation matrix, stitching an input image based on the transformation matrix, and outputting a stitched image.

[0023] Figures 6A to 6D The image shows a stitched image comparing conventional methods used for automatic and manual image pasting. Figure 6A Images that are automatically pasted and manually pasted are compared, where there is lateral movement exceeding the lateral movement tolerance of the automatic image pasting method during or between image capture. Figure 6B Images that were automatically pasted and manually pasted were compared, including images containing non-rigid artifacts such as gastric gas. Figure 6C The images were compared between automatically pasted and manually pasted images, and the incorrect scheme was selected for automatic image pasting. Figure 6D Images that are automatically pasted and manually pasted, including images containing foreign objects, are compared. TMPN can be trained to tolerate the above challenges and stitch together images similar to those stitched using manual pasting methods. Figure 7 An exemplary specific implementation of the trained TMPN described herein is shown, wherein a first image and a second image can be stitched together into a single image using a transformation matrix generated by the trained TMPN.

[0024] Figure 1 A block diagram 100 illustrates an exemplary embodiment of an image processing system 102 according to one embodiment. In some embodiments, the image processing system 102 is integrated into an X-ray imaging system. For example, the image processing system 102 may be provided within the X-ray imaging system as both a processor and a memory for the X-ray imaging system. In some embodiments, at least a portion of the image processing system 102 is located at a device (e.g., an edge device, server, etc.) communicatively coupled to the X-ray imaging system via a wired and / or wireless connection. In some embodiments, at least a portion of the image processing system 102 is located at a separate device (e.g., a workstation) capable of receiving images from the X-ray imaging system or from a storage device storing images / data generated by the X-ray imaging system. The image processing system 102 may be operatively / communically coupled to a user input device 132 and a display device 134. In at least some examples, the user input device 132 may include a user interface for the X-ray imaging system, while the display device 134 may include a display device for the X-ray imaging system. In some implementations, user input device 132 and display device 134 may be located at a separate device (e.g., a workstation) that can receive images from the X-ray imaging system or from a storage device storing images / data generated by the X-ray imaging system.

[0025] Image processing system 102 includes a processor 104 configured to execute machine-readable instructions stored in memory 106. Processor 104 may be a single-core or multi-core processor, and programs executing thereon may be configured for parallel or distributed processing. In some embodiments, processor 104 may optionally include individual components distributed across two or more devices, which may be located remotely and / or configured for collaborative processing. In some embodiments, one or more aspects of processor 104 may be virtualized and executed by remotely accessible, networked computing devices configured in a cloud computing configuration.

[0026] Memory 106 may store neural network module 108, training module 110, inference module 112, image database 114, and preprocessing module 116. In some embodiments, as further described herein, preprocessing module 116 may include two modules, including a preprocessing module and a segmentation module. Neural network module 108 may include at least a deep learning model (e.g., a deep learning (DL) neural network), such as a CNN, and instructions for implementing the neural network to transform the output matrix based on the input, as detailed below. Neural network module 108 may include trained and / or untrained neural networks, and may also include various data or metadata relating to one or more neural networks stored therein.

[0027] The memory 106 may also store a training module 110, which may include instructions for training one or more neural networks stored in the neural network module 108. The training module 110 may include instructions that, when executed by the processor 104, cause the image processing system 102 to perform one or more steps of a method 500 for generating a training dataset for training one or more neural networks, as will be referred to below. Figure 4 To describe in more detail. Figures 3A to 3B Workflows 300 and 350 respectively provide additional or alternative methods for training neural network models. In some implementations, training module 110 may include instructions for implementing one or more gradient descent algorithms, applying one or more loss functions, and / or training routines to tune one or more neural networks of neural network module 108. Training module 110 may include training datasets for one or more neural networks of neural network module 108. In some examples, training module 110 may reside on a device different from the image processing system, such as an edge device, server, or workstation communicatively coupled to the image processing system.

[0028] Memory 106 also stores inference module 112. Inference module 112 may include instructions for deploying a trained deep learning model (e.g., CNN) to generate a transformation matrix based on input pairs. Specifically, inference module 112 may include instructions for causing image processing system 102 to execute when executed by processor 104. Figure 5 The instructions for one or more steps of method 600 are detailed below.

[0029] The memory 106 also stores an image database 114. The image database 114 may include, for example, X-ray images acquired via an X-ray imaging system. The image database 114 may include one or more training sets for training one or more neural networks of the neural network module 108. In some examples, the image database 114 may reside on a device different from the image processing system, such as an edge device, server, or workstation communicatively coupled to the image processing system.

[0030] The memory 106 also stores the preprocessing module 116. In some embodiments, such as Figures 3A to 3B As further described, the preprocessing module 116 may be two separate modules, including a preprocessing module and a segmentation module. The preprocessing module and the segmentation module may include instructions for preprocessing and segmenting the acquired images (such as images stored in the image database 114), respectively. Figures 3A to 3B As further described, the preprocessing module and the segmentation module can generate images and masks from the acquired images independently or in series, respectively, for generating transformation matrices and / or calculating losses to train neural network models.

[0031] In some embodiments, memory 106 may include components disposed on two or more devices, which may be located remotely and / or configured for collaborative processing. In some embodiments, one or more aspects of memory 106 may include remotely accessible networked storage devices configured in a cloud computing configuration.

[0032] User input device 132 may include one or more of the following: a touchscreen, keyboard, mouse, touchpad, motion-sensing camera, or other devices configured to enable a user to interact with and manipulate data within image processing system 102. In one example, user input device 132 may enable a user to select an image for training a machine learning model, or for further processing using a trained machine learning model.

[0033] Display device 134 may include one or more display devices utilizing virtually any type of technology. In some embodiments, display device 134 may include a computer monitor and may display ultrasound images. Display device 134 may be combined with processor 104, memory 106 and / or user input device 132 in a shared housing, or it may be a peripheral display device and may include a monitor, touch screen, projector or other display device known in the art, enabling a user to view X-ray images generated by the X-ray imaging system and / or interact with various data stored in memory 106.

[0034] It should be understood that Figure 1 The image processing system 102 shown is for illustrative purposes and not for limitation. Another suitable image processing system may include more, fewer, or different components.

[0035] See Figure 2 An example of a training and implementation system 200 for a Transform Matrix Prediction Network (TMPN) is shown. The training and implementation system 200 can be provided by, for example... Figure 1 One or more implementations of the image processing system 102 are used to train a deep neural network, such as a CNN, to generate transformation matrices based on input pairs. In one embodiment, the training and implementation system 200 includes a neural network module 240 to be trained, which may be part of a neural network training module 208 of the image processing system. Although described herein as a system, the training workflow and implementation workflow of the training and implementation system 200 may be separate workflows and may be stored on different systems and implemented by different modules, as further described herein. The neural network module 240 may be trained on multiple training datasets that may be stored in the data module 210. Each training dataset 242 may include multiple training pairs and ground truth target data.

[0036] In one implementation, each training pair may include a first image and a first mask based on the first image. For example, a first training pair includes a first image 244 and a first mask 246. A second training pair includes a second image 245 and a second mask 247. The first image 244 and the second image 245 may be medical images acquired by an imaging device. For example, a medical image may be an X-ray image acquired by an X-ray imaging system during examination of one or more anatomical structures of a patient. Each of the first image 244 and the second image 245 may capture different views of a region of interest (ROI) such as an anatomical structure (e.g., the spine). For example, the ROI may be a rigid structure present in both the first image 244 and the second image 245. In another example, the ROI may be identified based on a selected imaging scheme, wherein the selected imaging scheme indicates a predefined region, such as the spine, tibia, ribs, etc., that is at least partially present in both the first image 244 and the second image 245. Furthermore, the first image 244 captures a first field of view (FOV) of the object and the second image 245 captures a second FOV of the object, wherein the first FOV and the second FOV partially overlap and each of the first FOV and the second FOV partially or completely includes the ROI. Therefore, the stitching of the first image 244 and the second image 245 can generate a single image including the complete ROI, as further described herein.

[0037] The first image 244 and the first mask 246, the second image 245 and the second mask 247 of the training dataset 242 can be obtained from the training image dataset 228. The training image dataset 228 can be an image dataset generated by an image processing system performing an inspection on an object, or it can be an external image dataset such as a public dataset of medical images. The training image dataset 228 can be stored in an image database 214, such as... Figure 1 Image database 114.

[0038] In various implementations, the dataset generator 230 may select a first image 244, a first mask 246, a second image 245, and a second mask 247 and group them into a training dataset 242. The dataset generator 230 may generate the training dataset by selecting a first image (e.g., first image 244) and a second image (e.g., second image 245), wherein the first image and the second image include the same ROI and overlap by at least a certain amount of overlap; then designating the first image as first image 244 and the second image as second image 245. Furthermore, the dataset generator 230 may preprocess the first image 244 and the second image 245 to reduce differences and increase the correspondence between the first image and the second image, such as... Figures 3A to 3B As stated above.

[0039] A first mask 246 may be generated based on a first image 244, and a second mask 247 may be generated based on a second image 245. Each of the first mask 246 and the second mask 247 may be generated by a segmentation model, computer vision, or user annotation, as further described herein. A dataset generator 230 may segment the first image 244 and the second image 245 to generate the first mask 246 and the second mask 247 based on the respective images. Segmentation may be performed by an AI-based model (e.g., a neural network) trained to recognize and “segment” ROIs within an image. The segmentation module may be dedicated to a single type of ROI (e.g., spine, leg bone, implant). For example, the type of ROI in an image may be selected by the user of the image processing system (e.g., based on a selected scanning scheme). The segmentation module may then identify and segment the ROI of each image. The image processing system may be configured to detect when an incorrect scanning scheme is selected, e.g., when a mask cannot be generated by segmenting the image because the selected segmentation model is used for a different ROI than the one present in the image. For example, a first mask 246 may be generated based on a first image 244, wherein the first mask 246 is configured to identify a Region of Interest (ROI). A second mask 247 may be generated based on a second image 245, wherein the second mask 247 is configured to identify an ROI. Alternatively, the mask may be generated via user annotation or a computer vision-based model.

[0040] The mask may include a certain pixel value (e.g., a pixel value of 1) for all pixels within the ROI and different pixel values ​​(e.g., pixel values ​​of 0) for all pixels outside the ROI. The mask may be input to the model along with the first image 244 and the second image 245; for example, the first mask 246 may be concatenated with the first image 244 and the second mask 247 with the second image 245, such that each input includes two layers, one including image pixel values ​​and the other including mask values. The mask layer may inform the neural network module 240 that the spatially corresponding pixel values ​​from the image (e.g., within the ROI defined by the mask) are more important for determining the transformation matrix. In other examples, each mask may inform the neural network module not to evaluate certain regions of the image (e.g., outside the ROI defined by the mask). In this way, the network can be trained to apply filters only to portions of the image containing at least some of the ROIs. In still other examples, only one mask (e.g., the first mask 246) may be used.

[0041] Additionally, training pairs (e.g., a first training pair and a second training pair) of training dataset 242 may be assigned ground truth 248. The ground truth 248 of training dataset 242 may be a ground truth transformation matrix, wherein the transformation matrix includes offset vectors indicating horizontal and / or vertical offsets, skew, scaling and / or rotation, which may be applied to at least one of the first image 244 and the second image 245 to stitch the first image 244 and the second image 245 together without distortion of the ROI.

[0042] The training and implementation system 200 can be used to train the neural network module 240 to learn to recognize correspondences between input images (e.g., ROIs partially present in each of the input images) and predict transformation matrices that facilitate image stitching. In one example, the neural network module 240 may be stored in the neural network training module 208 and configured to receive a training dataset 242 from the data module 210, wherein a first image 244, a first mask 246, a second image 245, and a second mask 247 are input into the neural network module 240 to generate transformation matrices (e.g., including offset vectors) based on them. The generated transformation matrices and ground truth 248 can be used to improve the neural network module 240, such as... Figure 3A Further described in the text. In the second example, the first image 244 and the second image 245 are input into the neural network module 240 to generate a prediction transformation matrix, and the first mask 246 and the second mask 247 can be used to scale the calculated loss between the ground truth 248 and the prediction transformation matrix, such as... Figure 3B As further described in the text.

[0043] The neural network training module 208 may also include a validator 250 for validating the performance of the neural network module 240. The validator 250 may take the partially trained neural network module 240, the output of the neural network module 240 (e.g., a transformation matrix based on training dataset 242), and training dataset 242 (including ground truth 248 (e.g., a ground truth transformation matrix)) as input. The validator 250 may use the transformation matrix to stitch together a first image (e.g., first image 244) and a second image (e.g., second image 245) to generate a stitched image. The validator 250 may then compare the stitched image with the ground truth 248 to compute a loss and use the computed loss to adjust the weights and biases of the neural network module 240. In another example, the validator 250 may compare the transformation matrix with the ground truth 248 (e.g., a ground truth transformation matrix) to compute a loss and use the computed loss to adjust the weights and biases of the neural network module 240.

[0044] For example, a test dataset of 50 sets from training dataset 242 can be used to validate the partially trained neural network module 240 of the image processing system, where each of the 50 sets in training dataset 242 includes a first image and a second image of an object. Each set in training dataset 242 may include images of the same object from different fields of view (FOV) or images of different objects. The first image may be acquired at a first field of view (FOV), and the second image may be acquired at a second FOV different from the first FOV. Each of the first and second images may include at least a portion of a region of interest (ROI) such that there is ROI overlap when the first and second images overlap. Validator 250 may feed the first and second images into the partially trained neural network module 240 and receive a prediction transformation matrix that can be used to stitch the first and second images into a single image. In some examples, the validator may also feed a first mask and a second mask into the partially trained neural network module 240. Validator 250 may then compare a single stitched image with ground truth 248, which may be estimated by an expert (e.g., a manually stitched image of the first and second images). Validator 250 may then iteratively adjust one or more parameters (e.g., weights and biases) of neural network module 240 to minimize two or more loss functions based on the predicted transformation matrix until the error rate drops below a first threshold error rate. If the error between a single stitched image (e.g., based on the predicted transformation matrix) and the ground truth is below the threshold error, the partially trained neural network module 240 can be validated and determined to be fully trained, neural network training module 208 can output a trained / validated neural network module 260, and the training phase can end.

[0045] The training and implementation system 200 may include an inference module 212, which includes a trained / verified neural network module 260 (e.g., a neural network 240 that has been verified by the verifier 250 as described above). The inference module 212 may also include instructions for deploying the trained / verified neural network module 260 to generate one or more transformation matrices 270 for one or more sets of input data 222.

[0046] In some implementation schemes, see the following references Figure 5As detailed, one or more new sets of input data 222 may include a first image 224 and a second image 225. Each of the first image 224 and the second image 225 may be the result of a preprocessed acquisition image 220, as further described below. For example, the acquisition image 220 may be acquired during the inspection of an object via an imaging device 218 such as an X-ray imaging system and may be stored in an image database 214. Similar to the first image 244 and the second image 245 of the training dataset 242, the first image 224 and the second image 225 may each be acquired at different FOVs of the object, each including at least a portion of the ROI, such that there is an overlap of ROIs when the first image 224 and the second image 225 overlap.

[0047] The preprocessing module 216 may also include instructions for preprocessing the acquired image 220. The preprocessed acquired image 220 may be similar to the preprocessed images of the training dataset 242, as described above. Figures 3A to 3B As described above. In short, the acquired image 220 can be preprocessed to minimize the differences between image pairs (e.g., the first image 224 and the second image 225) and maximize correspondence for image registration purposes. For example, preprocessing may include adjusting the brightness or contrast of one or both images so that the two images have similar contrast and brightness.

[0048] As described above, input data 222 can be fed into a trained / validated neural network module 260 to generate a transformation matrix 270. In some examples, validating the neural network module 240 to obtain a trained / validated neural network module 260 may instruct the neural network module to be sufficiently trained to predict the transformation matrix. Alternatively, the trained / validated neural network module 260 may output a linear offset vector instead of a transformation matrix.

[0049] Transformation matrix 270 may include offset vectors indicating, for example, horizontal and vertical offsets, as well as rotation, skew, scaling, etc., which can be applied to register and stitch the first image 224 and the second image 225. Therefore, by applying transformation matrix 270, the images in the image pair (e.g., the first image 224 and the second image 225) can be stitched into a single image, and the elements of the first or second image are not distorted. Thus, the complete ROI partially shown in each of the first image 224 and the second image 225 is fully shown in the stitched image. Further details regarding the application of the transformation matrix for image stitching are described in... Figure 5 middle.

[0050] Figure 3A An exemplary workflow 300 is shown for training a TMPN to generate a transformation matrix using input pairs including an image and a mask. Figure 3BAn exemplary workflow 350 is shown for using images to train a TMPN to generate a predicted offset vector (e.g., a predicted offset vector of a transformation matrix) and for using masking and ground truth offset vectors to improve the predicted offset vector. Figures 3A to 3B This will also be described in this article.

[0051] Acquire images (e.g., Figure 2 The acquired image 220 can be acquired as a raw digital medical imaging and communication (DICOM) image 302 from an imaging device such as an X-ray imaging system, but other image formats are also within the scope of this application. The raw DICOM image 302 may contain the image acquired from the imaging device and identification data linking the image to the corresponding patient. The image can be acquired using an applied collimation method, wherein the X-ray beam of the X-ray imaging system can be confined to an area defined by the user of the X-ray imaging system. Collimation of the X-ray beam can reduce the patient dose (e.g., X-ray beam exposure) to unwanted areas (e.g., areas outside the area where the X-ray beam is confined). In addition, collimation can reduce X-ray beam scattering by eliminating unwanted areas, which in turn can improve the image quality (e.g., sharpness) of the X-ray image. Images acquired using a collimation method may therefore include collimation boundaries, wherein image data within the collimation boundaries (e.g., image data of the area where the X-ray beam is confined) may be brighter and have higher resolution than image data outside the collimation boundaries. Workflows 300 and 350 may include collimation removal 304, wherein regions outside the collimation boundaries of the acquired image (e.g., the original DICOM image 302) are cropped out, and the resulting image displays non-collimated image data (e.g., image data within the collimation boundaries). The resulting image may include image data containing user-defined anatomical regions of the original DICOM image 302.

[0052] The resulting image can be preprocessed by preprocessing module 306 to generate image 308 and segmented by segmentation module 316 to generate mask 318. Preprocessing may include adjusting the brightness or contrast of the images so that when input into a TMPN containing a second image, the two images have similar contrast and brightness. In one example, the image brightness and / or contrast may be predetermined amounts, for example, set by parameters of the TMPN. In another example, the image brightness and / or contrast values ​​may be input by a user into the image processing system, and the preprocessing module may adjust the brightness and / or contrast of each image to equal input values. Preprocessing may also include additional or different adjustments to the images to reduce differences and increase the correspondence between images in a pair of images to be input into the TMPN 320 (e.g., a first image and a second image with the same ROI and partially overlapping FOV).

[0053] Mask 318 is configured to identify (e.g., by setting pixel values ​​in the mask to zero) all pixels of image 308 outside the Region of Interest (ROI). Thus, when mask 318 is used during training, it informs the CNN (e.g., TMPN) which pixels of the input image are important for generating the transformation matrix / offset vector. In this way, the CNN can be trained to rely on the ROI used to generate the transformation matrix, rather than the image data outside the ROI.

[0054] In one example, preprocessing module 306 and segmentation module 316 may be a single module. In another example, preprocessing module 306 and segmentation module 316 may be separate modules. In either example, segmentation module 316 may be omitted, and a mask 318 may be generated using computer vision or user annotations of the corresponding image.

[0055] In one example, such as shown in workflow 300, the resulting image 308 and mask 318 can be training pair 310. Training pair 310 can be Figure 2 One of two training pairs in the training dataset 242. In workflow 300, training pair 310 may be fed into TMPN 320 along with a second training pair (e.g., generated using workflow 300 or a different workflow) to compute loss 322.

[0056] In another example, such as shown in workflow 350, the resulting image 308 can be input into TMPN 320 to generate offset vector 323. For example, when training TMPN 320, inputting the first image 308 into TMPN 320 yields predicted offset vector 324. This can be equated to... Figure 2 The ground truth offset vector 326 of the ground truth 248 is compared with the predicted offset vector 324 to calculate the loss 328 of TMPN 320. Calculating the loss 328 may include using a mask 318 to scale the loss function value, such that the calculation of the loss 328 is based on image data within the ROI, such as those defined by the mask 318. For example, the mask 318 may be applied to zero out the loss associated with regions outside the ROI, such that only the loss associated with the ROI, leg bone, spine, etc., will be used for that loss, and thus the network parameters will implicitly learn to prioritize these regions. In one example, the predicted offset vector output by the CNN can be used to move the input image, and the ground truth offset vector can be used to move the same image, then the difference between the first moved image and the second moved image can be determined. The mask can then be used to consider only the differences in the unmasked regions of the image, rather than comparing all differences in pixel values ​​across all regions of the image.

[0057] Workflows 300 and 350 illustrate the training of the TMPN 320. For example, workflow 350 can be implemented at least twice to generate two images 308, each with a different field of view (FOV) of the object, and each FOV including a common region of interest (ROI). Loss 328 can be used to train the TMPN 320 such that the predicted offset vector 324 is equal to the ground truth offset vector 326. The trained TMPN 320 can then be implemented to generate a transformation matrix based on the input image pairs.

[0058] Workflows similar to those in Workflow 300 and Workflow 350 can be applied to the implementation of TMPN 320, such as... Figure 5 As further described below. However, the masking may only be used during training, for example, as shown in workflows 300 and 350, and may not be used during inference of the trained TMPN. The steps in workflows 300 and 350 prior to inputting image 308 into TMPN 320 illustrate the processing of a single image. An image pair comprising two images (as described above) generated using the preprocessing module 216 and having the same ROI partially overlapping with the FOV may be input into a trained neural network module to generate a transformation matrix based on the image pair. The transformation matrix may then be used to stitch the first and second images together to form a stitched image comprising the ROIs of the first and second images. The stitched image may be output on a display device and / or stored in memory.

[0059] Workflows 300 and 350 can be examples of training the TMPN320 using multiple training datasets, where each training dataset includes an image pair, a mask or a pair of masks generated from the image pair, and associated ground truth values, and where the model is trained by inputting the image mask pair as input to the model or by performing loss scaling using the masks. The TMPN 320 can additionally or alternatively be trained using multiple augmented training datasets. Augmented training datasets can be generated from corresponding training datasets in the multiple training datasets, and the augmented training datasets include augmented versions of the image pairs and the same associated ground truth values ​​as in the corresponding training datasets in the multiple training datasets. The augmented versions of the image pairs may include augmentations including at least one of local image gamma adjustment, local image brightness adjustment, foreign object artifacts, and overlap adjustment. In the presence of foreign objects, gas, dosage changes, etc., a series of augmentations can be used to train the robustness of the network. Figure 4 The enhanced training data is further described in the paper. This yields a robust TMPN, which is applicable to various anatomical structures, dose variations, etc.

[0060] The training neural network module may therefore include inputs, including a first training pair and a second training pair fed into the AI ​​model. Each of the first and second training pairs includes an image and a mask based on that image. In one example, the ROI of the first and second training pairs is the spine of an object. For example, the first training pair may show the top region of the spine (e.g., near the object's shoulders), while the second training pair may show the middle / bottom region of the spine (e.g., near the object's stomach and hips). As described above, the mask is configured to mask pixels of the corresponding image outside the ROI. For example, the mask of the corresponding image masks pixels of an image showing anatomical structures other than the spine, such that when the corresponding mask is applied to an image in the AI ​​model, convolution can be performed only on the image data depicting the spine.

[0061] The AI ​​model, referred to herein as TMPN, may include at least one DL neural network architecture. For example, the DL neural network could be a Siamese twin and a regression neural network, a Homography network, a CNN, or another suitable model. The AI ​​model is also trained with two or more losses, which may include grid loss, mean squared error (MSE) / root mean squared error (RMSE), and location-based loss. A combination of one or more loss functions and regularization can be used to ensure regression. For example, grid loss can be combined with MSE to increase the accuracy of affine parameter predictions, where grid loss ensures overall rigidity of the transformation. Because the transformation matrix may include multiple transformations (e.g., linear offset vectors, rotations, scaling, etc.), applying more than one loss function may be advantageous, as different loss functions can increase the accuracy of predictions for different transformations of the transformation matrix.

[0062] A trained AI model (e.g., TMPN) can be implemented to generate a transformation matrix. For example, the transformation matrix may include an offset vector, which may indicate horizontal or vertical offset, scaling, rotation, skew, etc., to be applied to the input (e.g., at least one image of the input), such that the images of the first training pair and the second training pair can be stitched together into a single image.

[0063] Figure 4 A flowchart of an exemplary method 500 for training a TMPN to generate a transformation matrix based on input data is shown. Method 500 can be used for training a TMPN. Figures 3A to 3B Examples of workflows and methods, and combined with Figure 1 The configuration has Figure 2 The image processing system for training and implementing the TMPN is described herein. Instructions for method 500 and other methods described herein can be stored in the memory of the image processing system and implemented as described herein. For example, a processor such as Figure 1The processor 104 can be configured to execute method 500 stored in training module 110 to train TMPN.

[0064] At 502, method 500 includes generating training datasets, where, at 504, each training dataset includes overlapping image pairs, mask pairs, and ground truth transformation matrices. (Reference) Figures 2 to 3B The training dataset can therefore include two training pairs, each with an image and a corresponding mask. For example, an overlapping image pair may include a first image and a second image having a rigid structure (e.g., a Region of Interest) present in both the first and second images. The first and second images may each show different FOVs of the ROI, with the first FOV of the first image and the second FOV of the second image partially overlapping.

[0065] The mask pair may include a first mask based on a first image and a second mask based on a second image. For example, a segmentation module such as Figures 1 to 3B The segmentation module described herein generates a first mask based on a first image via user annotations or a computer vision-based module. A second mask can be generated based on a second image using one of the methods for generating the first mask. The same or different methods can be used to generate both the first and second masks. As described above, the mask is configured to block all pixels of the corresponding image outside the ROI (if the mask is applied to the corresponding image).

[0066] The ground truth transformation matrix can be generated by the user or other experts in the field, and a series of horizontal offsets, vertical offsets, scaling, skew, rotations, etc. can be defined, which can be applied to at least one of the first image and the second image to stitch the first image and the second image into a single image at the overlap of the first image and the second image.

[0067] Additionally, generating the training dataset at position 502 may include generating an enhanced version of the training dataset with overlapping image pairs at position 506. Enhancing the overlapping image pairs may include one or more of device-related adjustments, patient-related adjustments, and process-related adjustments applied to one or more of each image (e.g., the first and second images). For example, enhancement may include at least one of local image gamma adjustment, local image brightness adjustment, foreign object artifact correction, overlap adjustment, etc. Device-related changes may include local or global intensity changes. Patient-related changes may be inclusion of metal artifacts. Process-related adjustments may be adjustments to the degree of overlap between images. Including an enhanced training dataset in addition to the training dataset increases the accuracy and robustness of TMPN, enabling TMPN to accurately generate transformation matrices when at least one of the input images includes foreign objects, gastric gas, radiation dose changes, etc. When images are enhanced, the ground truth may remain unchanged, such that the ground truth is based on the unenhanced image.

[0068] At 508, method 500 includes inputting training data, which may include a training dataset and an augmented training dataset, into the untrained TMPN. At 510, method 500 includes training the TMPN using the input training data. At 514, training the TMPN includes pairing each mask with a corresponding image to form the training input to the network. In one example, each mask may be concatenated with a corresponding image and input into the model as input. In other examples, masks may be used for loss scaling and only images may be input into the model.

[0069] At 516, training the TMPN also includes using two or more loss functions to compute the loss between the output layer and the ground truth transformation matrix. For example, loss functions may include grid loss, MSE / RMSE, and location-based loss. The output layer can be a transformation matrix generated by the TMPN and may include horizontal offset, vertical offset, rotation, skew, and / or scaling. The ground truth transformation matrix can be generated via user annotation, a computer vision-based module, or an alternative transformation matrix generation method to the TMPN. For example, when generated via user annotation, images can be manually stitched together, and a computer program can be used to generate a transformation matrix that can be used to stitch the images to obtain a stitched image equivalent to the manually stitched image. The ground truth transformation matrix may include one or more horizontal offsets, vertical offsets, rotations, skews, and scaling, which, when applied to at least one of the images in an image pair (e.g., at least one of the first and second images), allows the images to be stitched into a single image that includes the complete ROI (e.g., partially displayed in different FOVs in each of the first and second images). In this way, the ground truth transformation matrix can be identified as the transformation that accurately stitches together the input image when applied. By calculating the loss between the output layer and the ground truth transformation matrix, the accuracy of the output layer transformation matrix can be determined, and thus the accuracy of TMPN can be determined.

[0070] At 518, method 500 includes updating the weights and biases of the TMPN based on the loss. This improves the TMPN so that the generated transformation matrix can be used to accurately stitch the input images into a single stitched image including the complete ROI. After performing the convolution, at 518, the weights and biases of the TMPN can be updated based on the loss calculated at 516 and based on the application of a corresponding mask. For example, a corresponding mask can be applied to scale the loss function value, such that the weights and biases are updated using the loss calculated based on image data within the ROI as defined by the corresponding mask.

[0071] As explained in this article, a transformation matrix can include multiple transformations (e.g., six transformations). However, in some examples, some transformations may be more important for stitching images than others. To simplify training, one or more parameters of the transformation matrix can be "masked" to remove those parameters from the resulting transformation matrix. For example, in some cases, translating the image up or down (y-axis translation) may be preferred, and in such cases, training can be simplified by ignoring / setting zero transformation matrix parameters that do not affect the y-axis translation.

[0072] Method 500 returns to begin repeating and further training of the TMPN. The TMPN can be trained using multiple training datasets and augmented training datasets. In one example, the weights and biases of the TMPN can be updated based on the loss until the error from the loss is below a threshold, at which point it is determined that the transformation matrix generated by the TMPN is sufficiently accurate (e.g., the ROI of the stitched image can be interpreted by the user or other methods and used for patient diagnosis or treatment). In another example, the weights and biases can be updated against a certain number of training datasets and augmented training datasets (e.g., 1000 training datasets), after which it can be determined that the TMPN has been trained and can be implemented to generate the transformation matrix for stitching images.

[0073] After training the TMPN, the TMPN can be implemented to generate a transformation matrix for input image pairs, where the images in the input image pairs show different FOVs of the ROI, with the FOVs partially overlapping. Figure 5 A flowchart of an exemplary method 600 for implementing a trained TMPN is shown. Method 600 can be derived from... Figure 1 The inference module 112 is implemented.

[0074] At 602, method 600 includes generating input data. The input data may be generated using a first acquired image and a second acquired image (such as...). Figure 2 Acquired images 220 and / or Figures 3A to 3B The original DICOM image 302 is generated. The first and second acquired images may include an overlapping image pair, wherein the FOV of the first acquired image and the FOV of the second acquired image partially overlap.

[0075] Each of the first and second acquired images can be preprocessed, such as... Figures 3A to 3BThe method described above identifies the Region of Interest (ROI). At 604, the ROI is identified for the first and second acquired images in the overlapping image pair. In one example, the ROI can be identified based on the selected imaging scheme, where the user of the imaging device can select an anatomical structure as the ROI. As mentioned above, the ROI is a rigid structure present in the two images. Preprocessing can further reduce the difference between the first and second acquired images and increase their correspondence (e.g., equalize contrast, brightness, etc.) to obtain the first and second images respectively. The input data is defined as an overlapping image pair (e.g., a preprocessed first acquired image and a preprocessed second acquired image). In an example of training the TMPN using one or more masks as input (e.g., as in...), Figure 3A In the workflow, the input data input during inference may include one or more zero matrices. For example, each image may be paired / concatenated with a zero matrix, allowing TMPN to receive the expected input based on training, but the input "mask" does not include information; for example, the image is input along with one or more matrices of the expected size of the mask, but with all zero values ​​instead of meaningful values.

[0076] At 610, method 600 includes feeding input data into the trained TMPN. For example, the TMPN can be... Figures 3A to 5 The TMPN is trained using at least one of the methods described herein. When a CNN is configured within the TMPN, convolution can be performed on the image. The trained TMPN outputs a transformation matrix based on the input data, wherein the transformation matrix includes one or more of horizontal offset, vertical offset, rotation, skew, and scaling.

[0077] At position 612, the first and second images are stitched together based on a transformation matrix. Stitching the first and second images may include applying one or more horizontal offsets, vertical offsets, rotations, skews, and scaling of the transformation matrix to at least one of the first and second images. In one example, horizontal offsets and scaling of the first frame may be applied to the first image, while vertical offsets and scaling of the second frame (e.g., wider than the first frame) may be applied to the second image. Applying the transformation matrix to at least one of the first and second images allows for alignment of overlapping areas of the images, such that the complete Region of Interest (ROI) (e.g., the ROI partially shown in each of the first and second images) is displayed in the stitched image in a manner that does not distort the ROI.

[0078] The methods described herein can be used to stitch together two or more images. In one example, stitching together n images using the transformation matrix generated as described above, for a series of images from image 1 to image n, the stitching includes image 1 at the top of the initial stitched image. For the images below, the offset vectors indicate the vertical and horizontal offsets between adjacent images. The total length of the stitched image is the sum of the vertical dimensions of the images between image 1 and image n minus the sum of the vertical offset vectors of the images between image 1 and image n. The left-hand side of the stitched image is the rightmost left sub-image edge of the initial stitched image. The right-hand side of the stitched image is the leftmost right sub-image edge of the initial stitched image.

[0079] At point 614, the stitched image is output to a display device (such as display device 134) and / or stored in memory (e.g., stored in storage). Figure 1 (In the image database 114). In this way, input images that will display partial views of ROIs with different partially overlapping FOVs can be stitched together using the transformation matrix generated from the images, so that the ROI is fully displayed on the stitched single image without distortion.

[0080] Figure 5 The method described herein can be an example of implementing TMPN to generate a transformation matrix. The method described herein, in which images are input into a model trained to output a transformation matrix based on the images, can be implemented using other or alternative methods. For example, TMPN can use convolutional blocks along with pooling, normalization, and attention to predict the transformation matrix. Alternatively or alternatively, TMPN can be trained with at least one auxiliary task, such as classifying anatomical views, which can be used to improve TMPN. The addition of an auxiliary task can include auxiliary outputs to the network, where additional layers of the neural network target the auxiliary task. In this example, the input may not change (e.g., the training data is the same as TMPN training without an auxiliary task). Additional ground truth data can be included for training the auxiliary task. For example, TMPN can be trained using a linear combination of a first task (e.g., generating a transformation matrix) and an auxiliary task. The auxiliary task can provide additional attention to the input image to improve the performance of the first task. The auxiliary task can also solve additional downstream tasks, such as view classification (e.g., frontal, side, etc.). The addition of auxiliary tasks is independent of the network and can be used with any deep learning model (e.g., Siamese twin and regression neural networks, Homography networks, CNNs, or another suitable model).

[0081] In another example, TMPN can use either a single-image patch approach or a multi-image patch approach to predict the transformation matrix. For example, a single-image patch approach may involve inputting a single image containing anatomical landmarks (e.g., the spinal cord) into a model such as a TMPN configured with a CNN or other AI model, and outputting a single offset vector. A multi-image patch approach may involve making an overall prediction across multiple image pairs of patches (e.g., dividing each image into patches and making predictions for each corresponding set of patches) to determine the final transformation matrix using a weighted scheme. In some examples, the weighted scheme may generate a mean transformation matrix where each image maintains equal weights. In one example, image pasting may be used to paste images in parallel (e.g., instead of stitching the bottom edge of a first image to the top edge of a second image). In this example, multiple transformation matrices may be generated to reduce or adjust for disparity errors at different locations in the input image pair.

[0082] In some examples, during the execution of method 600, the image processing system may be configured to detect whether the user has selected an incorrect scanning protocol for imaging the patient. For example, various imaging system parameters, including image registration and stitching, and / or post-acquisition processing parameters, may be selected based on the scanning protocol, which may indicate the anatomical structure being imaged (e.g., spine, leg, etc.). If the user has selected a leg protocol instead of imaging the patient's spine, the image processing system may detect the incorrect protocol selection based on the segmentation module's inability to identify the ROI (e.g., leg bone, based on the selected protocol) and therefore the TMPN cannot output a transformation matrix. When the segmentation module cannot identify the ROI, a notification indicating that an incorrect scanning protocol has been selected may be output to the user, enabling the user to select an appropriate scanning protocol and rescan the patient with the desired settings tailored for the correct scanning protocol.

[0083] Conventional methods for automatically stitching images can encounter technical challenges when the input images each possess different characteristics, such as different fields of view (FOV), lateral motion between images, images generated using different X-ray doses, the presence of non-rigid artifacts, incorrect selection of the scheme used to identify the region of interest (ROI), and low radiation doses. The method described in this paper for automatically stitching images together using a trained model to generate a transformation matrix based on overlapping images to form a stitched image involves training the model (e.g., TMPN) to be tolerant of the aforementioned challenges. Figures 6A to 6D Exemplary images comparing automatically pasted (e.g., stitched) images generated using conventional image auto-pasting methods with images generated by manual pasting are shown.

[0084] Figure 6AAn exemplary image 700 comparing automatic pasting 702 and manual pasting 704 for a pair of X-ray images is shown, where lateral movement exceeds the lateral movement tolerance of a conventional algorithm. Lateral movement can be defined as movement in the horizontal direction. In the event of lateral movement, automatic pasting 702 of an image performed by a conventional method may result in inaccurate image registration. For example, the object being imaged (e.g., a patient) may move between the capture of a first image and the capture of a second image. Lateral movement may be due to physical movement of the object or movement of the imaging device, such as framing an area to be imaged differently (e.g., indicated by circle 706) or an offset view capturing that area. When lateral movement exceeds a first offset threshold (e.g., 5 cm), a conventional automatic image pasting algorithm may result in an offset image, as shown in automatic pasting 702. Manual pasting 704 of the same image that has been automatically pasted 702 shows that the first and second images are correctly aligned. As indicated by circle 706 in automatic pasting 702, the stitched first and second images may be offset. In manual pasting 704, the stitched first and second images may be aligned, as indicated by circle 706. The method disclosed herein for generating a transformation matrix and stitching a first image and a second image together based on the generated transformation matrix can be trained to adapt to arbitrary lateral offsets. For example, the method described herein can be trained for a wide range of horizontal and vertical offset values ​​(e.g., allowing horizontal and vertical offsets greater than 5 cm). Thus, the transformation matrix prediction network can generate a transformation matrix that, when applied to at least one of the first and second images, can stitch the first and second images together to align the regions of interest, similar to the manual pasting shown in exemplary image 700 704.

[0085] Furthermore, training the disclosed method to adapt to arbitrary lateral offsets allows the method to generate transformation matrices for image pairs (e.g., a first image pair and a second image pair) where the images in the image pair have low overlap. For example, low overlap might mean that the first FOV of the first image overlaps with the second FOV of the second image by approximately 3.5 cm. This allows the method to be performed while reducing the radiation dose to the patient, for example, by capturing fewer images with low overlap to image regions captured using more images with greater (e.g., greater than or equal to 4.5 cm) overlap by conventional methods.

[0086] Figure 6BAn exemplary image 720 is shown comparing automatic pasting 708 and manual pasting 710 for X-ray images containing non-rigid artifacts such as gastric gas (e.g., shown in circle 712). Conventional methods for automatic image stitching may be search-based and may not be trained to compensate for non-rigid artifacts within the image. For example, non-rigid artifacts such as gastric gas can move and change shape / volume between and during image captures. Therefore, when implementing conventional automatic image stitching methods, the first and second images may be misregistered during automatic pasting 708. Structures within the first and second images that may be partially or completely occluded by non-rigid artifacts during at least a portion of the image capture may be misregistered. Manual pasting 710 shows a stitched image of the first and second images where the region of interest is aligned despite the presence of non-rigid artifacts. The method disclosed herein for generating a transformation matrix and stitching a first image and a second image together based on the transformation matrix can be trained to adapt to non-rigid artifacts including gastric gas, such that the transformation matrix prediction network can generate a transformation matrix that, when applied to at least one of the first image and the second image, can stitch the first image and the second image together to align the regions of interest, similar to the manual pasting 710 shown in the exemplary image 720.

[0087] Figure 6C Exemplary image 740 is shown comparing automatic pasting 714 and manual pasting 716 for selecting pairs of X-ray images using an incorrect scheme employing conventional automatic image pasting methods. Figure 6C In the example, a leg scheme is selected for stitching the first and second images together, where the ROI is the spine. Therefore, the spine scheme may include a feature finder and / or search box extraction for spine features not present in the leg scheme. Automatic pasting 714 may result in poor registration when an incorrect scheme is selected, as shown in circle 718. Manual pasting 716 shows the resulting stitched image, where the first and second images have been correctly registered, as shown in circle 718. The method disclosed herein for generating a transformation matrix and stitching the first and second images together based on that transformation matrix can be trained to be invariant to the selected scheme and robust to incorrect scheme selection, for example, during object scan image capture and / or automatic stitching. Thus, the generated transformation matrix prediction network can generate a transformation matrix that, when applied to at least one of the first and second images, can stitch the first and second images together to align the regions of interest, similar to the manual pasting 716 shown in exemplary image 740.

[0088] Figure 6DAn exemplary X-ray image 760 including a foreign object is shown. The combination of parallax artifacts and the foreign object is shown in a first image 722, which is the result of stitching together a first image and a second image using a conventional automatic image pasting method. For example, circle 724 shows a metal object in the X-ray image, which may have been correctly pasted when stitching the first and second images. Circle 726 shows vertebral misregistration (e.g., inaccurate pasting), which may be due to the presence of a metal object in circle 724. Second image 728 shows multiple foreign objects, such as metal attachments around the pelvis, which may cause a conventional automatic image pasting method to fail to stitch the first and second images into a correctly registered final image. The foreign object may be distinguishable from implanted metal or a pacemaker, thus allowing the automatic image pasting method to be trained in some embodiments to remove foreign objects from the pasting series. In other embodiments, the automatic image pasting method may be intentionally trained not to remove foreign objects that may aid in user-defined measurements, such as, for example, measuring tools or guide signs. The method disclosed herein for generating a transformation matrix and stitching a first and a second image together based on that transformation matrix can be trained to be invariant to known foreign objects and to consider vertebrae when obtaining registration vectors (such as offset vectors of the transformation matrix). Thus, the generated transformation matrix prediction network can generate a transformation matrix that, when applied to at least one of the first and second images, can stitch the first and second images together to align the regions of interest.

[0089] Further clinical challenges can be addressed by training the transformation matrix prediction network to compensate for dose invariance and low radiation dose. For example, as... Figure 4 The aforementioned enhanced training dataset, including gamma adjustment, can train the TMPN to tolerate dose invariance and low radiation. Additionally or alternatively, training pairs used to train the TMPN may include images captured at different doses. Thus, the TMPN can be trained to stitch together images with varying dose levels (e.g., a first image with a first dose level and a second image with a second dose level).

[0090] Figure 7An exemplary image 800 is shown, illustrating an image 810 comparing an input image pair and a resulting stitched image 820 according to the method of this disclosure. A first image 802 shows an X-ray image of the top portion of the spine, and a second image 804 shows an X-ray image of the lower portion of the spine. The first image 802 and the second image 804 are stitched together according to the method described above to form a single stitched image 820. The stitched image 820 shows aligned vertebrae of the spine. Furthermore, portion 812 of the first image 802 is not shown in the region of the first image 802 of the stitched image 820 because portions 812 and 814 of the second image are overlapping FOV regions (e.g., portion 814 overlaps with portion 812), and are therefore the region where the first image 802 and the second image 804 are stitched together.

[0091] The technical advantages of training and implementing a deep neural network model to generate transformation matrices based on a first and second image, and to stitch the first and second images together using the transformation matrices, include reduced operator error, higher accuracy in automatically stitching X-ray image pairs compared to manual image stitching, reduced operator training costs, and increased patient diagnostic efficiency. The method is also trained for low overlap between scans, non-rigid objects, foreign bodies, and lateral offsets, resulting in reduced radiation dose to the patient. Therefore, the disclosed model can align and stitch images even with the aforementioned challenges (e.g., low overlap, lateral offsets, etc.), leading to fewer image retakes and thus improving the efficiency of the X-ray system and the computing device executing the model by reducing the number of image capture, registration, and stitching operations.

[0092] This disclosure also provides support for a method comprising: training a model based on a transformation matrix of the first and second images of an object as input; wherein the model is trained on multiple training datasets, each training dataset including an image pair, a mask indicating a region of interest (ROI), and associated ground truth values; automatically stitching the first and second images together based on the transformation matrix to form a stitched image; and outputting the stitched image for display on a display device and / or storing the stitched image in memory. In a first example of the method, the method further includes generating a mask based on the images from the image pair using a segmentation model, computer vision, or user annotations. In a second example of the method, optionally including the first example, each training dataset further includes a second mask generated based on another image from the image pair. In a third example of the method, optionally including one or both of the first and second examples, the ROI is a rigid structure present in the two images of the image pair. In a fourth example of the method, optionally including one or more of the first to third examples, or each of them, the model is a convolutional neural network (CNN) trained using two or more loss functions. In a fifth example, which optionally includes one or more of the first through fourth examples, each mask is paired with a corresponding image in an image pair to form the training input for the CNN. In a sixth example, which optionally includes one or more of the first through fifth examples, the mask is used to scale the loss function values ​​during training. In a seventh example, which optionally includes one or more of the first through sixth examples, the ground truth comprises a training transformation matrix generated based on the image pairs, wherein the model is trained using two or more loss functions, and wherein the two or more loss functions include two or more of grid loss, location-based loss, and mean squared error loss. In an eighth example, which optionally includes one or more of the first through seventh examples, multiple training datasets include data augmentation, which includes one or more of device-related adjustments, patient-related adjustments, and process-related adjustments applied to one or more images. In a ninth example, which optionally includes one or more of the first through eighth examples, the transformation matrix includes one or more of horizontal offset, vertical offset, rotation, skew, and scaling. In a tenth example, which optionally includes one or more of the first to ninth examples, the method further includes acquiring a first image using an imaging system with a first field of view (FOV) positioned to capture the object and acquiring a second image using an imaging system with a second FOV positioned to capture the object, wherein the first FOV and the second FOV overlap only partially. In an eleventh example, which optionally includes one or more of the first to tenth examples, the imaging system is an X-ray imaging system.

[0093] This disclosure also provides support for an image processing system comprising: a deep learning model trained on multiple training datasets, each training dataset including image degrees, mask pairs indicating regions of interest (ROIs) in the image pairs, and associated ground truth values; and a processor communicatively coupled to a memory storing the deep learning model and including instructions that, when executed, cause the processor to: train a deep learning model based on a first image and a second image of an object as input to output a transformation matrix of the first and second images; receive the transformation matrix based on the first and second images as output of the deep learning model; automatically stitch the first and second images together based on the transformation matrix to form a stitched image; and output the stitched image for display on a display device and / or store the stitched image in the memory. In a first example of the system, the deep learning model is trained using two or more loss functions. In a second example of the system, optionally including the first example, the memory also stores a segmentation model configured to output mask pairs based on the image pairs.

[0094] This disclosure also provides support for a method for an X-ray imaging system, the method comprising: acquiring a first image and a second image of an object using the X-ray imaging system; taking the first and second images as inputs and training the inputs to output a transformation matrix based on the first and second images, wherein the model is trained with multiple training datasets, each training dataset including a first mask of the first training image and a second mask of the second training image based on regions of interest (ROIs) in the first and second training images; automatically stitching the first and second images together based on the transformation matrix to form a stitched image; and outputting the stitched image for display on a display device and / or storing the stitched image in memory. In a first example of the method, the first and second masks are generated using a segmentation model, a computer vision-based model, or based on user input. In a second example of the method, optionally including the first example, the model is trained by taking the first training image, the second training image, the first mask, and the second mask as inputs to the model or by performing loss scaling using the first and second masks, and wherein each training dataset includes associated ground truth values. In a third example, which optionally includes one or both of the first and second examples, the model is further trained with multiple augmented training datasets, each augmented training set generated from a corresponding training dataset in the multiple training datasets and including augmented versions of the first and second training images, as well as the same associated ground truth values ​​from the corresponding training datasets in the multiple training datasets. In a fourth example, which optionally includes one or more of the first to third examples, the augmented versions of the first and second training images include enhancements including at least one of local image gamma adjustment, local image brightness adjustment, foreign object artifact adjustment, and overlap adjustment.

[0095] As used herein, elements or steps listed in the singular and beginning with the word "a" or "an" should be understood to not exclude a plurality of said elements or steps unless such exclusion is explicitly stated. Furthermore, references to "one embodiment" of the invention are not intended to be construed as excluding the existence of additional embodiments that also include the referenced features. Moreover, unless explicitly stated to the contrary, embodiments that "comprise," "include," or "have" elements or multiple elements having a particular characteristic may include additional such elements that do not have that characteristic. The terms "comprise" and "in..." are used as concise linguistic equivalents to the corresponding terms "comprising" and "wherein". Furthermore, the terms "first," "second," and "third," etc., are used merely as notations and are not intended to impose numerical requirements or a particular order of position on their objects.

[0096] This written description uses examples to disclose the invention, including the best mode, and also enables those skilled in the art to practice the invention, including making and using any device or system and performing any included methods. The scope of patentability of the invention is defined by the claims and may include other examples that would occur to those skilled in the art. Such other examples are intended to fall within the scope of the claims if they have structural elements that are not indistinguishable from the literal language of the claims, or if they include equivalent structural elements that have minor differences from the literal language of the claims.

Claims

1. A method for registering images, the method comprising: The first image and the second image of the object are input to train a model based on the first image and the second image to output a transformation matrix, wherein the model is trained with multiple training datasets, each training dataset including image pairs, masks indicating regions of interest, and associated ground truth values; The first image and the second image are automatically stitched together based on the transformation matrix to form a stitched image; as well as The stitched image is output to be displayed on a display device and / or stored in a memory.

2. The method of claim 1, further comprising generating the mask based on images from the image pair using a segmentation model, computer vision, or user annotations.

3. The method of claim 2, wherein each training dataset further includes a second mask generated based on another image from the image pair.

4. The method of claim 3, wherein the region of interest is a rigid structure present in both images of the image pair.

5. The method of claim 3, wherein the model is a convolutional neural network trained using two or more loss functions.

6. The method of claim 5, wherein each mask is paired with a corresponding image in the image pair to form the training input of the convolutional neural network.

7. The method of claim 1, wherein the mask is used to scale the loss function value during training.

8. The method of claim 1, wherein the ground truth comprises a training transformation matrix generated based on the image pair, wherein the model is trained using two or more loss functions, and wherein the two or more loss functions comprise two or more of grid loss, location-based loss, and mean squared error loss.

9. The method of claim 1, wherein the plurality of training datasets includes data augmentation, the data augmentation comprising one or more of device-related adjustments, patient-related adjustments, and process-related adjustments applied to one or more images.

10. The method of claim 1, wherein the transformation matrix comprises one or more of horizontal offset, vertical offset, rotation, skew, and scaling.

11. The method of claim 1, further comprising acquiring the first image with an imaging system positioned to capture a first field of view of the object and acquiring the second image with the imaging system positioned to capture a second field of view of the object, wherein the first field of view and the second field of view overlap only partially.

12. The method of claim 11, wherein the imaging system is an X-ray imaging system.

13. An image processing system, the image processing system comprising: A deep learning model, which is trained using multiple training datasets, each training dataset including image pairs, mask pairs indicating regions of interest in the image pairs, and associated ground truth values; and A processor, communicatively coupled to a memory storing the deep learning model and including instructions that, when executed, cause the processor to: The first image and the second image of the object are input to train the deep learning model to output the transformation matrix based on the first image and the second image; As the output of the deep learning model, the transformation matrix based on the first image and the second image is received; The first image and the second image are automatically stitched together based on the transformation matrix to form a stitched image; as well as The stitched image is output to be displayed on a display device and / or stored in a memory.

14. The image processing system of claim 13, wherein the deep learning model is trained using two or more loss functions.

15. The image processing system of claim 13, wherein the memory further stores a segmentation model configured to output the mask pair based on the image pair.

Images