System and methods for automatic image alignment of three-dimensional image volumes

Abstract

To provide systems and methods for automatic image alignment of three-dimensional (3D) medical image volumes.SOLUTION: The methods include: pre-processing the 3D medical image volume by selecting a sub-volume of interest; detecting anatomical landmarks in the sub-volume using a deep neural network; estimating transformation parameters based on the anatomical landmarks to adjust rotation angles and translation of the sub-volume; adjusting the rotation angles and translation to produce a first aligned sub-volume; determining confidence in the transformation parameters based on the first aligned sub-volume; and iteratively refining the transformation parameters if the confidence is below a predetermined threshold. The disclosed methods reduce the need for manual alignment and increases a probability of the 3D image volume converging to a desired orientation compared to conventional approaches.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the subject matter disclosed herein relate to medical imaging, and more particularly, to multiplanar reformation of three-dimensional medical images. [Background technology]

[0002] Three-dimensional (3D) medical imaging is a type of medical imaging technique that acquires imaging data of a volume of an imaged object, as opposed to a single projection or two-dimensional (2D) slice of the imaged object. 3D medical images can be acquired using various imaging modalities (e.g., computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, etc.). The 3D medical image data can include a series of cross-sectional images that can be combined to create a 3D representation of a patient's anatomy for use in diagnosing or treating various medical conditions. The series of cross-sectional images can include a series of images parallel to a first plane, which can be one of a set of predefined planes (e.g., axial, coronal, or sagittal), or the first plane can be dynamically determined to capture the anatomical region of interest in a desired orientation (e.g., scan plane planning that can be performed with 2D localizer images). However, the series of cross-sectional images may not capture the anatomical region of interest in a desired orientation. For example, if a plane defining a desired view of the anatomical part of interest is not parallel to the first plane, the series of cross-sectional images may not include an image capturing the anatomical part of interest at the desired orientation. In such cases, realignment / reformatting of the 3D medical image data may be performed to "synthesize" a view of the anatomical part of interest at the desired orientation.

[0003] In some embodiments, the 3D medical image data can be realigned by performing a multiplanar reconstruction (MPR) of a series of cross-sectional images along a second plane, which can be selected by a technician or radiologist. Orienting the anatomical features captured by the 3D medical images to coincide with one or more standard orientations allows for consistent and accurate interpretation of the medical images and subsequent diagnosis. Image registration is typically performed by manually adjusting the position of the images (e.g., by selecting a second plane / plane of interest for reformatting the series of cross-sectional images) until the images are in a desired orientation. This process is often performed by a technician or radiologist, who must carefully adjust the images until they are properly aligned. However, manual image registration is a tedious and time-consuming task, often taking up to three minutes per image. Moreover, manual registration is prone to errors, especially when performed by a novice technician. Inaccurate image registration can lead to misinterpretation of the image data, which can result in inaccurate diagnosis and treatment. Therefore, a more efficient and accurate method of image registration is needed. Summary of the Invention

[0004] In one embodiment, a method for automatically registering a 3D medical image volume to display a predetermined view of interest in a predetermined orientation includes pre-processing the 3D medical image volume by selecting a sub-volume of interest, detecting a plurality of anatomical landmarks in the sub-volume of interest using a deep neural network trained to identify anatomical landmarks, estimating transformation parameters based on the plurality of anatomical landmarks to adjust a rotation angle and a translation of the sub-volume of interest, adjusting a rotation angle and a translation of the sub-volume of interest using the transformation parameters to generate a first registered sub-volume of interest, determining a confidence level of the transformation parameters based on the first registered sub-volume of interest, and iteratively improving the transformation parameters if the confidence level is below a predetermined threshold.

[0005] In another embodiment, an image processing system includes a display device, a non-transitory memory including instructions, and a processor that, when executed, causes the image processing system to receive a 3D image volume, pre-process the 3D image volume by selecting a sub-volume of interest, detect a plurality of landmarks in the sub-volume of interest using a deep neural network trained to identify anatomical landmarks, estimate transformation parameters based on the plurality of landmarks to adjust a rotation angle and a translation of the sub-volume of interest, use the transformation parameters to adjust a rotation angle and a translation of the sub-volume of interest to generate a first aligned sub-volume of interest, determine a confidence level of the transformation parameters based on the first aligned sub-volume of interest, and iteratively improve the transformation parameters if the confidence level is below a predetermined threshold.

[0006] In another embodiment, a method includes receiving a 3D image volume comprising a first plurality of 2D images parallel to a first plane; pre-processing the 3D image volume by selecting a subset of the first plurality of 2D images to generate a sub-volume of interest comprising an anatomical region of interest; detecting at least three landmarks in the sub-volume of interest using a deep neural network trained to identify anatomical landmarks; estimating transformation parameters comprising an anatomical plane of interest intersecting the sub-volume of interest based on the at least three landmarks; performing a multi-planar reconstruction of the sub-volume of interest along the anatomical plane of interest to generate a first registered sub-volume of interest, the first registered sub-volume of interest comprising a second plurality of 2D images parallel to the anatomical plane of interest; determining a confidence level of the transformation parameters based on the first registered sub-volume of interest; and iteratively improving the transformation parameters if the confidence level is below a predetermined threshold.

[0007] In this manner, 3D medical image volumes can be automatically registered to show anatomical regions of interest in a standard orientation without the need for time-consuming manual registration. Furthermore, the likelihood of successful registration is increased by iteratively refining the transformation parameters to improve the registration when confidence falls below a predefined threshold. The inventors herein have recognized that the closer the initial orientation of the 3D image volume is to the desired orientation, the greater the likelihood that the 3D image volume will approach the desired orientation.

[0008] It should be appreciated that the above summary has been provided to introduce in a simplified form some of the concepts further described in the detailed description. It is not intended to identify key features or essential features of the claimed subject matter, the scope of which is defined solely by the claims. Moreover, the claimed subject matter is not limited to implementations that solve the disadvantages noted above or in any part of this disclosure. [Brief description of the drawings]

[0009] [Figure 1] 1 is a flow chart illustrating a method 100 for automatic registration of 3D image volumes, according to an embodiment of the present disclosure. [Diagram 2] 2 is a flow chart illustrating a method 200 for pre-processing a 3D image volume to generate a sub-volume of interest, according to an embodiment of the present disclosure. [Diagram 3] 3 is a flowchart illustrating a method 300 for detecting landmarks in a sub-volume of interest, according to an embodiment of the present disclosure. [Figure 4] 4 is a flow chart illustrating a method 400 for estimating a rotation angle and a translation vector of an anatomical plane of interest, according to an embodiment of the present disclosure. [Diagram 5] 5 is a flow chart illustrating a method 500 for determining a confidence level of registration of sub-volumes of interest, according to an embodiment of the present disclosure. [Figure 6] 6 is a flow chart illustrating a method 600 for adjusting a rotation angle and a translation of a sub-volume of interest using transformation parameters, according to an embodiment of the present disclosure. [Figure 7] FIG. 1 is a block diagram of an MRI system according to one embodiment of the present disclosure. [Figure 8] FIG. 1 is a block diagram of an image processing system according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The following description relates to various methods and systems for automatically registering 3D image volumes to capture anatomical features of interest in standard planes and orientations, avoiding the need for time-consuming and potentially inconsistent manual image registration. In some embodiments, the automatic image registration can include four aspects: pre-processing, landmark detection, registration correction, and registration confidence estimation, where pre-processing verifies whether the acquired 3D image volume is suitable for registration, landmark detection identifies anatomical reference points for registration, registration correction aligns the 3D image volume to a desired plane at a desired orientation, and registration confidence estimation estimates the accuracy of the automatic registration process and triggers an iterative registration adjustment process if a confidence threshold is not met.

[0011] In one embodiment, the 3D image volume is pre-processed to determine whether a desired field of view (FOV) is included in the 3D image volume. The sub-volume of interest includes the desired field of view, and can be segmented from the 3D image volume by identifying a 2D image slice of the 3D image volume that includes an anatomical feature of interest. Additionally, the pre-processing can include a left-right separation step that separates left and right anatomical structures to allow separate processing.

[0012] In another aspect, the landmarks of the sub-volume can be detected using a deep neural network trained to segment multiple anatomical landmarks, which can be points, 3D cylindrical structures, or anatomical structures. The landmarks detected in the sub-volume are based on predefined anatomical landmarks associated with the desired view plane.

[0013] In another aspect, the detected landmarks can be filtered by confidence, and the confidence of the detected landmarks can be based on the features (e.g., shape and appearance) of the detected landmarks. In one embodiment, a feature distribution for each landmark type is created, and the confidence of the detected landmarks in the sub-volume is determined by calculating the probability that the landmark encoded by that feature belongs to the feature distribution of the respective landmark type. The filtered landmarks (e.g., the three or more landmarks with the highest confidence) can be used to determine an anatomical plane of interest relative to the image coordinate system, and further, a rotation angle and a translation vector of the anatomical plane of interest can be determined by comparing the positions of the filtered landmarks to one or more predetermined geometric constraints. The anatomical plane of interest, together with the rotation angle and the translation vector, may be collectively referred to herein as transformation parameters. The 3D image volume can then be reformatted using techniques such as multiplanar reconstruction (MPR) based on the transformation parameters to generate the registered sub-volume.

[0014] In another aspect, the confidence in the transformation parameters used to generate the registered sub-volumes can be determined based on the satisfaction of the geometric constraints of the landmarks of the registered sub-volumes and further based on the confidence in using the projections. In one embodiment, a maximum intensity projection (MIP) is generated from the registered sub-volumes and a machine learning / deep learning network is used to classify whether the registered sub-volumes are within a predetermined tolerance of rotation from the desired orientation. If the confidence falls below a threshold, the registration workflow can be repeated for a predetermined number of iterations. If the confidence falls below a threshold after the predetermined number of iterations, a failure of the registration can be declared.

[0015] In one embodiment, a 3D medical image (referred to herein as a 3D image, 3D medical image volume, and related terms) may be acquired by an imaging device (such as the MRI device 710 shown in FIG. 7). The 3D medical image volume acquired by the imaging device may be automatically registered using an image processing system 802 shown in FIG. 8 to generate one or more standard views of one or more anatomical regions of interest. The image processing system 802 may perform one or more operations of an automatic image registration method, such as method 100 shown in FIG. 1, including performing one or more pre-processing operations to obtain a sub-volume of interest (described in method 200 shown in FIG. 2), detecting a plurality of landmarks in the sub-volume of interest (described in method 300 shown in FIG. 3), determining transformation parameters for the sub-volume of interest based on the detected landmarks (described in method 400 shown in FIG. 4), applying the transformation parameters to the sub-volume of interest to generate a positioned sub-volume of interest (described in method 600 shown in FIG. 6), and determining a confidence level of the transformation parameters based on the positioned sub-volume of interest (e.g., the sub-volume of interest after applying the determined transformation parameters) (described in method 500 shown in FIG. 5).

[0016] 1, a method 100 for automatically registering a 3D image volume is shown. Method 100 can be used to register an anatomical region captured in a 3D image volume to a standard plane and orientation, allowing a radiologist or other medical imaging practitioner to quickly view the anatomical region of interest in a standard orientation and make a quick and accurate diagnosis based on the 3D image volume. In some embodiments, method 100 can be performed by an image processing system.

[0017] Method 100 begins at step 102. In step 102, an image processing system receives a 3D image volume. The 3D image volume may include a series of 2D images that represent a volume of an imaged object. The 3D image volume may be acquired using a variety of imaging modalities, such as computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, or other suitable imaging techniques. The 3D image volume may include a particular region of the imaged object, such as the heart, brain, or other anatomical structure, or may include multiple anatomical regions.

[0018] In step 104, the image processing system pre-processes the 3D image volume to generate a sub-volume of interest. In some embodiments, step 104 includes verifying whether the 3D volume is suitable for registration before proceeding with other pre-processing. In one embodiment, the image processing system can determine whether a desired field of view (FOV) is included in the 3D image volume. If the desired FOV is not included in the 3D image volume, the image processing system can discard the 3D image volume or request a new 3D image volume that includes the desired FOV. This ensures that the anatomical region of interest is included in the 3D image volume before proceeding with other pre-processing or image registration.

[0019] In one embodiment, step 104 includes selecting a subset of the plurality of 2D images that includes the sub-volume of interest. The subset may then be cropped to remove non-anatomical regions and focus on relevant anatomical features. In some embodiments, the 3D image volume may include symmetrical anatomical features. For example, the 3D image volume may include both the left and right hemispheres of the brain, or both the left and right lungs. In such cases, the image processing system may identify pairs of symmetrical anatomical features in the 3D image volume and select a sub-volume that includes one anatomical feature of the pair for further processing. This may reduce computational complexity and improve efficiency of the image registration process. In some embodiments, the sub-volume of interest selected in step 104 may be determined by identifying pairs of symmetrical anatomical features in the 3D image volume. A sub-volume that includes a first anatomical feature and does not include a second anatomical feature may be selected as the sub-volume of interest. The 3D image volume may also be pre-processed in step 104 to remove non-anatomical regions. For example, the image processing system may crop the 3D image volume such that regions that do not contain relevant anatomical information are removed. This pre-processing step allows the image registration process to be focused on the anatomical region of interest.

[0020] In step 106, the image processing system detects a plurality of landmarks in the subvolume of interest. The landmarks may correspond to specific anatomical structures or regions in the subvolume of interest. In some embodiments, the plurality of landmarks includes at least three different landmarks, providing sufficient points in the image coordinate system to allow an anatomical plane of interest to be determined relative to the image coordinate system. In some embodiments, the image processing system may employ a deep neural network trained to identify anatomical landmarks in the subvolume of interest. The deep neural network may be trained using various machine learning techniques and configured to identify anatomical landmarks (e.g., points, 3D cylindrical structures, or specific anatomical structures). The landmarks detected by the deep neural network may be based on predefined anatomical landmarks associated with the desired plane of view. In some embodiments, the image processing system may filter or sort the detected landmarks based on a confidence score. The confidence of the detected landmarks may be determined based on the features of the detected landmarks (e.g., the shape and appearance of the landmarks). In one embodiment, a feature distribution for each landmark type may be created, and the confidence of the detected landmarks in the subvolume may be determined by calculating the probability that the landmark encoded by the feature belongs to the feature distribution of the respective landmark type. In one embodiment, the image processing system can select a subset of landmarks from the plurality of landmarks detected within the subvolume of interest, which may include at least three landmarks, and can use these landmarks to determine an anatomical plane of interest relative to the image coordinate system.

[0021] In step 108, the image processing system estimates transformation parameters based on the plurality of landmarks to adjust the rotation angle and translation of the sub-volume of interest. The transformation parameters can be used to adjust the sub-volume of interest to a desired orientation or position. In some embodiments, the image processing system estimates transformation parameters based on the plurality of landmarks detected within the sub-volume of interest. These transformation parameters can include rotation angles and translation vectors used to adjust the orientation and position of the sub-volume of interest. In one embodiment, the image processing system can select a subset of landmarks from the plurality of landmarks detected within the sub-volume of interest. This subset of landmarks can include at least three landmarks, which can be used to determine an anatomical plane of interest relative to the image coordinate system. The selection of the subset of landmarks can be based on a confidence score of the landmarks, and the most confident landmarks can be selected for further processing. Once the anatomical plane of interest is determined, the image processing system can estimate the rotation angles and translation vectors required to align the sub-volume of interest with respect to the anatomical plane of interest. This includes comparing the positions of the selected landmarks to one or more pre-determined geometric constraints. The geometric constraints can be based on known or expected positions, or relative positions, of landmarks in a correctly aligned sub-volume of interest. By comparing the actual positions of the landmarks against these geometric constraints, the image processing system can estimate the rotation angles and translation vectors required to align the sub-volume of interest with respect to the anatomical plane of interest. The image processing system can use a variety of techniques to estimate the transformation parameters. For example, the image processing system can estimate the rotation angles and translation vectors using optimization algorithms, machine learning techniques, or other suitable methods.

[0022] In step 110, the image processing system adjusts the rotation angle and translation of the subvolume of interest using the transformation parameters such that a registered subvolume of interest is generated. The transformation parameters include a rotation angle and a translation vector and are applied to the subvolume of interest to generate a first registered subvolume. In some embodiments, adjusting the rotation angle and translation of the subvolume of interest includes performing a multiplanar reconstruction of the subvolume of interest along the anatomical plane of interest determined in step 108, and then rotating the plane of interest until the location of the landmarks satisfies one or more geometric constraints with respect to the image coordinate system. In one embodiment, the transformation parameters are used to rotate and translate the anatomical features captured in the subvolume of interest with respect to the image coordinate system such that a predefined anatomical plane is presented in a standard orientation with respect to the image coordinate system. This predefined anatomical plane can be determined based on a subset of landmarks detected in the subvolume of interest. In another embodiment, the transformation parameters are used to adjust the subvolume of interest such that the subvolume of interest is adjusted to a desired orientation. This desired orientation can be predefined based on the type of anatomical structure being imaged or the particular diagnostic or therapeutic method being performed.

[0023] In step 112, the image processing system determines the confidence of the transformation parameters based on the registered sub-volumes of interest. The confidence can indicate the accuracy or reliability of the registration of the registered sub-volumes. In one embodiment, the determination of the confidence of the transformation parameters includes detecting a plurality of landmarks in the registered sub-volumes using the deep neural network used in step 106. The positions of the detected landmarks in the registered sub-volumes are then compared to one or more pre-determined geometric constraints. The satisfaction of the geometric constraints of the landmarks is an indication of the accuracy of the registration process. For example, if the landmarks of the registered sub-volumes satisfy the geometric constraints within a small deviation, the registration process is considered successful and therefore the transformation parameters are reliable. In another embodiment, the determination of the confidence of the transformation parameters includes projecting the registered sub-volumes onto a projection plane to generate a projection image. The projection plane is parallel to the plurality of 2D images including the registered sub-volumes. The projection image is mapped to a confidence score using a confidence estimation network. The confidence score provides a quantitative measure of the accuracy of the registration process. A high confidence score suggests that the registration process was successful and that the transformation parameters are reliable.

[0024] In step 114, the image processing system checks whether the confidence of the transformation parameters exceeds a threshold. If the confidence does not exceed the threshold, the method 100 returns to step 106 and steps 106-114 are repeated as described above to iteratively improve the transformation parameters, but using the first registered sub-volume instead of the originally generated sub-volume. This iterative improvement process includes detecting landmarks in the first registered sub-volume, estimating improved transformation parameters based on these landmarks, adjusting the rotation angle and translation of the first registered sub-volume using the improved transformation parameters such that a second registered sub-volume is generated, and determining a confidence of the improved transformation parameters based on the second registered sub-volume. In some embodiments, the iterative improvement of the transformation parameters may be repeated until the confidence of the resulting registered volume either exceeds a confidence threshold and exceeds a predetermined number of iterations, and the image processing system may indicate that the automatic registration has failed and prompt the user to perform image registration manually. However, if, at step 114 , the image processing system determines that the confidence exceeds the confidence threshold, then method 100 proceeds to step 116 .

[0025] In step 116, the image processing system displays the registered sub-volumes on a display device. The registered sub-volumes may be visualized and presented to a user for further analysis or evaluation. In some embodiments, the image processing system may display the anatomical plane of interest to which the rotation and translation determined in step 108 have been applied. In some embodiments, the image processing system may display the confidence score determined in step 112 alongside the registered sub-volumes. This may provide the user with an indication of the reliability of the registration process. The image processing system may also provide the user with the option to manually adjust the registration if the confidence score falls below a certain level. This may ensure that the user has the final say in the registration process. After step 116, the method 100 may end.

[0026] 2, a method 200 for pre-processing a 3D image volume to generate sub-volumes of interest is shown. Method 200 can be used to select a subset of 2D images from the 3D image volume that contains a desired field of view, and to subdivide the subset of images along a plane of symmetry.

[0027] In step 202, the image processing system checks whether the 3D image volume includes a desired field of view (FOV). The image processing system can determine whether the desired FOV is included in the 3D image volume by analyzing the spatial extent of the 3D image volume and comparing the spatial extent to a predetermined spatial extent of the desired FOV. The desired FOV can be predetermined based on the type of anatomical structure being imaged or the particular diagnostic or therapeutic procedure being performed. For example, if the 3D image volume is intended for cardiac imaging, the desired FOV includes the entire heart and the surrounding vasculature. If the 3D image volume is intended for brain imaging, the desired FOV includes the entire brain and surrounding structures. In some embodiments, the image processing system can employ a machine learning model that automatically determines whether an anatomical region of interest is included in the acquired 3D image volume. In some embodiments, the machine learning model can be trained to classify the 3D image volume according to anatomical features captured in the 3D image volume and / or output a probability score indicative of the probability that the 3D image volume includes an anatomical feature or region of interest. If the 3D image volume does not include the desired FOV, the method 200 can proceed to step 204. In some embodiments, the image processing system may also check the image quality (such as image resolution, signal to noise ratio, or other image quality metrics) of the 3D image volume in step 202 to ensure that the 3D image volume is suitable for subsequent image registration processing. If the image quality is below a predefined threshold, the image processing system may proceed to step 204.

[0028] If the 3D image volume does not include the desired field of view, or if the quality of the 3D image volume is below a predetermined threshold, method 200 proceeds to step 204, where the image processing system displays a request to rescan the object via a display device. In some embodiments, the image processing system provides feedback to a user or operator at step 204, indicating that the 3D image volume does not include the desired FOV and / or suggesting appropriate corrective action (such as adjusting imaging parameters or repositioning the object). The request prompts the user to rescan the object to obtain a 3D image volume that includes the desired field of view. After step 204, method 200 may end.

[0029] However, if in step 202 the image processing system determines that the 3D image volume includes the desired field of view, the method 200 proceeds to step 206. In step 206, the image processing system selects a subset of the multiple 2D images that make up the 3D image volume that includes the subvolume of interest. The subset of images is selected based on the desired field of view. This selection process includes identifying 2D images that include anatomical features of interest. As an example, if the desired FOV includes the heart, the image processing system may select 2D images that include the heart and the surrounding vasculature. In some embodiments, the image processing system may employ a machine learning model trained to identify anatomical features of interest in the 2D images. The machine learning model may be trained using various machine learning techniques and may be configured to identify anatomical features (e.g., points, 3D cylindrical structures, or specific anatomical structures). In some embodiments, the image processing system may also consider the quality of the 2D images when selecting the subset of images. For example, the image processing system may select 2D images that have a high signal-to-noise ratio, high resolution, or other desirable image quality indicators. This may ensure that the selected subset of images is suitable for subsequent image registration processing.

[0030] In step 208, the image processing system can optionally partition the symmetrical anatomical features by sub-dividing the plurality of 2D images along a plane of symmetry. This partitioning can separate the symmetrical anatomical features into separate sub-volumes. As an example, if the 3D image volume includes both the left and right knees to be imaged, the image processing system can separate the left and right knees for separate processing. This can include identifying a plane of symmetry in the 3D image volume and partitioning the 3D image volume along the plane of symmetry. This can result in two sub-volumes, each of which includes one of the symmetrical anatomical features. Each of these sub-volumes is processed separately in the subsequent image registration process. This can improve the efficiency of the image registration process and can also improve the accuracy of the registration by reducing the complexity of the anatomical features to be registered. In some embodiments, only the first of the two symmetrical anatomical features is selected for subsequent image registration.

[0031] In step 210, the image processing system crops a subset of the plurality of 2D images such that non-anatomical regions are removed. In some embodiments, the image processing system may identify boundaries of anatomical features of interest in each 2D image and remove portions of the 2D images that are outside of these boundaries. This may be accomplished by applying a segmentation algorithm to each 2D image to segment the anatomical features of interest from the background or non-anatomical regions. The segmentation algorithm may be a traditional image processing algorithm (such as thresholding, edge detection, region growing, or watershed segmentation) or may be a machine learning based algorithm (such as a convolutional neural network trained to segment anatomical features of interest). In some embodiments, the image processing system may employ a deep learning model trained to identify and segment anatomical features of interest in the 2D images. In some embodiments, the deep neural network may output a binary mask for each 2D image, in which pixels corresponding to the anatomical features of interest are assigned a value of 1 and all other pixels are assigned a value of 0. The image processing system may then apply these binary masks to the corresponding 2D images to remove non-anatomical regions. After step 210, the method 200 may end.

[0032] In this manner, the method 200 allows for efficient pre-processing of the 3D image volume to generate the sub-volume of interest. This pre-processing checks whether the 3D image volume contains the desired field of view, thereby increasing the success rate of the resulting image registration. Furthermore, by selecting a subset of the 2D image that contains the desired field of view, the image processing system can focus on relevant anatomical features and ignore irrelevant regions, thereby reducing computational complexity and improving the efficiency of the image registration process.

[0033] 3, a method 300 for detecting landmarks in a sub-volume of interest is shown, which is implemented by an image processing system and includes mapping the sub-volume of interest to a plurality of landmarks using a trained deep neural network, deriving one or more landmarks based on the positions of the plurality of landmarks, determining a confidence level for each detected landmark, and filtering the plurality of landmarks based on the confidence levels to obtain at least three landmarks in the sub-volume.

[0034] Method 300 begins at step 302, where an image processing system maps a subvolume of interest to a plurality of landmarks using a trained deep neural network. The deep neural network can be trained using various machine learning techniques and can be configured to identify anatomical landmarks (e.g., points, 3D cylindrical structures, or specific anatomical structures). The landmarks detected by the deep neural network can be based on predefined anatomical landmarks associated with a desired viewing plane. The deep neural network can be trained on a large dataset of 3D image volumes, each of which is annotated with the locations of important anatomical landmarks. The training process can include tuning parameters of the deep neural network to minimize the difference between the predicted locations of the landmarks and the actual locations of the landmarks in the training dataset. The trained deep neural network can then be used to predict the locations of landmarks in new, yet to be observed, 3D image volumes. In some embodiments, the deep neural network can include multiple layers, each layer transforming input data to a higher level of abstraction. For example, a first layer may detect edges in a 3D image volume, a second layer may detect primitive shapes, and subsequent layers may detect more complex anatomical structures. A final layer of the deep neural network may output predicted locations of landmarks in the 3D image volume. In some embodiments, the deep neural network may be a convolutional neural network (CNN). The CNN may include a convolutional layer that applies a set of filters to input data, a pooling layer that reduces the dimensionality of the data, and a fully connected layer that outputs predicted locations of landmarks.

[0035] In some embodiments, the deep neural network may be a recurrent neural network (RNN) configured to process sequential data. RNNs may be advantageously used when a 3D image volume is displayed as a series of 2D image slices. The RNN may process each 2D image slice sequentially, maintaining an internal state that captures information from the previous slice, and output a predicted location of landmarks in the 3D image volume. In some embodiments, the deep neural network may be a combination of a CNN and an RNN. The CNN portion of the network may process each 2D image slice independently so that local features are detected, and the RNN portion of the network may process a series of 2D image slices to capture spatial relationships between features across different slices. In some embodiments, the deep neural network may be trained using transfer learning, where a pre-trained network is fine-tuned on the specific task of landmark detection. The pre-trained network may be trained on a large generic dataset and has learned to extract features from image data. By fine-tuning a pre-trained network on the specific task of landmark detection, an image processing system can leverage the feature extraction capabilities of the pre-trained network while adapting the feature extraction capabilities to the specific task of anatomical landmark detection. In some embodiments, a deep neural network can be trained using unsupervised learning, where the network learns to extract useful features from a 3D image volume without the need for annotated training data. Unsupervised learning can include training the network to reconstruct input data, generate new data similar to the input data, or cluster the input data into groups. The features learned by the network during unsupervised learning can then be used to detect landmarks in the 3D image volume.

[0036] In step 304, the image processing system may optionally derive one or more landmarks based on the locations of the multiple landmarks. The derivation process may include utilizing the location of one or more landmarks detected in step 302 and combining the landmarks with known anatomical relationships to obtain location information for one or more anatomical landmarks that were not directly detected. For example, in one embodiment, the image processing system may derive a landmark corresponding to the center of an anatomical structure by averaging the locations of two or more landmarks detected in step 302 that are known to be located at the boundaries of the anatomical structure. In another embodiment, the image processing system may derive a landmark corresponding to an anatomical feature that is known to be located at a fixed distance and direction from the landmark detected in step 302. For example, if a landmark corresponding to the apex is detected, a landmark corresponding to the base of the heart may be derived by moving a fixed distance from the apex. In yet another embodiment, the image processing system may derive a landmark corresponding to the intersection of two or more anatomical structures by identifying landmarks located at each anatomical structure detected in step 302 and calculating their intersection. For example, if a landmark corresponding to the borders of the left and right lungs is detected, a landmark corresponding to the mediastinum, the space between the lungs, can be derived by calculating the intersection of the lung borders. In some embodiments, the image processing system can derive additional landmarks based on the locations of multiple landmarks detected in step 302 and known anatomical relationships between these landmarks. For example, the image processing system can derive a landmark corresponding to the centroid of an anatomical structure by calculating a weighted average of the locations of the landmarks detected in step 302 that are located within the anatomical structure, where the weights are proportional to the voxel intensity at the landmark location. In some embodiments, the image processing system can derive landmarks corresponding to anatomical features that are not directly visible in the 3D image volume.For example, the image processing system can derive landmarks corresponding to the center of a blood vessel by fitting a 3D cylindrical model to the locations of landmarks detected in step 302 that are located on the border of the blood vessel. In some embodiments, the image processing system can derive landmarks corresponding to anatomical features that are only partially visible in the 3D image volume. For example, the image processing system can derive landmarks corresponding to the tips of bones by extrapolating from the locations of landmarks detected in step 302 that are located on visible parts of the bone. These derived landmarks can provide additional reference points for registering the 3D image volumes, improving the accuracy and robustness of the registration process.

[0037] In step 306, the image processing system determines a confidence level for each detected landmark. By determining this confidence level, the system can assess the reliability and accuracy of each landmark. In one embodiment, the confidence level determination in step 306 can include calculating a probability that each landmark encoded by a feature belongs to a feature distribution for the respective landmark type. The feature distribution for each landmark type can be created based on a training data set of known anatomical landmarks. The features of the detected landmarks can include, but are not limited to, shape, size, texture, and appearance features. For example, a point landmark can have a feature distribution based on the spatial coordinates of the point, and a landmark that is an anatomical structure can have a feature distribution based on the shape and size features of the anatomical structure. In another embodiment, the confidence level determination in step 306 can include comparing the detected landmark to a set of predefined anatomical landmarks associated with the desired field of view. The comparison can include calculating a similarity score between the detected landmark and the predefined anatomical landmarks. The similarity score can be based on a variety of factors, such as the spatial proximity of the detected landmarks to predefined anatomical landmarks, the similarity in shape and appearance between the detected landmarks and the predefined anatomical landmarks, and the consistency of the detected landmarks with known anatomical relationships.

[0038] In step 308, the image processing system filters the plurality of landmarks based on the confidence to obtain at least three landmarks in the subvolume. Filtering the landmarks based on the probability that the landmarks belong to the feature distribution of each of its landmark types / classes reduces the probability of using false landmarks in subsequent image registration. In one embodiment, the filtering process in step 308 can include sorting the detected landmarks based on the confidence scores of the detected landmarks and selecting the top three landmarks with the highest confidence scores. In another embodiment, the filtering process in step 308 can include setting a confidence threshold and selecting the landmarks whose confidence scores exceed this threshold. The confidence threshold can be determined based on the requirements of a particular application, the quality of the 3D image volume, or the performance of the deep neural network. In yet another embodiment, the filtering process in step 308 can include applying a machine learning model to predict the confidence of each landmark based on the features and confidence score of each landmark. The machine learning model can be trained on a dataset of known landmarks, and each landmark is labeled with the confidence of each landmark. The features used by the machine learning model include the confidence scores of the landmarks, the features of the landmarks encoded by the deep neural network, and other relevant features (such as the quality of the 3D image volume, or the performance of the deep neural network). The machine learning model can output a confidence score for each landmark, and the landmark with the highest confidence score can be selected for further processing. In some embodiments, the filtering process in step 308 can also include clustering the detected landmarks based on their spatial proximity to each other and selecting one landmark from each cluster based on the landmark's confidence score. This can ensure that the selected landmarks are distributed across different anatomical regions, which can improve the robustness and accuracy of the subsequent image registration process.

[0039] In some embodiments, the filtering process in step 308 may include a combination of the above embodiments. For example, the image processing system may first sort the detected landmarks based on their confidence scores, then apply a confidence threshold to select a subset of the sorted landmarks, and finally apply a machine learning model to predict the confidence of the subset of landmarks and select the most confident landmarks for further processing. After step 308, method 300 ends.

[0040] In this manner, the method 300 allows the image processing system to accurately and efficiently detect and select anatomical landmarks in the subvolume of interest. This is achieved by using a trained deep neural network that can identify various anatomical landmarks based on predefined anatomical landmarks associated with the desired view plane. The method further improves the accuracy of the landmark detection by deriving additional landmarks based on the location and known anatomical relationships of the detected landmarks. This provides a comprehensive set of reference points for the subsequent image registration process. Furthermore, the method 300 incorporates a confidence determination step that evaluates the confidence of each detected landmark based on the landmark's features and the probability that the landmark belongs to the feature distribution of the respective landmark type. This can ensure that only potentially false landmarks are not selected for further processing, thereby reducing the risk of erroneous landmarks affecting the image registration process.

[0041] 4, a method 400 for estimating a rotation angle and a translation vector of an anatomical plane of interest is shown. The method 400 can be implemented by an image processing system to determine an anatomical plane of interest relative to an image coordinate system based on a number of landmarks (the anatomical plane of interest intersects with the sub-volume) and to determine a rotation angle and a translation vector of the anatomical plane of interest.

[0042] In step 402, the image processing system determines an anatomical plane of interest relative to the image coordinate system based on the plurality of landmarks. This determination allows the system to identify a particular plane of interest within the sub-volume for registration. In one embodiment, determining the anatomical plane of interest includes utilizing a subset of landmarks from the plurality of landmarks determined in step 308 of method 300. The subset of landmarks includes at least three landmarks, which may provide sufficient points in the image coordinate system to determine the anatomical plane of interest relative to the image coordinate system. The selection of the subset of landmarks may be based on a confidence score of the landmarks, with the most confident landmark being selected for further processing. Once the subset of landmarks is selected, the image processing system may determine the anatomical plane of interest by fitting a plane to the positions of the selected landmarks in the image coordinate system. This may be done using a least squares fitting method or other suitable plane fitting technique. Thus, the anatomical plane of interest is determined to best fit the positions of the selected landmarks in the image coordinate system.

[0043] In some embodiments, the three or more landmarks do not define an anatomical plane of interest, but rather a plane that has a fixed relationship to the anatomical plane of interest, and in such embodiments, after fitting a plane to the three or more landmarks, the anatomical plane of interest is derived from the fitted plane with a fixed relationship between the fitted plane and the anatomical plane of interest. As an example, the three or more landmarks can define a reference plane that is parallel to the anatomical plane of interest but does not coincide with the anatomical plane of interest. In such a case, the anatomical plane of interest can be derived from the fitted plane by applying a translation along a normal to the fitted plane. The magnitude and direction of the translation can be determined based on a known or expected distance between the landmarks and the anatomical plane of interest. For example, if the landmarks are located on the surface of an organ and the anatomical plane of interest is located at the center of the organ, the translation can be equal to half the thickness of the organ and go towards the interior of the organ. In some embodiments, the three or more landmarks can define a reference plane that is orthogonal to the anatomical plane of interest. In such cases, the anatomical plane of interest can be derived from the fitted plane by rotating the fitted plane about an axis parallel to the intersection of the fitted plane with the anatomical plane of interest. The rotation angle can be determined based on a known or expected angle between the landmark and the anatomical plane of interest. For example, if the landmark is located on the surface of an organ and the anatomical plane of interest is a cross-section of the organ, the rotation angle can be 90 degrees.

[0044] In yet other embodiments, three or more landmarks can define a reference plane that is oblique to the anatomical plane of interest. In such cases, the anatomical plane of interest can be derived from the fitted plane by applying a combination of rotation and translation. Rotation can align the normal of the fitted plane with the normal of the anatomical plane of interest, and translation can move the fitted plane to coincide with the anatomical plane of interest. The axis of rotation, the angle of rotation, the direction of translation, and the magnitude of translation can be determined based on a known or expected relationship between the landmarks and the anatomical plane of interest.

[0045] In step 404, the image processing system determines a rotation angle and a translation vector of the anatomical plane of interest. This determination allows the image processing system to adjust the orientation and position of the sub-volume so that the sub-volume is aligned with the desired anatomical plane. In one embodiment, the determination of the rotation angle and the translation vector includes comparing the positions of the selected landmarks to one or more pre-determined geometric constraints. These geometric constraints may be based on known or expected positions of the landmarks of the correctly aligned sub-volume of interest with respect to the image coordinate system, or the relative positions of the landmarks. By comparing the actual positions of the landmarks to these geometric constraints, the image processing system can estimate the rotation angle and the translation vector required to align the sub-volume of interest with the anatomical plane of interest. In some embodiments, the geometric constraints may include the distance of the pair of landmarks with respect to one or more axes of the image coordinate system. As an example, a geometric constraint that may be used to determine the rotation angle may specify that in a correctly aligned sub-volume, the difference between a first y-coordinate of a first landmark and a second y-coordinate of a second landmark is zero or within a threshold of zero. In another example, a geometric constraint that can be used to determine the translation vector can specify that the location of the landmark is within a threshold distance of the midpoint of the range of an axis of the image coordinate system. In another embodiment, the geometric constraint can include an angle between a line connecting a pair of landmarks with respect to one or more axes of the image coordinate system. For example, the geometric constraint can specify that in the correctly registered sub-volume, the angle between a line connecting a first landmark and a second landmark and a point on the x-axis of the image coordinate system that is equal to the x-coordinate of the second landmark is 90 degrees or within a threshold of 90 degrees. In yet another embodiment, the geometric constraint can include the relative positions of multiple landmarks. For example, the geometric constraint can specify that in the correctly registered sub-volume, a first landmark is located above a second landmark or a third landmark is located to the right of a fourth landmark.This is advantageous when registering anatomical structures that have known spatial relationships to each other in properly registered sub-volumes.

[0046] In some embodiments, the image processing system can directly calculate the rotation angle and translation vector using an analytical solution to a system of equations that includes the current positions of the landmarks or the projection of the landmarks onto one or more axes of the image coordinate system and the geometric constraints. In some embodiments, the image processing system can estimate the rotation angle and translation vector using an optimization algorithm. The optimization algorithm can minimize a cost function that measures the deviation between the actual positions of the landmarks and the expected positions of the landmarks, subject to the geometric constraints. The cost function can include a term for each geometric constraint and can be weighted to favor certain constraints over other constraints. The optimization algorithm can include gradient descent, Newton's method, or other suitable optimization techniques. After step 404, the method 400 can end.

[0047] 5, a method 500 for determining a confidence level of an alignment of sub-volumes of interest is shown. The method 500 may be implemented by an image processing system and may include detecting a plurality of landmarks in the aligned sub-volumes of interest using a deep neural network, comparing the positions of the plurality of landmarks in the aligned sub-volumes of interest with one or more pre-determined geometric constraints to determine a satisfaction level of the geometric constraints, determining a geometric constraint confidence level based on the satisfaction level of the geometric constraints, which may be employed by the image processing system, projecting the aligned sub-volumes of interest onto a projection plane to generate a projection image, mapping the projection image to a projection confidence level using a confidence estimation network, and combining the geometric constraint confidence level and the projection confidence level to determine an overall confidence level of the aligned sub-volumes.

[0048] In step 502, the image processing system detects a plurality of landmarks in the registered sub-volumes of interest using a deep neural network. This detection enables the system to identify the locations of the landmarks in the registered sub-volumes of interest. In one embodiment, the deep neural network used to detect the landmarks in the registered sub-volumes of interest can be the same or similar to the deep neural network used in the initial detection of the landmarks in the sub-volumes of interest (such as the network used in step 302). The deep neural network can be trained using various machine learning techniques and can be configured to identify anatomical landmarks (e.g., points, 3D cylindrical structures, or specific anatomical structures). The landmarks detected by the deep neural network can be based on predefined anatomical landmarks associated with the desired plane of view.

[0049] Following landmark detection, in step 504, the image processing system compares the positions of the landmarks of the registered sub-volume of interest to one or more pre-determined geometric constraints to determine the satisfaction of the geometric constraints. This comparison allows the system to evaluate the extent to which the registered sub-volume meets the desired geometric constraints. In one embodiment, the geometric constraints may be based on known or expected positions, or relative positions, of the landmarks of the properly registered sub-volume of interest. For example, if the anatomical region of interest is the heart, the geometric constraints may specify that certain landmarks (e.g., the apex and base) should be located in a particular position relative to each other and to the image coordinate system in the properly registered sub-volume. By comparing the actual positions of the landmarks of the registered sub-volume to these geometric constraints, the image processing system can determine the extent to which the geometric constraints are met.

[0050] In some embodiments, comparing the landmark positions to the geometric constraints includes calculating a distance between each landmark and the corresponding positions of the landmarks specified by the geometric constraints. These distances may then be combined (e.g., added or averaged) to generate a measure of satisfaction of the geometric constraints. In other embodiments, the comparison may include calculating a measure of deviation of the landmark positions from the geometric constraints. This measure of deviation may be based on a statistical measure (such as standard deviation or variance) or a geometric measure (such as the volume of the smallest polyhedron that encompasses all the landmarks and the corresponding positions of the landmarks specified by the geometric constraints).

[0051] In step 506, the image processing system determines a geometric constraint confidence based on the satisfaction of the geometric constraint. This determination allows the system to quantify a level of confidence in the registration based on the satisfaction of the geometric constraint. In one embodiment, the geometric constraint confidence can be a function of the satisfaction of the geometric constraint, with higher satisfaction corresponding to higher confidence. The function can be a linear function, a non-linear function, or a step function, among others. In some embodiments, the geometric constraint confidence can be normalized to a range between a confidence minimum and a confidence maximum. For example, the geometric constraint confidence can be normalized to a range between 0 and 1, where 0 represents no confidence and 1 represents maximum confidence. The normalization can be performed using various techniques, such as min-max normalization, z-score normalization, or decimal scaling normalization, among others. In some embodiments, the geometric constraint confidence can be adjusted based on additional factors, such as the quality of the 3D image volume, the complexity of the anatomical region of interest, or the performance of the deep neural network in detecting landmarks. For example, if the 3D image volume is of poor quality (e.g., due to motion artifacts or low signal-to-noise ratio), the geometric constraint confidence may decrease, reflecting increased uncertainty in the registration. Similarly, if the anatomical region of interest is complex (e.g., due to the presence of multiple overlapping structures or due to anatomical structure variability between different individuals), the geometric constraint confidence may decrease, reflecting increased difficulty in registration.

[0052] Following the determination of the geometric constraint confidence, in step 508, the image processing system projects the aligned sub-volumes of interest onto a projection plane to generate a projection image. This projection process includes converting the aligned 3D sub-volumes into a 2D projection image. The projection plane is parallel to the multiple 2D images that include the aligned sub-volumes of interest. This projection process can be achieved using various projection techniques known in the field of medical imaging, such as maximum intensity projection (MIP), minimum intensity projection (MinIP), or average intensity projection (AIP). The projection image provides a comprehensive view of the aligned sub-volumes and can be used to further evaluate the quality of the alignment at reduced computational cost compared to approaches that utilize unprojected data of the aligned sub-volumes.

[0053] In step 510, the image processing system maps the projection images to projection confidences using a confidence estimation network. This mapping allows the system to determine the confidence of the alignment based on the quality of the projection images. The confidence estimation network can be a machine learning or deep learning network trained to estimate the confidence of the alignment based on the projection images. The network can be trained using various machine learning techniques and can be configured to map the projection images to confidence scores. The confidence score provides a quantitative measure of the accuracy of the alignment process. A high confidence score suggests that the alignment process is successful and the transformation parameters are reliable.

[0054] In step 512, the image processing system determines an overall confidence of the registered sub-volume by combining the geometric constraint confidence and the projection confidence. This overall confidence provides an overall measure of the accuracy of the registration process, taking into account both the satisfaction of the geometric constraint and the projection confidence. The overall confidence can be determined using various combination methods, such as weighted average, maximum, minimum, or other suitable combination methods. The overall confidence can be used to determine whether the registration process is successful or whether further improvement of the transformation parameters is required. If the overall confidence is above a predetermined threshold, the registration process can be considered successful. If the overall confidence is not above a predetermined threshold, the registration process can be iteratively improved until the overall confidence exceeds a threshold or a predetermined number of iterations is reached. After step 512, the method 500 can end.

[0055] In this manner, method 500 allows an image processing system to determine the confidence of the registration of the subvolumes of interest in a comprehensive and robust manner. By combining the confidence of the geometric constraints with the confidence of the projection, the system can take into account both the satisfaction of the geometric constraints and the quality of the projection images when assessing the accuracy of the registration process. This dual approach to confidence estimation can provide a more reliable measure of registration accuracy than either method alone, which may lead to improved diagnostic accuracy and efficiency in medical imaging applications. In this manner, method 500 can perform automatic image registration in medical imaging applications, improving diagnostic accuracy and efficiency, reducing the need for manual image registration and improving patient care.

[0056] 6, there is shown a method 600 for registering a sub-volume of interest to an anatomical plane of interest. The method 600 may be performed by an image processing system to adjust the orientation and position of the sub-volume so that it is registered to a desired anatomical plane.

[0057] In step 602, the image processing system reformats the sub-volume of interest along an anatomical plane of interest to generate a reformatted sub-volume of interest. This step includes re-slicing the sub-volume of interest into a series of 2D image slices parallel to the anatomical plane of interest. In one embodiment, the reformatting process in step 602 can include using a multiplanar reconstruction (MPR) technique. MPR is a post-processing technique that allows visualization of images in various planes. In this context, MPR can be used to re-slice the sub-volume of interest into a series of 2D image slices parallel to the anatomical plane of interest. This is particularly useful when the original imaging plane does not coincide with the anatomical plane of interest. As an example, the sub-volume of interest is a region of the brain captured in the sagittal plane, but the anatomical plane of interest is in the coronal plane. With MPR, the image processing system can reformatted the sagittal image into a series of 2D image slices parallel to the coronal plane. This allows for more accurate and detailed analysis of the region of interest.

[0058] In step 604, the image processing system applies a rotation angle to the reformatted subvolume of interest. This step includes rotating the reformatted subvolume about an appropriate axis such that the desired rotation angle is achieved. In some embodiments, the image processing system may apply a rotation matrix to the subvolume of interest in step 604. The rotation matrix may be applied to each voxel in the subvolume of interest to rotate the subvolume to rotate anatomical landmarks in the subvolume of interest to positions estimated to satisfy one or more geometric constraints. The rotation may include adjusting the orientation of the subvolume with respect to one or more rotation axes.

[0059] In step 606, the image processing system applies a translation vector to the reformatted subvolume of interest to generate an aligned subvolume of interest. This step includes shifting the reformatted subvolume along the appropriate axis to achieve the desired translation. The translation vector may include a vector in three-dimensional space and therefore may include a three-value vector indicating the magnitude of translation to be applied to each voxel in the subvolume of interest along each of the three basis vectors / axes of the image coordinate system. After step 606, the method 600 may end.

[0060] Method 600 enables the image processing system to register the subvolume of interest to an anatomical plane of interest, and to accurately analyze and visualize anatomical structures within the subvolume. Although method 600 is intended to provide one exemplary approach to applying transformation parameters to register the subvolume of interest, it is understood that alternative methods for defining the registration transformation and applying said transformation are encompassed by this disclosure. As an example, in certain embodiments, the image processing system can apply an affine transformation to the subvolume of interest to generate a registered subvolume of interest. The affine transformation can include a combination of rotation, translation, scaling, and shearing operations. The parameters of the affine transformation can also be the transformation parameters estimated in step 108 (FIG. 1). The affine transformation can be applied to each voxel within the subvolume of interest to adjust the orientation, position, size, and shape of the subvolume to register the subvolume to the anatomical plane of interest. In some embodiments, the image processing system can apply a non-linear transformation to the subvolume of interest. The nonlinear transformation can be used to correct distortions in the subvolume of interest that are difficult to correct using linear transformations (such as rotation, translation, scaling, and shear). Parameters of the nonlinear transformation can be determined based on the transformation parameters estimated in step 108 (FIG. 1) and additional parameters estimated based on characteristics of the subvolume of interest. The nonlinear transformation can be applied to each voxel of the subvolume of interest to adjust the orientation, position, size, and shape of the subvolume and to align the subvolume with respect to the anatomical plane of interest.

[0061] Referring to FIG. 7, a magnetic resonance imaging (MRI) device 710 is shown, which includes a static magnetic field magnet unit 712, a gradient coil unit 713, an RF coil unit 714, an RF body coil unit or volume coil unit 715, a transmit / receive (T / R) switch 720, an RF driver unit 722, a gradient coil driver unit 723, a data acquisition unit 724, a controller unit 725, a patient table or bed 726, a data processing unit 731, an operation console unit 732, and a display unit 733. In some embodiments, the RF coil unit 714 is a surface coil, which is typically a local coil placed in close proximity to an anatomical structure of interest of a subject 716. Here, the RF body coil unit 715 is a transmit coil that transmits RF signals, and the local surface RF coil unit 714 receives MR signals. Thus, the transmit body coil (e.g., RF body coil unit 715) and the surface receive coil (e.g., RF coil unit 714) are separate but electromagnetically coupled components. The MRI apparatus 710 performs a scan by transmitting electromagnetic pulse signals to a subject 716 placed in an imaging space 718 in which a static magnetic field is formed, and obtains magnetic resonance signals from the subject 716. Based on the magnetic resonance signals thus obtained by the scan, one or more images of the subject 716 can be reconstructed.

[0062] The static magnetic field magnet unit 712 includes, for example, a ring-shaped superconducting magnet mounted in a toroidal vacuum vessel, which defines a cylindrical space surrounding the subject 716 and generates a constant main static magnetic field B0.

[0063] The MRI apparatus 710 also includes a gradient coil unit 713, which forms a gradient magnetic field in the imaging space 718 so as to impart three-dimensional position information to the magnetic resonance signal received by the RF coil array. The gradient coil unit 713 includes three gradient coil systems, each of which generates a gradient magnetic field along one of three mutually orthogonal spatial axes, and generates a gradient magnetic field in each of the frequency encoding direction, the phase encoding direction, and the slice selection direction according to the imaging conditions. More specifically, the gradient coil unit 713 applies a gradient magnetic field in a slice selection direction (or a scan direction) of the subject 716 to select a slice, and the RF body coil unit 715 or the local RF coil array can transmit an RF pulse to the selected slice of the subject 716. The gradient coil unit 713 also applies a gradient magnetic field in a phase encoding direction of the subject 716 to phase encode the magnetic resonance signal from the slice excited by the RF pulse. Next, the gradient coil unit 713 applies a gradient magnetic field in a frequency encoding direction to the subject 716 to frequency encode the magnetic resonance signal from the slice excited by the RF pulse.

[0064] The RF coil unit 714 is arranged, for example, to surround the region of the subject 716 to be imaged. In some examples, the RF coil unit 714 may be called a surface coil or a receiving coil. In a static magnetic field space or imaging space 718 in which a static magnetic field B0 is formed by the static magnetic field magnet unit 712, the RF coil unit 715 transmits RF pulses, which are electromagnetic waves, to the subject 716 based on a control signal from the controller unit 725, thereby generating a high-frequency magnetic field B1. This excites the spins of protons in a slice to be imaged of the subject 716. The RF coil unit 714 receives, as a magnetic resonance signal, the electromagnetic waves generated when the spins of protons excited in this way in the slice to be imaged of the subject 716 return to align with the initial magnetization vector. In some embodiments, the RF coil unit 714 can transmit RF pulses and receive MR signals. In other embodiments, the RF coil unit 714 may be used only for receiving MR signals and not for transmitting RF pulses.

[0065] The RF body coil unit 715 is arranged, for example, to surround the imaging space 718 and generates RF magnetic field pulses orthogonal to the main magnetic field B0 generated by the static magnetic field magnet unit 712 in the imaging space 718 to excite nuclei. In contrast to the RF coil unit 714, which can be separated from the MRI device 710 and replaced with another RF coil unit, the RF body coil unit 715 is fixedly attached and connected to the MRI device 710. Furthermore, while a local coil (such as the RF coil unit 714) can transmit signals to or receive signals from only a localized area of ​​the subject 716, the RF body coil unit 715 generally has a wide coverage area. The RF body coil unit 715 can be used, for example, to transmit signals to or receive signals from the entire body of the subject 716. The use of a receive-only local coil and a transmit body coil provides uniform RF excitation and good image uniformity at the expense of high RF power applied to the subject. In the case of a transmit / receive local coil, the local coil RF excites the region of interest and receives the MR signal, thereby reducing the RF power delivered to the subject. It should be understood that the RF coil unit 714 and / or the RF body coil unit 715 have specific uses depending on the imaging application.

[0066] The T / R switch 720 can selectively electrically connect the RF body coil unit 715 to the data acquisition unit 724 when operating in a receive mode and selectively electrically connect the RF body coil unit 715 to the RF driver unit 722 when operating in a transmit mode. Similarly, the T / R switch 720 can selectively electrically connect the RF coil unit 714 to the data acquisition unit 724 when operating in a receive mode and selectively electrically connect the RF coil unit 714 to the RF driver unit 722 when operating in a transmit mode. When both the RF coil unit 714 and the RF body coil unit 715 are used in a single scan, for example when the RF coil unit 714 is configured to receive MR signals and the RF body coil unit 715 is configured to transmit RF signals, the T / R switch 720 can route the received MR signals from the RF coil unit 714 to the data acquisition unit 724 while routing the control signals from the RF driver unit 722 to the RF body coil unit 715. The coils of the RF body coil unit 715 may be configured to operate in a transmit-only mode or in a transmit-receive mode, and the coils of the local RF coil unit 714 may be configured to operate in a transmit-receive mode or in a receive-only mode.

[0067] The RF driver unit 722 includes a gate modulator (not shown), an RF power amplifier (not shown), and an RF oscillator (not shown), which are used to drive an RF coil (e.g., the RF coil unit 715) and form a high-frequency magnetic field in the imaging space 718. Based on a control signal from the controller unit 725, the RF driver unit 722 uses the gate modulator to modulate an RF signal received from the RF oscillator into a signal having a predetermined envelope and a predetermined timing. The RF signal modulated by the gate modulator is amplified by the RF power amplifier and then output to the RF coil unit 715.

[0068] The gradient coil driver unit 723 drives the gradient coil unit 713 based on a control signal from the controller unit 725, thereby generating a gradient magnetic field in the imaging space 718. The gradient coil driver unit 723 includes three driver circuits (not shown) corresponding to the three gradient coils included in the gradient coil unit 713.

[0069] The data collection unit 724 includes a preamplifier (not shown), a phase detector (not shown), and an analog / digital converter (not shown), and is used to acquire the magnetic resonance signal received by the RF coil unit 714. In the data collection unit 724, the phase detector uses the output from the RF oscillator of the RF driver unit 722 as a reference signal to phase-detect the magnetic resonance signal received from the RF coil unit 714 and amplified by the preamplifier, and outputs the phase-detected analog magnetic resonance signal to the analog / digital converter for conversion into a digital signal. The digital signal thus obtained is output to the data processing unit 731.

[0070] The MRI apparatus 710 includes a bed 726 on which a subject 716 is placed. By moving the bed 726 based on a control signal from a controller unit 725, the subject 716 can be moved in and out of an imaging space 718.

[0071] The controller unit 725 includes a computer and a storage medium in which a program executed by the computer is stored. When the program is executed by the computer, it causes each part of the device to perform an operation corresponding to a predetermined scan. Examples of the storage medium include a ROM, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, and a non-volatile memory card. The controller unit 725 is connected to an operation console unit 732, processes an operation signal input to the operation console unit 732, and further outputs control signals to the bed 726, the RF driver unit 722, the gradient coil driver unit 723, and the data acquisition unit 724 to control them. The controller unit 725 also controls the data processing unit 731 and the display unit 733 based on the operation signal received from the operation console unit 732 to obtain a desired image.

[0072] The operation console unit 732 includes a user input device (such as a touch panel, a keyboard, a mouse, etc.). The operation console unit 732 is used by an operator to input data such as an imaging protocol and to set an area where an imaging sequence is to be executed. Data related to the imaging protocol and the execution area of ​​the imaging sequence is output to the controller unit 725.

[0073] The data processing unit 731 includes a computer and a storage medium in which a program to be executed by the computer to perform a predetermined data processing is stored. The data processing unit 731 is connected to the controller unit 725 and executes data processing based on a control signal received from the controller unit 725. The data processing unit 731 is also connected to the data acquisition unit 724 and applies various image processing operations to the magnetic resonance signals output from the data acquisition unit 724 to generate spectrum data.

[0074] In one example, the storage medium of the data processing unit 731 may include instructions for automatic image registration of acquired 3D medical image volumes.

[0075] The display unit 733 includes a display device, and displays an image on a display screen of the display device based on a control signal received from the controller unit 725. The display unit 733 displays, for example, an image related to an input item of operation data input by an operator from the operation console unit 732. The display unit 733 also displays a two-dimensional (2D) slice image or a three-dimensional (3D) image of the subject 716 generated by the data processing unit 731.

[0076] Although an MRI system has been described as an example, it should be understood that the present technique of automatic registration of 3D medical image volumes is also useful when applied to images acquired using other imaging modalities, such as CT, tomosynthesis, PET, C-arm angiography, etc. This description of the MRI imaging modality is provided merely as an example of one suitable imaging modality.

[0077] Referring to FIG. 8, an image processing system 802 is shown incorporated in an imaging system 800. The imaging system 800 can be configured to acquire 3D medical images of an imaging object using an imaging device 826 and process the acquired 3D medical images using the image processing system 802, for example, to automatically register the 3D medical images to one or more standard views. The registered images are displayed to a user by a display device 824. User input can be received by the image processing system through a user input device 822, and one or more of the image acquisition and image registration can be adjusted based on the received user input. As an example, following 3D image acquisition, a user can specify an anatomical region of interest for which the user would like to display one or more standard views. The image processing system 802 can receive user input from the user input device 822 and automatically determine transformation parameters for registering the anatomical region of interest and generate the requested standard views, for example, using a multiplanar reconstruction (MPR) module 812. The image processing system 802 can then automatically apply the transformation parameters to the 3D medical image to generate a registered sub-volume of interest and display the registered sub-volume of interest (or one or more 2D image slices of the registered sub-volume of interest) to a user via the display device 824.

[0078] In some embodiments, at least a portion of the image processing system 802 is located in a device (e.g., an edge device, a server, etc.) that is communicatively coupled to the MRI system 710 through wired and / or wireless connections. In some embodiments, at least a portion of the image processing system 802 is located in a separate device (e.g., a workstation) that can receive images from the MRI system 710 or from a storage device that stores images / data generated by the MRI system 710. The image processing system 802 can be operatively / communicatively coupled to a user input device 822 and a display device 824. The user input device 822 can be integrated into the MRI system, for example, in an operator console unit 732 of the MRI system 710. Similarly, the display device 824 can be integrated into the MRI system, for example, in a display unit 733 of the MRI system 710.

[0079] The image processing system 802 includes a processor 804 configured to execute machine-readable instructions stored in a non-transitory memory 806. The processor 804 may be single-core or multi-core, and programs executed by the processor may be configured for parallel or distributed processing. In some embodiments, the processor 804 may include individual components that are optionally distributed across two or more devices, which may be remotely located and / or configured for cooperative processing. In some embodiments, one or more aspects of the processor 804 may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.

[0080] The non-transitory memory 806 can store a pre-processing module 808, a landmark detection module 810, and a multi-planar reconstruction (MPR) module 812.

[0081] The pre-processing module 808 can include instructions for identifying and isolating one or more anatomical regions of interest, as well as instructions for segmenting symmetrical anatomical regions of interest, and instructions for performing other image processing operations. In one example, the pre-processing module 808 can include a machine learning (ML) model trained to map and classify 2D medical images as including or belonging to an anatomical region of interest. In some embodiments, the pre-processing module 808 can include instructions for selecting a sub-volume of interest from the 3D medical image volume. The selection can be based on identifying a subset of a plurality of 2D images that includes the anatomical region of interest. The selected subset of 2D images can then be cropped to remove non-anatomical regions, thereby focusing on relevant anatomical features. In certain embodiments, the 3D image volume may include symmetrical anatomical features. In such cases, the pre-processing module 808 can be configured to identify pairs of symmetrical anatomical features and select a sub-volume that includes one of the contrasting anatomical features for further processing. This is particularly useful when the anatomical feature of interest is bilaterally symmetrical (such as lungs or kidneys). The pre-processing module 808 may also include a left-right separation step to separate the left and right anatomical structures for separate processing. This is beneficial in situations where the anatomical structures on one side of the body are more interesting or more relevant to a medical diagnosis or treatment plan than the anatomical structures on the other side. Furthermore, the pre-processing module 808 may verify whether the acquired 3D image volume is suitable for registration before proceeding to other pre-processing. This verification may include checking the quality of the 3D image volume (such as image resolution and signal-to-noise ratio) to ensure that the quality of the 3D image volume meets the requirements of the subsequent image registration process. In some embodiments, the pre-processing module 808 may include a machine learning model trained to classify 2D medical images as including or belonging to an anatomical region of interest.The model can be used to assist in selecting sub-volumes of interest from a 3D medical image volume. The model can be trained using a variety of machine learning techniques.

[0082] The non-transitory memory 806 can further store a landmark detection module 810, which can include instructions for detecting a plurality of anatomical landmarks in the subvolume of interest using a deep neural network trained to identify anatomical landmarks. The landmark detection module 810 can include a feature extraction component configured to extract features from the subvolume of interest. These features can be used to identify candidates for anatomical landmarks in the subvolume of interest. In some embodiments, the landmark detection module 810 can include a landmark classification component configured to classify anatomical landmarks based on the extracted features. The landmark classification component can use a machine learning model (such as a deep neural network) trained to classify anatomical landmarks based on the features of the anatomical landmarks. In certain embodiments, the landmark detection module 810 can include a landmark coding component configured to code the classified anatomical landmark as a feature vector. The landmark coding component can compare the feature vector to a distribution of previously coded feature vectors of the same landmark type to determine a probability that the landmark belongs to the distribution of previously coded feature vectors. A confidence level for the landmark can be determined based on the probability. These features include, but are not limited to, the location of the landmark in the subvolume of interest, the intensity of the landmark, the shape of the landmark, and the texture of the landmark. The extracted features may be coded as a feature vector for each landmark. In another embodiment, the landmark detection module 810 may include a feature normalization component that normalizes the extracted features before coding them as a feature vector. The feature normalization component may use various normalization techniques (such as min-max normalization, z-score normalization, or decimal scaling normalization) to ensure that the features have similar scales.This helps prevent features with large scales from dominating the coding process. In yet another embodiment, landmark detection module 810 can include a feature selection component that selects a subset of the extracted features for coding. The feature selection component can use various feature selection techniques (such as mutual information, chi-squared test, or recursive feature elimination) to select the most informative features for coding.

[0083] In other embodiments, the landmark detection module 810 can include a feature coding component that uses a machine learning model (such as an autoencoder) to code the selected features as a feature vector. The autoencoder is trained to minimize the reconstruction error between the input features and the reconstructed features, resulting in a feature vector that captures the most important information about the landmark. In another embodiment, the landmark detection module 810 can include a feature comparison component that compares the coded feature vector to a distribution of previously coded feature vectors of the same landmark type. The feature comparison component can use various distance measures (such as Euclidean distance, Manhattan distance, or cosine similarity) to determine the similarity between the coded feature vector and the distribution of previously coded feature vectors. The confidence of the landmark can be determined based on this similarity.

[0084] The non-transitory memory 806 also stores an MPR module 812, which may include instructions for performing one or more MPR algorithms known in the art of medical imaging, including resampling image data from an image sequence along a landmark plane or other plane of interest to synthesize a view of an anatomical structure of interest in a desired orientation. The MPR module 812 may further include instructions for estimating transformation parameters based on the detected anatomical landmarks. These transformation parameters may include rotation angles and translation vectors used to adjust the orientation and position of the sub-volume of interest. The MPR module 812 may be configured to apply these transformation parameters to the sub-volume of interest to generate a first registered sub-volume of interest. In some embodiments, the MPR module 812 may include instructions for determining an anatomical plane of interest relative to an image coordinate system based on a subset of the landmarks detected by the landmark detection module 810. The subset of landmarks may include at least three landmarks detected within the sub-volume of interest. The MPR module 812 can then perform a multi-planar reconstruction of the sub-volume of interest along the anatomical plane of interest based on one or more geometric constraints imposed on the subset of landmarks.

[0085] The non-transitory memory 806 further stores a geometric constraint module 814, which may include instructions for determining whether the position of one or more landmarks detected by the landmark detection module 810 satisfies one or more pre-determined geometric constraints, which may be stored in the geometric constraint module 814 (e.g., a database). The geometric constraint module 814 may include instructions for determining relative positions of landmarks in a plurality of landmarks. For example, one landmark may be required to be located above another landmark, and one landmark may be required to be located to the right of another landmark. The geometric constraints may also include distances of pairs of landmarks with respect to one or more axes of the image coordinate system. For example, the difference between the y coordinates of the two landmarks may be constrained to be zero or within a threshold. Additionally, the geometric constraint module 814 may include instructions for determining an angle between a line connecting the pair of landmarks with respect to one or more axes of the image coordinate system. For example, the angle between a line connecting the two landmarks and a point on the x-axis may be constrained to be 90 degrees or within a threshold of 90 degrees.

[0086] The geometric constraint module 814 may also include instructions for determining a satisfaction level of the geometric constraints of the landmarks. The satisfaction level provides an indication of the accuracy of the registration process. A higher satisfaction level corresponds to a higher confidence level of the registration. Furthermore, the geometric constraint module 814 may include instructions for determining a confidence level of the geometric constraints based on the satisfaction level of the geometric constraints. The confidence level may be a function of the satisfaction level, with a higher satisfaction level corresponding to a higher confidence level. The geometric constraint confidence level may be normalized to a range between a minimum confidence value and a maximum confidence value. For example, it may be normalized to a range between 0 and 1, where 0 represents no confidence and 1 represents maximum confidence. The geometric constraint module 814 may also include instructions for adjusting the geometric constraint confidence level based on additional factors, such as the quality of the 3D image volume, the complexity of the anatomical region of interest, or the performance of the deep neural network in detecting the landmarks.

[0087] The non-transitory memory also stores a confidence estimation module 816, which may include instructions for determining a confidence of one or more landmarks detected by the landmark detection module 810 and instructions for determining a confidence for the transformation parameters based on the sub-volumes aligned by the transformation parameters. The confidence estimation module 816 may include instructions for determining a confidence of the transformation parameters based on a satisfaction of geometric constraints of the landmarks of the aligned sub-volumes. The confidence determination process may include comparing the positions of the landmarks in the aligned sub-volumes to one or more pre-determined geometric constraints stored in the geometric constraint module 814. The satisfaction of these geometric constraints provides an indication of the accuracy of the alignment process. The confidence based on the satisfaction of the geometric constraints quantifies the confidence of the alignment based on the satisfaction of the geometric constraints. Additionally, the confidence estimation module 816 may include instructions for projecting the aligned sub-volumes onto a projection plane to generate a projection image. The projection image is mapped to a projection confidence using a confidence estimation network. The projection confidence provides a quantitative measure of the accuracy of the registration process based on the quality of the projection images. The confidence estimation module 816 may also include instructions for determining an overall confidence of the registered sub-volume by combining the geometric constraint confidence and the projection confidence. This overall confidence provides an overall measure of the accuracy of the registration process, taking into account both the satisfaction of the geometric constraints and the quality of the projection images.

[0088] In some embodiments, non-transitory memory 806 may include components located on two or more devices, and the two or more devices may be located remotely and / or configured for cooperative processing. In some embodiments, one or more aspects of non-transitory memory 806 may include remotely accessible, networked storage devices configured in a cloud computing arrangement.

[0089] The user input device 822 may include one or more of a touch screen, keyboard, mouse, track pad, motion sensing camera, or other device configured to allow a user to interact with and manipulate data in the image processing system 802. In one embodiment, the user input device 822 may enable a user to select an anatomical region of interest that the user desires to view in one or more standard views.

[0090] The display device 824 can include one or more display devices utilizing virtually any type of technology. In some embodiments, the display device 824 can include a computer monitor and can display the registered medical image volume (such as an automatically registered image volume generated according to one or more of the methods disclosed herein). The display device 824 can be combined with the processor 804, the non-transitory memory 806, and / or the user input device 822 in a shared housing or can be a peripheral display device and can include a monitor, touch screen, projector, or other display device known in the art that allows a user to view the MRI generated by the MRI system and / or various data stored in the non-transitory memory 806.

[0091] It should be understood that the imaging system 800 depicted in Figure 8 is intended to be illustrative and not limiting, and other suitable imaging systems may include more, fewer, or different components.

[0092] The present disclosure also supports a method for automatic image registration of three-dimensional (3D) medical image volumes, the method including preprocessing the 3D medical image volume by selecting a sub-volume of interest, detecting a plurality of anatomical landmarks in the sub-volume of interest using a deep neural network trained to identify anatomical landmarks, estimating transformation parameters based on the plurality of anatomical landmarks to adjust a rotation angle and a translation of the sub-volume of interest, adjusting a rotation angle and a translation of the sub-volume of interest using the transformation parameters to generate a first registered sub-volume of interest, determining a confidence level of the transformation parameters based on the first registered sub-volume of interest, and iteratively improving the transformation parameters if the confidence level is below a predetermined threshold. In a first embodiment of the method, the 3D medical image volume is an image sequence including a plurality of two-dimensional (2D) images, and preprocessing the 3D medical image volume by selecting the sub-volume of interest includes selecting a subset of the plurality of 2D images that includes the sub-volume of interest, and cropping the subset of the plurality of 2D images to remove regions outside the sub-volume of interest. In a second embodiment of the method, optionally including the first embodiment, selecting the sub-volume of interest includes identifying a pair of symmetrical anatomical features of the 3D medical image volume, the pair of symmetrical anatomical features including a first anatomical feature and a second anatomical feature, and selecting a sub-volume that includes the first anatomical feature and does not include the second anatomical feature. In a third embodiment of the method, optionally including one or both of the first and second embodiments, the plurality of anatomical landmarks includes at least three different landmarks.In a fourth embodiment of the method, optionally including any one or more of the first to third embodiments, estimating transformation parameters based on the plurality of anatomical landmarks to adjust a rotation angle and a translation of the sub-volume of interest includes selecting a subset of landmarks from the plurality of anatomical landmarks detected in the sub-volume of interest, the subset of landmarks including at least three landmarks, and determining an anatomical plane of interest relative to an image coordinate system based on the subset of landmarks. In a fifth embodiment of the method, optionally including any one or more of the first to fourth embodiments, adjusting a rotation angle and a translation of the sub-volume of interest using the transformation parameters to generate the first registered sub-volume of interest includes performing a multi-planar reconstruction of the sub-volume of interest along an anatomical plane of interest based on one or more geometric constraints imposed on the subset of landmarks. A sixth embodiment of the method optionally includes any one or more of the first to fifth embodiments, and determining a confidence level of the transformation parameters based on the first registered sub-volume of interest includes detecting a plurality of anatomical landmarks in the first registered sub-volume of interest using the deep neural network, comparing positions of the plurality of anatomical landmarks in the first registered sub-volume of interest to one or more predetermined geometric constraints to determine a satisfaction level of the geometric constraints, and determining the confidence level of the transformation parameters based on the satisfaction level of the geometric constraints. A seventh embodiment of the method optionally includes any one or more of the first to sixth embodiments, and the transformation parameters include a rotation angle and a translation vector.In an eighth embodiment of the method, optionally including one or more or each of the first to seventh embodiments, iteratively improving the transformation parameters includes using the deep neural network to detect a plurality of anatomical landmarks in the first aligned sub-volume of interest; estimating improved transformation parameters based on the plurality of anatomical landmarks to adjust a rotation angle and a translation of the first aligned sub-volume of interest; adjusting a rotation angle and a translation of the first aligned sub-volume of interest using the improved transformation parameters to generate a second aligned sub-volume of interest; determining a confidence level of the improved transformation parameters based on the second aligned sub-volume of interest; and iteratively further improving the improved transformation parameters if the confidence level is below a predetermined threshold. In a ninth embodiment of the method, optionally including one or more or each of the first to eighth embodiments, the method further comprises determining a confidence level of the improved transformation parameters based on the second aligned sub-volume of interest by projecting the second aligned sub-volume of interest onto a plurality of projection planes to generate a plurality of projection images, mapping the plurality of projection images to a corresponding plurality of confidence scores using a confidence estimation network, and determining a confidence level of the improved transformation parameters by majority voting using the plurality of confidence scores corresponding to the plurality of projection images.

[0093] The present disclosure also supports an image processing system, the image processing system including a display device, a non-transitory memory including instructions, and a processor that, when executed, causes the image processing system to receive a three-dimensional (3D) image volume, pre-process the 3D image volume by selecting a sub-volume of interest, detect a plurality of landmarks in the sub-volume of interest using a deep neural network trained to identify anatomical landmarks, estimate transformation parameters based on the plurality of landmarks to adjust a rotation angle and a translation of the sub-volume of interest, use the transformation parameters to adjust a rotation angle and a translation of the sub-volume of interest to generate a first registered sub-volume of interest, determine a confidence level of the transformation parameters based on the first registered sub-volume of interest, and iteratively improve the transformation parameters if the confidence level is below a predetermined threshold. In a first embodiment of the system, the plurality of landmarks includes at least three different landmarks. In a second embodiment of the system, optionally including the first embodiment, the processor further causes the image processing system to select a subset of landmarks from the plurality of landmarks detected in the sub-volume of interest, the subset of landmarks including at least three landmarks, and to determine an anatomical plane of interest relative to an image coordinate system based on the subset of landmarks. In a third embodiment of the system, optionally including one or both of the first and second embodiments, the processor further causes the image processing system to perform a multi-planar reconstruction of the sub-volume of interest along the anatomical plane of interest.A fourth embodiment of the system optionally includes one or more of the first to third embodiments, and the processor further causes the image processing system to: detect the plurality of landmarks in the first aligned sub-volume of interest using the deep neural network; compare positions of the plurality of landmarks in the first aligned sub-volume of interest to one or more predetermined geometric constraints to determine a satisfaction of the geometric constraints; and determine the confidence of the transformation parameters based on the satisfaction of the geometric constraints. A fifth embodiment of the system optionally includes one or more of the first to fourth embodiments, and the processor further causes the image processing system to: project the first aligned sub-volume of interest onto a projection plane to generate a projection image, the projection plane being parallel to a plurality of 2D images including the first aligned sub-volume of interest; and map the projection image to a confidence score using the confidence estimation network.

[0094] The disclosure also supports a method, including receiving a three-dimensional (3D) image volume including a first plurality of two-dimensional (2D) images parallel to a first plane, preprocessing the 3D image volume by selecting a subset of the first plurality of 2D images to generate a sub-volume of interest including an anatomical region of interest, detecting at least three landmarks in the sub-volume of interest using a deep neural network trained to identify anatomical landmarks, estimating transformation parameters including an anatomical plane of interest intersecting the sub-volume of interest based on the at least three landmarks, performing a multi-planar reconstruction of the sub-volume of interest along the anatomical plane of interest to generate a first registered sub-volume of interest, the first registered sub-volume of interest including a second plurality of 2D images parallel to the anatomical plane of interest, determining a confidence level of the transformation parameters based on the first registered sub-volume of interest, and iteratively improving the transformation parameters if the confidence level is below a predetermined threshold. In a first embodiment of the method, detecting the at least three landmarks includes detecting a plurality of landmarks including four or more landmarks in the subvolume of interest, and for each landmark of the plurality of landmarks includes coding the landmark as a feature vector, comparing the feature vector to a distribution of previously coded feature vectors of the same landmark type as the landmark, determining a probability that the landmark belongs to a distribution of previously coded feature vectors, and determining a confidence level for the landmark based on the probability. In a second embodiment of the method, optionally including the first embodiment, the method further includes filtering the plurality of landmarks based on a confidence level to obtain the at least three landmarks.In a third embodiment of the method, optionally including one or both of the first and second embodiments, determining a confidence level of the transformation parameters based on the first aligned sub-volume of interest includes: detecting at least three landmarks in the first aligned sub-volume of interest using the deep neural network; comparing positions of the at least three landmarks in the first aligned sub-volume of interest with one or more predetermined geometric constraints to determine a satisfaction level of the geometric constraints; determining a confidence level of a geometric constraint based on the satisfaction level of the geometric constraints; projecting the first aligned sub-volume of interest onto a projection plane to generate a projection image; mapping the projection image to a projection confidence level using a confidence estimation network; and determining an overall confidence level of the first aligned sub-volume of interest by combining the confidence level of the geometric constraints and the confidence level of the projection.

[0095] In this specification, elements or steps described in the singular and preceded by the words "a" or "an" should be understood as not excluding a plurality of elements or steps, unless the exclusion is expressly stated. Furthermore, references to "one embodiment" of the invention are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. Furthermore, unless expressly stated to the contrary, embodiments "comprising," "including," or "having" an element or elements having a particular characteristic may include additional elements that do not have that characteristic. The terms "including" and "in which" are used as the plain-language equivalents of the terms "comprising" and "wherein," respectively. Furthermore, terms such as "first," "second," and "third" are used merely as labels, and are not intended to impose numerical requirements and a specific positional order on the objects.

[0096] The description set forth herein uses examples to disclose the invention, including the best mode, and also enables any person skilled in the art to practice the invention, including making and using any devices or systems, and performing any methods incorporated herein. The patentable scope of the invention is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they contain structural elements that do not differ from the literal language of the claims, or if they contain equivalent structural elements that do not differ substantially from the literal language of the claims. [Explanation of symbols]

[0097] 712 Static Magnetic Field Magnet Unit 713 Gradient Coil Unit 714 RF coil unit 716 Subject 718 Imaging Space 720 TR Switch 722 RF driver unit 723 Gradient Coil Driver Unit 724 Data Acquisition Unit 725 Controller Unit 726 Beds 731 Data Processing Unit 732 Operation console unit 733 Display Unit 800 Imaging System 802 Image Processing System 804 Processor 806 Temporary Memory 808 Pre-treatment Module 810 Landmark Detection Module 812 MPR Module 814 Geometric Constraint Module 816 Reliability Estimation Module 822 User Input Device 824 Display device 826 Imaging Equipment

Claims

1. 1. A method for automatic image registration of three-dimensional (3D) medical image volumes, comprising: pre-processing the 3D medical image volume by selecting a sub-volume of interest; detecting a plurality of anatomical landmarks in the subvolume of interest using a deep neural network trained to identify anatomical landmarks; estimating transformation parameters based on the plurality of anatomical landmarks to adjust a rotation angle and a translation of the sub-volume of interest; adjusting a rotation angle and a translation of the sub-volume of interest using the transformation parameters to generate a first aligned sub-volume of interest; determining a confidence level for the transformation parameters based on the first registered sub-volume of interest; and iteratively improving the transformation parameters if the confidence falls below a predefined threshold. A method comprising:

2. The 3D medical image volume is an image sequence including a plurality of two-dimensional (2D) images, and pre-processing the 3D medical image volume by selecting the sub-volume of interest comprises: selecting a subset of the plurality of 2D images that includes the subvolume of interest; and Cropping a subset of the plurality of 2D images to remove areas outside the subvolume of interest. The method of claim 1 , comprising:

3. Selecting the sub-volume of interest comprises: identifying a pair of bilaterally symmetric anatomical features in the 3D medical image volume, the pair of bilaterally symmetric anatomical features including a first anatomical feature and a second anatomical feature; and Selecting a sub-volume that includes the first anatomical feature and does not include the second anatomical feature. The method of claim 1 , comprising:

4. The method of claim 1 , wherein the plurality of anatomical landmarks includes at least three different landmarks.

5. estimating transformation parameters based on the plurality of anatomical landmarks to adjust a rotation angle and a translation of the sub-volume of interest, selecting a subset of landmarks from the plurality of anatomical landmarks detected within the subvolume of interest, the subset of landmarks including at least three landmarks; and Determining an anatomical plane of interest relative to an image coordinate system based on a subset of the landmarks. The method of claim 1 , comprising:

6. 6. The method of claim 5, wherein adjusting a rotation angle and a translation of the sub-volume of interest using the transformation parameters to generate the first aligned sub-volume of interest comprises performing a multi-planar reconstruction of the sub-volume of interest along an anatomical plane of interest based on one or more geometric constraints imposed on a subset of landmarks.

7. Determining a confidence level of the transformation parameters based on the first registered sub-volume of interest includes: detecting the plurality of anatomical landmarks in the first registered sub-volume of interest using the deep neural network; comparing positions of the plurality of anatomical landmarks of the first registered sub-volume of interest to one or more predetermined geometric constraints to determine satisfaction of the geometric constraints; and determining the confidence of the transformation parameters based on satisfaction of geometric constraints; The method of claim 1 , comprising:

8. The method of claim 1 , wherein the transformation parameters include a rotation angle and a translation vector.

9. Iteratively improving the transformation parameters includes: detecting the plurality of anatomical landmarks in the first registered sub-volume of interest using the deep neural network; estimating improved transformation parameters based on the plurality of anatomical landmarks to adjust a rotation angle and a translation of the first registered sub-volume of interest; adjusting a rotation angle and a translation of the first registered sub-volume of interest using the improved transformation parameters to generate a second registered sub-volume of interest; determining a confidence level of the improved transformation parameters based on the second registered sub-volume of interest; and if the confidence falls below a predetermined threshold, iteratively further improving the improved transformation parameters. The method of claim 1 , comprising:

10. The method further comprises: determining a confidence level of the improved transformation parameters based on the second registered sub-volume of interest; projecting the second registered sub-volume of interest onto a plurality of projection planes to generate a plurality of projection images; mapping the plurality of projection images to a corresponding plurality of confidence scores using a confidence estimation network; and determining a confidence level of the improved transformation parameters by majority voting using the plurality of confidence scores corresponding to the plurality of projection images; The method of claim 9 , comprising determining by:

11. 1. An image processing system, comprising: display device, a non-transitory memory containing instructions; and A processor, the processor, when executing the instructions, causing the image processing system to: receiving a three-dimensional (3D) image volume; pre-processing the 3D image volume by selecting a sub-volume of interest; detecting a plurality of landmarks in the subvolume of interest using a deep neural network trained to identify anatomical landmarks; estimating transformation parameters based on the plurality of landmarks to adjust a rotation angle and a translation of the sub-volume of interest; adjusting a rotation angle and a translation of the sub-volume of interest using the transformation parameters to generate a first aligned sub-volume of interest; determining a confidence level for the transformation parameters based on the first registered sub-volume of interest; and iteratively improving the transformation parameters if the confidence falls below a predefined threshold. A processor that executes 2. An image processing system comprising:

12. The image processing system of claim 11 , wherein the plurality of landmarks includes at least three different landmarks.

13. The processor further configures the image processing system to: selecting a subset of landmarks from the plurality of landmarks detected within the subvolume of interest, the subset of landmarks including at least three landmarks; and Determining an anatomical plane of interest relative to an image coordinate system based on a subset of the landmarks. The image processing system according to claim 11 ,

14. The processor further configures the image processing system to: The image processing system of claim 13 , further configured to perform a multi-planar reconstruction of the sub-volume of interest along the anatomical plane of interest.

15. The processor further configures the image processing system to: detecting the plurality of landmarks in the first registered sub-volume of interest using the deep neural network; comparing the positions of the plurality of landmarks of the first registered sub-volume of interest to one or more predetermined geometric constraints to determine satisfaction of the geometric constraints; and determining the confidence level of the transformation parameters based on satisfaction of geometric constraints; The image processing system according to claim 11 ,

16. The processor further configures the image processing system to: projecting the first registered sub-volume of interest onto a projection plane to generate a projection image, the projection plane being parallel to a plurality of 2D images including the first registered sub-volume of interest; and Mapping the projection images to confidence scores using the confidence estimation network. The image processing system according to claim 11 ,

17. 1. A method comprising: receiving a three-dimensional (3D) image volume including a first plurality of two-dimensional (2D) images parallel to a first plane; pre-processing the 3D image volume by selecting a subset of the first plurality of 2D images to generate a sub-volume of interest that includes an anatomical region of interest; detecting at least three landmarks in the sub-volume of interest using a deep neural network trained to identify anatomical landmarks; estimating transformation parameters including an anatomical plane of interest that intersects with the sub-volume of interest based on the at least three landmarks; performing a multi-planar reconstruction of the sub-volume of interest along the anatomical plane of interest to generate a first registered sub-volume of interest, the first registered sub-volume of interest including a second plurality of 2D images parallel to the anatomical plane of interest; determining a confidence level for the transformation parameters based on the first registered sub-volume of interest; and iteratively improving the transformation parameters if the confidence falls below a predefined threshold. A method comprising:

18. Detecting the at least three landmarks includes: detecting a plurality of landmarks, including four or more landmarks, in the sub-volume of interest; For each landmark of the plurality of landmarks, encoding the landmarks as feature vectors; comparing the feature vector to a distribution of feature vectors previously coded with the same landmark type as the landmark; determining the probability that the landmark belongs to a distribution of previously coded feature vectors; and determining a confidence level of the landmark based on the probability; 20. The method of claim 17, comprising:

19. The method further comprises filtering the plurality of landmarks based on a confidence level to obtain the at least three landmarks.

20. The method of claim 18, further comprising:

20. Determining a confidence level of the transformation parameters based on the first registered sub-volume of interest includes: detecting the at least three landmarks in the first registered sub-volume of interest using the deep neural network; comparing positions of the at least three landmarks in the first registered sub-volume of interest to one or more predetermined geometric constraints to determine satisfaction of the geometric constraints; determining a confidence level for the geometric constraint based on the satisfaction level of the geometric constraint; projecting the first aligned sub-volume of interest onto a projection plane to generate a projection image; mapping said projection images to projection confidences using a confidence estimation network; and determining an overall confidence level for the first registered sub-volume by combining a confidence level for the geometric constraint and a confidence level for the projection; 20. The method of claim 17, comprising:

Images