System and method for automatic image alignment of three-dimensional image volume

Abstract

The present disclosure provides systems and methods for automatic image alignment of a three-dimensional (3D) medical image volume. The method comprises: preprocessing the 3D medical image volume by selecting a sub-volume of interest (104); detecting an anatomical landmark in the sub-volume using a deep neural network (106); estimating transformation parameters based on the anatomical landmark to adjust the angle of rotation and translation of the sub-volume (108); adjusting the angle of rotation and the translation to produce a first aligned sub-volume (110); determining a confidence of the transformation parameter based on the first alignment sub-volume (112); and iteratively refining the transformation parameter if the confidence is below a predetermined threshold (114). The disclosed method for automatic image alignment reduces the need for manual alignment and increases the probability that the 3D image volume converges to the desired orientation compared to conventional methods.

Description

Technical Field

[0001] Embodiments of the subject matter disclosed herein relate to medical imaging, and more particularly, to multi-planar reformatting of three-dimensional medical images. Background Art

[0002] Three-dimensional (3D) medical imaging is a medical imaging technique that acquires imaging data of a volume of an imaged subject, rather than a single projection or two-dimensional (2D) slice of the imaged subject. 3D medical images can be acquired using a variety of imaging modalities, including computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound. 3D medical image data includes a series of cross-sectional images that can be combined to create a 3D representation of the patient's anatomical structure for diagnosis or treatment of various medical conditions. The series of cross-sectional images may include a sequence of images parallel to a first plane, where the first plane may be one of a set of predetermined planes (such as an axial plane, a coronal plane, or a sagittal plane), or the first plane may be dynamically determined to capture the anatomical region of interest at a desired orientation (e.g., scan plane planning, which may be performed using a 2D localizer image). However, the series of cross-sectional images may not capture the anatomical region of interest at a desired orientation. For example, if the plane defining the desired view of the anatomical structure of interest is not parallel to the first plane, the series of cross-sectional images may not include an image that captures the anatomical structure of interest at a desired orientation. In such cases, a realignment / reorganization of the 3D medical imaging data may be performed to "synthesize" a view of the anatomy of interest at a desired orientation.

[0003] In some embodiments, the 3D medical imaging data can be realigned by performing multi-planar reformation (MPR) of the series of cross-sectional images along a second plane, wherein the second plane can be selected by a technician or a radiologist. By orienting the anatomical features captured by the 3D medical images to match one or more standard orientations, a more consistent and more accurate interpretation and subsequent diagnosis of the medical images can be performed. The alignment of the images is usually completed by manually adjusting the position of the images (e.g., by selecting a second plane / plane of interest, along which the series of cross-sectional images are reorganized) until they are in the desired orientation. This process is usually completed by a technician or a radiologist, who must carefully adjust the images until the images are properly aligned. However, manual alignment of images is a tedious and time-consuming process, typically taking up to three minutes per image. In addition, manual alignment is prone to error, especially when performed by a novice technician. Inaccurate image alignment may lead to incorrect interpretation of imaging data, resulting in incorrect diagnosis and treatment. Therefore, a more efficient and accurate image alignment method is needed. Summary of the invention

[0004] In one embodiment, a method for automatically aligning a 3D medical image volume to show a predetermined view of interest at a predetermined orientation includes: preprocessing the 3D medical image volume by selecting a subvolume of interest; detecting multiple landmarks in the subvolume of interest using a deep neural network trained to identify anatomical landmarks; estimating transformation parameters based on the multiple anatomical landmarks to adjust the rotation angle and translation of the subvolume of interest; using the transformation parameters to adjust the rotation angle and translation of the subvolume of interest to produce a first aligned subvolume of interest; determining a confidence level of the transformation parameters based on the first aligned subvolume of interest; and iteratively refining the transformation parameters if the confidence level is below a predetermined threshold.

[0005] In another embodiment, an image processing system includes: a display device; a non-volatile memory comprising instructions; and a processor, wherein when the instructions are executed, the processor causes the image processing system to: receive a 3D image volume; pre-process the 3D image volume by selecting a subvolume of interest; detect multiple landmarks in the subvolume of interest using a deep neural network trained to identify anatomical landmarks; estimate transformation parameters based on the multiple landmarks to adjust the rotation angle and translation of the subvolume of interest; use the transformation parameters to adjust the rotation angle and translation of the subvolume of interest to produce a first aligned subvolume of interest; determine a confidence level of the transformation parameters based on the first aligned subvolume of interest; and iteratively refine the transformation parameters if the confidence level is below a predetermined threshold.

[0006] In another embodiment, a method includes: receiving a 3D image volume including a first plurality of 2D images parallel to a first plane; preprocessing the 3D image volume by selecting a subset of the first plurality of 2D images to generate a subvolume of interest including an anatomical region of interest; detecting at least three landmarks in the subvolume of interest using a deep neural network trained to identify anatomical landmarks; estimating transformation parameters of an anatomical plane of interest intersecting the subvolume of interest based on the at least three landmarks; performing multiplanar reformation of the subvolume of interest along the anatomical plane of interest to generate a first aligned subvolume of interest, wherein the first aligned subvolume of interest includes a second plurality of 2D images parallel to the anatomical plane of interest; determining confidence in the transformation parameters based on the first aligned subvolume of interest; and iteratively refining the transformation parameters if the confidence is below a predetermined threshold.

[0007] In this way, the 3D medical image volume can be automatically aligned to show the anatomical region of interest in a standard orientation without time-consuming manual alignment. In addition, if the confidence is below a predetermined threshold, the likelihood of successful alignment is increased by iteratively refining the transformation parameters to improve the alignment, because the inventors of this article have found that the closer the initial orientation of the 3D image volume is to the desired orientation, the greater the likelihood that the 3D image volume will converge to the desired orientation.

[0008] It should be understood that the above brief description is provided to introduce in a simplified form selected concepts that are further described in the detailed description. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is solely defined by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a flow chart illustrating a method 100 for automatic alignment of a 3D image volume according to an embodiment of the present disclosure;

[0010] Figure 2 is a flow chart illustrating a method 200 for preprocessing a 3D image volume to generate a sub-volume of interest according to an embodiment of the present disclosure;

[0011] Figure 3 is a flow chart illustrating a method 300 for detecting landmarks in a subvolume of interest according to an embodiment of the present disclosure;

[0012] Figure 4 is a flow chart illustrating a method 400 for estimating a rotation angle and a translation vector of an anatomical plane of interest according to an embodiment of the present disclosure;

[0013] Figure 5 is a flow chart illustrating a method 500 for determining confidence in alignment of a subvolume of interest according to an embodiment of the present disclosure;

[0014] Figure 6 is a flow chart illustrating a method 600 for adjusting a rotation angle and a translation of a subvolume of interest using transformation parameters according to an embodiment of the present disclosure;

[0015] Figure 7 is a block diagram of an MRI system according to an embodiment of the present disclosure; and

[0016] Figure 8 is a block diagram of an image processing system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The following description relates to various methods and systems for automatically aligning a 3D image volume to capture anatomical features of interest in standard planes and orientations, thereby avoiding the need for manual image alignment that can be time-consuming and inconsistent. In some embodiments, automatic image alignment may include four aspects: preprocessing, landmark detection, alignment correction, and alignment confidence estimation, wherein preprocessing may verify whether the acquired 3D image volume is suitable for alignment, landmark detection identifies anatomical reference points for alignment, alignment correction aligns the 3D image volume with a desired plane in a desired orientation, and alignment confidence estimation estimates the accuracy of the automatic alignment process, and if a confidence threshold is not met, an iterative alignment adjustment process may be invoked.

[0018] In one aspect, the 3D image volume is pre-processed to determine whether a desired field of view (FOV) is included in the 3D image volume. A sub-volume of interest including the desired field of view can be segmented from the 3D image volume by identifying a 2D image slice of the 3D image volume that includes an anatomical feature of interest. Additionally, the pre-processing can include a laterality separator step to separate left and right anatomical structures for separate processing.

[0019] In another aspect, a deep neural network trained to segment anatomical landmarks can be used to detect landmarks in the subvolume, where the landmarks may include points, 3D cylindrical structures, or anatomical structures. The landmarks detected in the subvolume are based on predefined anatomical landmarks associated with the desired viewing plane.

[0020] In another aspect, the detected landmarks may be filtered by confidence, wherein the confidence of the detected landmarks may be based on features (e.g., shape and appearance) of the detected landmarks. In one embodiment, a feature distribution is created for each landmark type, and the confidence of the landmarks detected in the subvolume is determined by calculating the probability that the landmark, as encoded by its features, belongs to the feature distribution of its corresponding landmark type. The filtered landmarks (e.g., three or more landmarks with the highest confidence) may be used to determine an anatomical plane of interest relative to the image coordinate system, and further, the rotation angle and translation vector of the anatomical plane of interest may be determined by comparing the position of the filtered landmarks with one or more predetermined geometric constraints. The anatomical plane of interest and the rotation angle and translation vector may be collectively referred to herein as transformation parameters. The 3D image volume may then be reorganized based on the transformation parameters using techniques such as multi-planar reformation (MPR) to produce aligned subvolumes.

[0021] In another aspect, confidence in the transformation parameters used to generate the aligned subvolumes can be determined based on satisfaction of the geometric constraints by the landmarks in the aligned subvolumes and further based on the projection-based confidence. In one embodiment, a maximum intensity projection (MIP) is generated from the aligned subvolumes and a machine learning / deep learning network is used to classify whether the aligned subvolumes are within a predetermined rotational tolerance limit relative to the desired orientation. If the confidence is below a threshold, the alignment workflow can be repeated for a predetermined number of iterations. If the confidence remains below the threshold after a predetermined number of iterations, the alignment can be declared a failure.

[0022] In one embodiment, the imaging device (such as Figure 7 ) to acquire a 3D medical image (also referred to herein as a 3D image, 3D medical imaging volume, and related terms). Figure 8 The image processing system 802 shown in FIG. 1 is used to automatically align a 3D medical image volume acquired by an imaging device to generate one or more standard views of one or more anatomical regions of interest. The image processing system 802 may perform an automatic image alignment method (such as Figure 1 The automatic image alignment method includes performing one or more pre-processing operations (such as Figure 2 Detect multiple landmarks (such as Figure 3 300 ); determining the transformation parameters of the subvolume of interest based on the detected landmarks (eg Figure 4 400); applying the transformation parameters to the subvolume of interest to produce an aligned subvolume of interest (eg, Figure 6 and determining a confidence level (e.g., a confidence level) of the transformation parameters based on the aligned subvolume of interest (e.g., the subvolume of interest after applying the determined transformation parameters). Figure 5 500 as described above).

[0023] refer to Figure 1 , a method 100 for automatically aligning a 3D image volume is shown. The method 100 can be used to align an anatomical region captured within a 3D image volume with a standard plane and orientation, so that a radiologist or other medical imaging practitioner can quickly view the anatomical region of interest in a standard orientation, which can facilitate rapid and accurate diagnosis based on the 3D image volume. In some embodiments, the method 100 can be performed by an image processing system.

[0024] Method 100 begins at operation 102, in which an image processing system receives a 3D image volume. The 3D image volume may include a sequence of 2D images representing a volume of an imaged subject. The 3D image volume may be acquired using various imaging modalities, such as computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, or other suitable imaging techniques. The 3D image volume may include a specific region of the imaged subject, such as the heart, brain, or other anatomical structures, and may include multiple anatomical regions.

[0025] At operation 104, the image processing system preprocesses the 3D image volume to generate a sub-volume of interest. In some embodiments, operation 104 includes verifying whether the 3D volume is suitable for alignment before further preprocessing. In one embodiment, the image processing system may determine whether a desired field of view (FOV) is included in the 3D image volume. If the desired FOV is not included in the 3D image volume, the image processing system may discard the 3D image volume or request a new 3D image volume that includes the desired FOV. This can ensure that the anatomical region of interest is included in the 3D image volume before further preprocessing or image alignment is performed.

[0026] In one embodiment, operation 104 includes selecting a subset of the plurality of 2D images that includes a subvolume of interest. The subset may then be cropped to remove non-anatomical regions and focus on relevant anatomical features. In some embodiments, the 3D image volume may include symmetrical anatomical features. For example, the 3D image volume may include both the left and right hemispheres of the brain, or both the left and right lungs. In such cases, the image processing system may identify symmetrical anatomical feature pairs in the 3D image volume and select a subvolume containing one of these anatomical features for further processing. This may reduce computational complexity and improve the efficiency of the image alignment process. In some embodiments, the subvolume of interest selected at operation 104 may be determined by identifying symmetrical anatomical feature pairs in the 3D image volume. A subvolume containing a first anatomical feature but not a second anatomical feature may be selected as the subvolume of interest. The 3D image volume may also be preprocessed at operation 104 to remove non-anatomical regions. For example, the image processing system may crop the 3D image volume to remove regions that do not contain relevant anatomical information. This preprocessing step may focus the image alignment process on the anatomical region of interest.

[0027] At operation 106, the image processing system detects a plurality of landmarks in the subvolume of interest. The landmarks may correspond to specific anatomical structures or regions within the subvolume of interest. In some embodiments, the plurality of landmarks may include at least three different landmarks, thereby providing sufficient points in the image coordinate system to determine the anatomical plane of interest relative to the image coordinate system. In some embodiments, the image processing system may employ a deep neural network trained to identify anatomical landmarks within the subvolume of interest. The deep neural network may be trained using various machine learning techniques and may be configured to identify anatomical landmarks, including points, 3D cylindrical structures, or specific anatomical structures. The landmarks detected by the deep neural network may be based on predefined anatomical landmarks associated with a desired viewing plane. In some embodiments, the image processing system may filter or sort the landmarks based on the confidence scores of the detected landmarks. The confidence of the detected landmarks may be determined based on the features of the detected landmarks, such as their shape and appearance. In one embodiment, a feature distribution for each landmark type may be created, and the confidence of the landmark detected in the subvolume may be determined by calculating the probability that the landmark, as encoded by its features, belongs to the feature distribution of its corresponding landmark type. In one embodiment, the image processing system may select a subset of landmarks from a plurality of landmarks detected in the subvolume of interest. The subset of landmarks may include at least three landmarks that may be used to determine an anatomical plane of interest relative to an image coordinate system.

[0028] At operation 108, the image processing system estimates transformation parameters based on the plurality of landmarks to adjust the rotation angle and translation of the subvolume of interest. The transformation parameters may be used to align the subvolume of interest with a desired orientation or position. In some embodiments, the image processing system estimates the transformation parameters based on the plurality of landmarks detected in the subvolume of interest. These transformation parameters may include rotation angles and translation vectors for adjusting the orientation and position of the subvolume of interest. In one embodiment, the image processing system may select a subset of landmarks from the plurality of landmarks detected in the subvolume of interest. The subset of landmarks may include at least three landmarks that may be used to determine an anatomical plane of interest relative to the image coordinate system. The selection of the subset of landmarks may be based on their confidence scores, wherein the landmark with the highest confidence is selected for further processing. Once the anatomical plane of interest is determined, the image processing system may estimate the rotation angle and translation vector required to align the subvolume of interest with the anatomical plane of interest. This may involve comparing the position of the selected landmark with one or more predetermined geometric constraints. The geometric constraints may be based on known or expected positions or relative positions of landmarks in the subvolume of interest that are correctly aligned. By comparing the actual positions of the landmarks with these geometric constraints, the image processing system can estimate the rotation angle and translation vector required to align the subvolume of interest with the anatomical plane of interest. The image processing system can use a variety of techniques to estimate the transformation parameters. For example, the image processing system can use optimization algorithms, machine learning techniques, or other suitable methods to estimate the rotation angle and translation vector.

[0029] At operation 110, the image processing system uses the transformation parameters to adjust the rotation angle and translation of the subvolume of interest to produce an aligned subvolume of interest. The transformation parameters including the rotation angle and the translation vector are applied to the subvolume of interest to generate a first aligned subvolume. In some embodiments, adjusting the rotation angle and translation of the subvolume of interest includes performing a multiplanar reorganization of the subvolume of interest along an anatomical plane of interest, as determined in operation 108, and then rotating the plane of interest until the position of the landmark satisfies one or more geometric constraints relative to the image coordinate system. In one embodiment, the transformation parameters are used to rotate and translate the anatomical features captured in the subvolume of interest relative to the image coordinate system so that a predefined anatomical plane is shown in a standard orientation relative to the image coordinate system. The predefined anatomical plane can be determined based on a subset of the landmarks detected in the subvolume of interest. In another embodiment, the transformation parameters are used to adjust the subvolume of interest so that the subvolume of interest is aligned with a desired orientation. The desired orientation can be predefined based on the type of anatomical structure being imaged or a specific diagnostic or therapeutic procedure being performed.

[0030] At operation 112, the image processing system determines the confidence of the transformation parameters based on the aligned subvolume of interest. The confidence may indicate the accuracy or reliability of the alignment of the aligned subvolume. In one embodiment, the determination of the confidence of the transformation parameters involves using the deep neural network used at operation 106 to detect multiple landmarks in the aligned subvolume. Then, the positions of these detected landmarks in the aligned subvolume are compared with one or more predetermined geometric constraints. The degree to which the landmarks satisfy these geometric constraints provides an indication of the accuracy of the alignment process. For example, if the landmarks in the aligned subvolume satisfy the geometric constraints (within a small deviation range), the alignment process may have been successful and the transformation parameters are therefore reliable. In another embodiment, the determination of the confidence of the transformation parameters involves projecting the aligned subvolume onto a projection plane to produce a projection image. The projection plane is parallel to a plurality of 2D images including the aligned subvolume. Then, the projection image is mapped to a confidence score using a confidence estimation network. The confidence score provides a quantitative measure of the accuracy of the alignment process. A high confidence score will indicate that the alignment process has been successful and the transformation parameters are reliable.

[0031] At operation 114, the image processing system checks whether the confidence of the transformation parameters is above a threshold. If the confidence is not above the threshold, the method 100 continues to iteratively refine the transformation parameters by returning to 106, where operations 106 to 114 may be repeated as described above, but using a first aligned subvolume opposite to the initially generated subvolume. The iterative refinement process involves: detecting landmarks in the first aligned subvolume; estimating the refined transformation parameters based on these landmarks; adjusting the rotation angle and translation of the first aligned subvolume using the refined transformation parameters to produce a second aligned subvolume; and determining the confidence of the refined transformation parameters based on the second aligned subvolume. In some embodiments, the iterative refinement of the transformation parameters may be repeated until the confidence of the resulting aligned volume exceeds a confidence threshold, or a predetermined number of iterations is exceeded, where the image processing system may indicate that the automatic alignment was unsuccessful and may prompt the user to perform manual image alignment. However, if the image processing system determines at operation 114 that the confidence is above the confidence threshold, the method 100 proceeds to operation 116.

[0032] At operation 116, the image processing system displays the aligned subvolumes via a display device. The aligned subvolumes may be visualized or presented to the user for further analysis or evaluation. In some embodiments, the image processing system may display the anatomical plane of interest with the rotation and translation determined at operation 108 applied. In some embodiments, the image processing system may display the confidence score determined at operation 112 next to the aligned subvolumes. This may provide the user with an indication of the reliability of the alignment process. If the confidence score is below a certain level, the image processing system may also provide the user with the option of manually adjusting the alignment. This may ensure that the user has the final say in the alignment process. After operation 116, method 100 may end.

[0033] refer to Figure 2 , a method 200 for preprocessing a 3D image volume to generate a sub-volume of interest is shown. The method 200 may be used to select a subset of 2D images from a 3D image volume that includes a desired field of view, and to partition symmetrical anatomical features by subdividing the subset of images along a symmetry plane.

[0034] At operation 202, the image processing system checks whether the 3D image volume includes a desired field of view (FOV). The image processing system may determine whether the desired FOV is included in the 3D image volume by analyzing the spatial extent of the 3D image volume and comparing it to a predetermined spatial extent of the desired FOV. The desired FOV may be predetermined based on the type of anatomical structure being imaged or the specific diagnostic or therapeutic procedure being performed. For example, if the 3D image volume is intended for cardiac imaging, the desired FOV may include the entire heart and the surrounding vasculature. If the 3D image volume is intended for brain imaging, the desired FOV may include the entire brain and surrounding structures. In some embodiments, the image processing system may employ a machine learning model to automatically determine whether an anatomical region of interest falls within the acquired 3D image volume. In some embodiments, the machine learning model may be trained to classify the 3D image volume according to the anatomical features captured therein, and / or output a probability score indicating the probability that the 3D image volume includes an anatomical feature of interest or a region of interest. If the 3D image volume does not include the desired FOV, the method 200 may proceed to operation 204. In some embodiments, the image processing system may also check the quality of the 3D image volume, such as image resolution, signal-to-noise ratio, or other image quality metrics, at operation 202 to ensure that the 3D image volume is suitable for a subsequent image alignment process. If the image quality is below a predetermined threshold, the image processing system may proceed to operation 204.

[0035] If the 3D image volume does not include the desired field of view, or the quality of the 3D image volume is below a predetermined threshold, the method 200 proceeds to operation 204, where the image processing system displays a request to rescan the imaged subject via a display device. In some embodiments, the image processing system may provide feedback to a user or operator at operation 204, indicating that the 3D image volume does not include the desired FOV and suggesting appropriate corrective actions, such as adjusting imaging parameters or repositioning the imaged subject. The request prompts the user to rescan the subject in order to obtain a 3D image volume that includes the desired field of view. After operation 204, the method 200 may end.

[0036] However, if at operation 202, the image processing system determines that the 3D image volume includes the desired field of view, the method 200 proceeds to operation 206. At operation 206, the image processing system selects a subset of the plurality of 2D images that include the 3D image volume, the 3D image volume including the subvolume of interest. The subset of images is selected based on the desired field of view. The selection process may involve identifying a 2D image that contains an anatomical feature of interest. As an example, if the desired FOV includes the heart, the image processing system may select a 2D image that contains the heart and the surrounding vascular system. In some embodiments, the image processing system may employ a machine learning model that is trained to identify anatomical features of interest in the 2D image. The machine learning model may be trained using a variety of machine learning techniques and may be configured to identify anatomical features, including points, 3D cylindrical structures, or specific anatomical structures. In some embodiments, the image processing system may also consider the quality of the 2D image when selecting an image subset. For example, the image processing system may select a 2D image with a high signal-to-noise ratio, high resolution, or other desired image quality metrics. This may ensure that the selected image subset is suitable for a subsequent image alignment process.

[0037] At operation 208, the image processing system may optionally divide the symmetrical anatomical feature by subdividing multiple 2D images along a symmetry plane. This division allows the symmetrical anatomical feature to be separated into separate sub-volumes. As an example, if the 3D image volume includes both the left knee and the right knee of the imaged subject, the image processing system may separate the left knee and the right knee for separate processing. This may involve identifying a symmetry plane in the 3D image volume and dividing the 3D image volume along the symmetry plane. This may produce two sub-volumes, each of which contains one of the symmetrical anatomical features. Each of these sub-volumes can then be processed separately during the image alignment process. This can improve the efficiency of the image alignment process and can also improve the accuracy of the alignment by reducing the complexity of the anatomical features to be aligned. In some embodiments, only the first of the two symmetrical anatomical features will be selected for subsequent imaging alignment.

[0038] At operation 210, the image processing system crops a subset of the plurality of 2D images to remove non-anatomical regions. In some embodiments, the image processing system may identify the boundaries of the anatomical features of interest in each of the 2D images and remove the portions of the 2D images that are outside of the boundaries. This may be achieved by applying a segmentation algorithm to each of the 2D images to segment the anatomical features of interest from the background or non-anatomical regions. The segmentation algorithm may be a conventional image processing algorithm, such as thresholding, edge detection, region growing, or watershed segmentation, or the segmentation algorithm may also be a machine learning-based algorithm, such as a convolutional neural network trained to segment the anatomical features of interest. In some embodiments, the image processing system may employ a deep learning model trained to identify and segment the anatomical features of interest in the 2D images. In some embodiments, the deep neural network may output a binary mask for each 2D image, wherein pixels corresponding to the anatomical features of interest are assigned a value of one, and all other pixels are assigned a value of zero. The image processing system may then apply these binary masks to the corresponding 2D images to remove non-anatomical regions. After operation 210, method 200 may end.

[0039] In this way, the method 200 can effectively pre-process the 3D image volume to generate a sub-volume of interest. This pre-processing can increase the success rate of downstream image alignment because it checks whether the 3D image volume includes the desired field of view. In addition, by selecting a subset of 2D images that includes the desired field of view, the image processing system can focus on relevant anatomical features and ignore irrelevant areas, thereby reducing computational complexity and improving the efficiency of the image alignment process.

[0040] refer to Figure 3 , a method 300 for detecting landmarks in a subvolume of interest is shown. The method 300 can be employed by an image processing system to: map a subvolume of interest to a plurality of landmarks using a trained deep neural network; derive one or more landmarks based on the positions of the plurality of landmarks; determine a confidence level for each of the detected landmarks; and filter the plurality of landmarks based on the confidence level to obtain at least three landmarks in the subvolume.

[0041] Method 300 begins at operation 302, in which an image processing system maps a subvolume of interest to a plurality of landmarks using a trained deep neural network. The deep neural network may be trained using a variety of machine learning techniques and may be configured to identify anatomical landmarks, including points, 3D cylindrical structures, or specific anatomical structures. The landmarks detected by the deep neural network may be based on predefined anatomical landmarks associated with a desired viewing plane. The deep neural network may be trained on a large dataset of 3D image volumes, each of which is annotated with the location of anatomical landmarks of interest. The training process may involve adjusting the parameters of the deep neural network to minimize the difference between the predicted location of the landmarks in the training dataset and the actual location of the landmarks. The trained deep neural network may then be used to predict the location of the landmarks in new, unseen 3D image volumes. In some embodiments, the deep neural network may include multiple layers, each of which transforms the input data into a higher level of abstraction. For example, a first layer may detect edges in a 3D image volume, a second layer may detect basic shapes, and subsequent layers may detect more complex anatomical structures. The last layer of the deep neural network may output the predicted location of the landmarks in the 3D image volume. In some embodiments, the deep neural network can be a convolutional neural network (CNN). A CNN can include a convolutional layer that applies a set of filters to the input data, a pooling layer that reduces the dimensionality of the data, and a fully connected layer that outputs the predicted position of the landmark.

[0042] In some embodiments, the deep neural network may be a recurrent neural network (RNN) configured to process sequential data. When the 3D image volume is viewed as a sequence of 2D image slices, the RNN may be advantageously employed. The RNN may process each 2D image slice in sequence, maintain an internal state that captures information from the previous slice, and output a predicted position of a landmark in the 3D image volume. In some embodiments, the deep neural network may be a combination of a CNN and an RNN. The CNN portion of the network may process each 2D image slice independently to detect local features, and the RNN portion of the network may process a sequence of 2D image slices to capture spatial relationships between features across different slices. In some embodiments, transfer learning may be used to train a deep neural network, wherein a pre-trained network is fine-tuned on a specific task of landmark detection. The pre-trained network may have been trained on a large general data set and may have learned to extract features from image data. By fine-tuning the pre-trained network on the specific task of landmark detection, the image processing system may utilize the feature extraction capabilities of the pre-trained network while adapting it to the specific task of anatomical landmark detection. In some embodiments, a deep neural network can be trained using unsupervised learning, where the network learns to extract useful features from a 3D image volume without annotated training data. Unsupervised learning can involve training the network to reconstruct input data, generate new data similar to the input data, or cluster the input data into groups. The features learned by the network during unsupervised learning can then be used to detect landmarks in the 3D image volume.

[0043] At operation 304, the image processing system may optionally derive one or more landmarks based on the positions of the plurality of landmarks. The derivation process may include utilizing the positions of the one or more landmarks detected at operation 302 and combining them with known anatomical relationships to obtain position information of one or more anatomical landmarks that are not directly detected. For example, in one embodiment, the image processing system may derive a landmark corresponding to the center of the anatomical structure by averaging the positions of two or more landmarks detected at operation 302 that are known to be located on the boundary of the anatomical structure. In another embodiment, the image processing system may derive landmarks corresponding to anatomical features that are known to be located at a fixed distance and direction from the landmark detected at operation 302. For example, if a landmark corresponding to the apex of the heart is detected, a landmark corresponding to the base of the heart may be derived by moving a fixed distance away from the apex of the heart. In yet another embodiment, the image processing system may derive a landmark corresponding to the intersection of two or more anatomical structures by identifying the landmarks detected at operation 302 that are located on each anatomical structure in the anatomical structure and calculating their intersections. For example, if landmarks corresponding to the left lung boundary and the right lung boundary are detected, a landmark corresponding to the mediastinum (the space between the lungs) can be derived by calculating the intersection of the lung boundaries. In some embodiments, the image processing system can derive additional landmarks based on the locations of multiple landmarks detected at operation 302 and the known anatomical relationships between these landmarks. For example, the image processing system can derive a landmark corresponding to the center of gravity of the anatomical structure by calculating a weighted average of the locations of the landmarks detected at operation 302 and located within the anatomical structure, wherein the weight is proportional to the voxel intensity at the landmark location. In some embodiments, the image processing system can derive landmarks corresponding to anatomical features that are not directly visible in the 3D image volume. For example, the image processing system can derive a landmark corresponding to the center of a blood vessel by fitting a 3D cylindrical model to the location of a landmark detected at operation 302 and located on the blood vessel boundary. In some embodiments, the image processing system can derive landmarks corresponding to anatomical features that are only partially visible in the 3D image volume. For example, the image processing system may derive landmarks corresponding to the tips of the bones by extrapolating from the locations of the landmarks located on the visible portion of the bone detected at operation 302. These derived landmarks may provide additional reference points for aligning the 3D image volume and may increase the accuracy and robustness of the alignment process.

[0044] At operation 306, the image processing system determines the confidence of each of the detected landmarks. This confidence determination allows the system to evaluate the reliability and accuracy of each landmark. In one embodiment, the confidence determination at operation 306 may involve calculating the probability that each landmark, as encoded by its features, belongs to the feature distribution of its corresponding landmark type. The feature distribution for each landmark type may be created based on a training data set of known anatomical landmarks. The features of the detected landmarks may include, but are not limited to, shape, size, texture, and appearance characteristics. For example, a landmark as a point may have a feature distribution based on its spatial coordinates, while a landmark as an anatomical structure may have a feature distribution based on its shape and size characteristics. In another embodiment, the confidence determination at operation 306 may involve comparing the detected landmark with a set of predefined anatomical landmarks associated with the desired viewing plane. The comparison may involve calculating a similarity score between the detected landmark and the predefined anatomical landmark. The similarity score may be based on various factors, such as the spatial proximity of the detected landmarks to predefined anatomical landmarks, the similarity in shape and appearance between the detected landmarks and predefined anatomical landmarks, and the consistency of the detected landmarks with known anatomical relationships.

[0045] At operation 308, the image processing system filters the plurality of landmarks based on confidence to obtain at least three landmarks in the subvolume. By filtering the landmarks based on the probability that the landmark belongs to the corresponding feature distribution of its landmark type / category, the probability of pseudo landmarks used for downstream image alignment is reduced. In one embodiment, the filtering process at operation 308 may involve sorting the detected landmarks based on their confidence scores and selecting the top three landmarks with the highest confidence scores. In another embodiment, the filtering process at operation 308 may involve setting a confidence threshold and selecting landmarks with confidence scores exceeding the threshold. The confidence threshold may be determined based on the requirements of a particular application, the quality of the 3D image volume, or the performance of a deep neural network. In yet another embodiment, the filtering process at operation 308 may involve applying a machine learning model to predict the reliability of each landmark based on the features and confidence scores of each landmark. The machine learning model may be trained on a data set of known landmarks, where each landmark is labeled with its reliability. Features used by the machine learning model may include confidence scores for landmarks, features of landmarks encoded by the deep neural network, and other relevant features such as the quality of the 3D image volume or the performance of the deep neural network. The machine learning model may output a reliability score for each landmark, and the landmark with the highest reliability score may be selected for further processing. In some embodiments, the filtering process at operation 308 may also involve clustering the detected landmarks based on their spatial proximity to each other, and selecting a landmark from each cluster based on its confidence score. This can help ensure that the selected landmarks are distributed across different anatomical regions, thereby improving the robustness and accuracy of the subsequent image alignment process.

[0046] In some embodiments, the filtering process at operation 308 may involve a combination of the above embodiments. For example, the image processing system may first sort the detected landmarks based on their confidence scores, then apply a confidence threshold to select a subset of the sorted landmarks, and finally apply a machine learning model to predict the reliability of the landmark subset and select the most reliable landmark for further processing. After operation 308, method 300 ends.

[0047] In this way, method 300 enables an image processing system to accurately and efficiently detect and select anatomical landmarks within a subvolume of interest. This is achieved by using a trained deep neural network that is capable of identifying a variety of anatomical landmarks based on predefined anatomical landmarks associated with a desired viewing plane. The method also enhances the accuracy of landmark detection by deriving additional landmarks based on the locations of the detected landmarks and known anatomical relationships. This provides a more comprehensive set of reference points for the subsequent image alignment process. In addition, method 300 is combined with a confidence determination step that evaluates the reliability of each detected landmark based on its features and the probability that it belongs to the feature distribution of its corresponding landmark type. This ensures that only potential pseudo-landmarks are not selected for further processing, thereby reducing the possibility of erroneous landmarks affecting the image alignment process.

[0048] refer to Figure 4 , a method 400 for estimating a rotation angle and a translation vector of an anatomical plane of interest is shown. The method 400 can be employed by an image processing system to: determine an anatomical plane of interest relative to an image coordinate system based on a plurality of landmarks, wherein the anatomical plane of interest intersects a subvolume; and determine a rotation angle and a translation vector of the anatomical plane of interest.

[0049] At operation 402, the image processing system determines an anatomical plane of interest relative to the image coordinate system based on a plurality of landmarks. This determination allows the system to identify a specific plane within the subvolume of interest for alignment. In one embodiment, the determination of the anatomical plane of interest involves utilizing a subset of landmarks from the plurality of landmarks determined at operation 308 of method 300. The subset of landmarks may include at least three landmarks that provide sufficient points in the image coordinate system to determine the anatomical plane of interest relative to the image coordinate system. The selection of the subset of landmarks may be based on their confidence scores, wherein the landmark with the highest confidence is selected for further processing. Once the subset of landmarks has been selected, the image processing system may determine the anatomical plane of interest by fitting the plane to the position of the selected landmark in the image coordinate system. This may involve using a least squares fitting method or other suitable plane fitting technique. Thus, the anatomical plane of interest is determined so that it best fits the position of the selected landmark in the image coordinate system.

[0050] In some embodiments, the three or more landmarks do not define an anatomical plane of interest, but define a plane having a fixed relationship with the anatomical plane of interest, and in such embodiments, after fitting the plane to the three or more landmarks, the anatomical plane of interest is derived from the fitting plane using the fixed relationship between the fitting plane and the anatomical plane of interest. As an example, the three or more landmarks may define a reference plane that is parallel to but not coincident with the anatomical plane of interest. In such cases, the anatomical plane of interest may be derived from the fitting plane by applying a translation along the normal of the fitting plane. The magnitude and direction of the translation may be determined based on the known or expected distance between the landmark and the anatomical plane of interest. For example, if the landmark is located on the surface of the organ and the anatomical plane of interest is located at the center of the organ, the translation may be equal to half the thickness of the organ and point to the interior of the organ. In some embodiments, the three or more landmarks may define a reference plane orthogonal to the anatomical plane of interest. In such cases, the anatomical plane of interest may be derived from the fitting plane by rotating the fitting plane around an axis parallel to the intersection of the fitting plane and the anatomical plane of interest. The rotation angle may be determined based on the known or expected angle between the landmark and the anatomical plane of interest. For example, if the landmark is located on the surface of an organ and the anatomical plane of interest is a cross-sectional plane of the organ, the rotation angle may be 90 degrees.

[0051] In other embodiments, three or more landmarks may define a reference plane that is tilted relative to the anatomical plane of interest. In such cases, the anatomical plane of interest may be derived from the fitted plane by applying a combination of rotations and translations. The rotation may align the normal of the fitted plane with the normal of the anatomical plane of interest, and the translation may shift the fitted plane to coincide with the anatomical plane of interest. The rotation axis, rotation angle, translation direction, and translation magnitude may be determined based on a known or expected relationship between the landmarks and the anatomical plane of interest.

[0052] At operation 404, the image processing system determines the rotation angle and translation vector of the anatomical plane of interest. This determination allows the image processing system to adjust the orientation and position of the subvolume to align it with the desired anatomical plane. In one embodiment, the determination of the rotation angle and translation vector involves comparing the position of the selected landmark with one or more predetermined geometric constraints. These geometric constraints may be based on the known or expected position or relative position of the landmark in the subvolume of interest that is correctly aligned with the image coordinate system. By comparing the actual position of the landmark with these geometric constraints, the image processing system is able to estimate the rotation angle and translation vector required to align the subvolume of interest with the anatomical plane of interest. In some embodiments, the geometric constraints may include the distance between the landmark pairs relative to one or more axes of the image coordinate system. As an example, the geometric constraints that can be used to determine the rotation angle may specify that in the correctly aligned subvolume, the difference between the first y coordinate of the first landmark and the second y coordinate of the second landmark is zero, or within a threshold of zero. In another example, the geometric constraints that can be used to determine the translation vector may specify that the position of the landmark is within a threshold distance of the midpoint of the range of the axis of the image coordinate system. In another embodiment, the geometric constraints may include the angle between the lines connecting the landmark pairs relative to one or more axes of the image coordinate system. For example, a geometric constraint may specify that in a properly aligned subvolume, the angle between a line connecting a first landmark and a second landmark and a point on the x-axis of the image coordinate system that is equal to the x-coordinate of the second landmark is 90 degrees, or is within a threshold of 90 degrees. In yet another embodiment, a geometric constraint may include the relative positions of the landmarks with respect to each other. For example, a geometric constraint may specify that in a properly aligned subvolume, a first landmark is located above a second landmark, or a third landmark is located to the right of a fourth landmark. This may be advantageous when aligning anatomical structures that have a known spatial relationship to each other in a properly aligned subvolume.

[0053] In some embodiments, the image processing system can directly calculate the rotation angle and translation vector using an analytical solution to a set of equations, which includes the current position of the landmark or the projection of the landmark on one or more axes of the image coordinate system and the geometric constraints. In some embodiments, the image processing system can estimate the rotation angle and translation vector using an optimization algorithm. The optimization algorithm can minimize a cost function that measures the deviation of the actual position of the landmark from their expected position based on the geometric constraints. The cost function can include items for each of these geometric constraints and can be weighted to give certain constraints more importance than other constraints. The optimization algorithm can include gradient descent, Newton's method or other suitable optimization techniques. After operation 404, method 400 can end.

[0054] refer to Figure 5, a method 500 for determining confidence in alignment of a subvolume of interest is shown. The method 500 may be employed by an image processing system to: detect multiple landmarks in an aligned subvolume of interest using a deep neural network; compare the positions of the multiple landmarks in the aligned subvolume of interest with one or more predetermined geometric constraints to determine a degree of geometric constraint satisfaction; determine a geometric constraint confidence based on the degree of geometric constraint satisfaction; project the aligned subvolume of interest onto a projection plane to produce a projection image; map the projection image to a projection confidence using a confidence estimation network; and determine an overall confidence of the aligned subvolume by combining the geometric constraint confidence with the projection confidence.

[0055] At operation 502, the image processing system detects a plurality of landmarks in the aligned subvolume of interest using a deep neural network. The detection allows the system to identify the locations of the landmarks within the aligned subvolume. In one embodiment, the deep neural network used to detect the landmarks in the aligned subvolume of interest may be the same or similar to the deep neural network used in the initial detection of the landmarks in the subvolume of interest, such as at operation 302. The deep neural network may be trained using various machine learning techniques and may be configured to identify anatomical landmarks, including points, 3D cylindrical structures, or specific anatomical structures. The landmarks detected by the deep neural network may be based on predefined anatomical landmarks associated with a desired viewing plane.

[0056] After landmark detection, at operation 504, the image processing system compares the positions of multiple landmarks in the aligned subvolume of interest with one or more predetermined geometric constraints to determine the degree of geometric constraint satisfaction. This comparison allows the system to evaluate the degree to which the aligned subvolumes satisfy the desired geometric constraints. In one embodiment, the geometric constraints may be based on the known or expected positions, or relative positions, of the landmarks in the correctly aligned subvolume of interest. For example, if the anatomical region of interest is the heart, the geometric constraints may specify that certain landmarks (e.g., the apex of the heart and the base of the heart) should be located at certain positions relative to each other and relative to the image coordinate system in the correctly aligned subvolume. By comparing the actual positions of the landmarks in the aligned subvolumes with these geometric constraints, the image processing system can determine the degree to which the geometric constraints are satisfied.

[0057] In some embodiments, the comparison of the locations of the landmarks to the geometric constraints may involve calculating the distance between each landmark and its corresponding location specified by the geometric constraints. These distances may then be aggregated, for example by summing or averaging, to produce a measure of the degree to which the geometric constraints are satisfied. In other embodiments, the comparison may involve calculating a measure of the deviation of the locations of the landmarks from the geometric constraints. Such a deviation measure may be based on a statistical measure, such as a standard deviation or variance, or on a geometric measure, such as the volume of a minimum polyhedron containing all the landmarks and their corresponding locations specified by the geometric constraints.

[0058] At operation 506, the image processing system determines the geometric constraint confidence based on the geometric constraint satisfaction. This determination allows the system to quantify the confidence level of the alignment based on the satisfaction of the geometric constraints. In one embodiment, the geometric constraint confidence can be a function of the geometric constraint satisfaction, wherein a higher satisfaction corresponds to a higher confidence. The function can be a linear function, a nonlinear function, or a step function, etc. In some embodiments, the geometric constraint confidence can be normalized to a range between a minimum confidence value and a maximum confidence value. For example, the geometric constraint confidence can be normalized to a range between 0 and 1, wherein 0 represents no confidence and 1 represents full confidence. A variety of techniques can be used to perform normalization, such as minimum-maximum normalization, z-score normalization, or decimal scaling normalization, etc. In some embodiments, the geometric constraint confidence can be adjusted based on additional factors such as the quality of the 3D image volume, the complexity of the anatomical region of interest, or the performance of the deep neural network when detecting landmarks. For example, if the 3D image volume is of low quality (e.g., due to motion artifacts or low signal-to-noise ratio), the geometric constraint confidence may be reduced to reflect the increased uncertainty in the alignment. Similarly, if the anatomical region of interest is complex (e.g., due to the presence of multiple overlapping structures or due to anatomical differences between different individuals), the geometric constraint confidence may be reduced to reflect the increased difficulty in the alignment.

[0059] After determining the geometric constraint confidence, at operation 508, the image processing system projects the aligned subvolume of interest onto a projection plane to generate a projection image. The projection process involves transforming the 3D aligned subvolume into a 2D projection image. The projection plane is parallel to a plurality of 2D images including the aligned subvolume of interest. The projection process can be implemented using various projection techniques known in the art of medical imaging, such as maximum intensity projection (MIP), minimum intensity projection (MinIP), or average intensity projection (AIP). Compared to methods utilizing unprojected data of the aligned subvolume, the projection image provides a comprehensive view of the aligned subvolume, which is computationally less expensive and can be used to further assess the quality of the alignment.

[0060] At operation 510, the image processing system maps the projection image to a projection confidence using a confidence estimation network. This mapping allows the system to determine the confidence of the alignment based on the quality of the projection image. The confidence estimation network can be a machine learning or deep learning network that is trained to estimate the confidence of the alignment based on the projection image. The network can be trained using various machine learning techniques and can be configured to map the projection image to a confidence score. The confidence score provides a quantitative measure of the accuracy of the alignment process. A high confidence score will indicate that the alignment process has been successful and the transformation parameters are reliable.

[0061] At operation 512, the image processing system determines an overall confidence of the aligned subvolumes by combining the geometric constraint confidence with the projection confidence. The overall confidence provides a comprehensive measure of the accuracy of the alignment process, taking into account both the satisfaction of the geometric constraints and the projection confidence. Various combination methods (e.g., weighted average, maximum, minimum, or other suitable combination methods) may be used to determine the overall confidence. The overall confidence may be used to determine whether the alignment process was successful or whether further refinement of the transformation parameters is required. If the overall confidence is above a predetermined threshold, the alignment process may be considered successful. Otherwise, the alignment process may be iteratively refined until the overall confidence exceeds a threshold or a predetermined number of iterations is reached. After operation 512, method 500 may end.

[0062] In this way, method 500 enables an image processing system to determine the confidence of the alignment of a subvolume of interest in a comprehensive and robust manner. By combining the geometric constraint confidence with the projection confidence, the system can consider both the satisfaction of the geometric constraints and the quality of the projected image when evaluating the accuracy of the alignment process. This dual approach to confidence estimation can provide a more reliable alignment accuracy metric than either approach alone, thereby potentially improving diagnostic accuracy and efficiency in medical imaging applications. In this way, method 500 can facilitate automatic image alignment in medical imaging applications, thereby potentially improving diagnostic accuracy and efficiency, reducing the need for manual image alignment, and enhancing patient care.

[0063] refer to Figure 6 , a method 600 for aligning a subvolume of interest with an anatomical plane of interest is shown. The method 600 may be employed by an image processing system to adjust the orientation and position of a subvolume to align the subvolume with a desired anatomical plane.

[0064] At operation 602, the image processing system reorganizes the subvolume of interest along the anatomical plane of interest to produce a reorganized subvolume of interest. This operation involves re-slicing the subvolume of interest into a sequence of 2D image slices parallel to the anatomical plane of interest. In one embodiment, the reorganization process at operation 602 may involve the use of a multi-planar reformation (MPR) technique. MPR is a post-processing technique that allows images to be visualized in various planes. In this case, MPR can be used to re-slice the subvolume of interest into a sequence of 2D image slices parallel to the anatomical plane of interest. This may be particularly useful when the original imaging plane is not aligned with the anatomical plane of interest. As an example, the subvolume of interest may be a brain region captured in the sagittal plane, but the anatomical plane of interest may be the coronal plane. Using MPR, the image processing system may reorganize the sagittal image into a sequence of 2D image slices parallel to the coronal plane. This allows a more accurate and detailed analysis of the region of interest.

[0065] At operation 604, the image processing system applies the rotation angle to the reconstructed subvolume of interest. This operation involves rotating the reconstructed subvolume around the appropriate axis to achieve the desired rotation angle. In some embodiments, the image processing system may apply a rotation matrix to the subvolume of interest at operation 604. The rotation matrix may be applied to each voxel in the subvolume of interest to rotate the subvolume so that anatomical landmarks within the subvolume of interest are rotated to positions estimated to satisfy one or more geometric constraints. The rotation may involve adjusting the orientation of the subvolume around one or more rotation axes.

[0066] At operation 606, the image processing system applies the translation vector to the reorganized subvolume of interest to produce an aligned subvolume of interest. This operation involves moving the reorganized subvolume along the appropriate axis to achieve the desired translation. The translation vector may include a vector in three-dimensional space, and thus may include a vector having three values ​​indicating the magnitude of the translation to be applied to each voxel in the subvolume of interest along each of the three basis vectors / base axes of the image coordinate system. After operation 606, method 600 may end.

[0067] Method 600 enables an image processing system to align a subvolume of interest with an anatomical plane of interest, thereby allowing accurate analysis and visualization of anatomical structures within the subvolume. Method 600 is intended to provide an exemplary method for applying transformation parameters to align a subvolume of interest, however it should be understood that the present disclosure encompasses alternative methods for defining an alignment transformation and applying the transformation. As an example, in some embodiments, an image processing system may apply an affine transformation to a subvolume of interest to produce an aligned subvolume of interest. An affine transformation may include a combination of rotation, translation, scaling, and shearing operations. The parameters of the affine transformation may be the parameters of the affine transformation in operation 108 ( Figure 1 ) at the estimated transformation parameters. An affine transformation may be applied to each voxel in the subvolume of interest to adjust the orientation, position, size, and shape of the subvolume so that the subvolume of interest is aligned with the anatomical plane of interest. In some embodiments, the image processing system may apply a nonlinear transformation to the subvolume of interest. Nonlinear transformations may be used to correct distortions in the subvolume of interest that may be difficult to correct using linear transformations such as rotation, translation, scaling, and shearing. The image processing system may apply a nonlinear transformation to the subvolume of interest based on the transformation parameters estimated at operation 108 ( Figure 1 ) and additional parameters estimated based on characteristics of the subvolume of interest. The nonlinear transformation may be applied to each voxel in the subvolume of interest to adjust the orientation, position, size, and shape of the subvolume so that the subvolume of interest is aligned with the anatomical plane of interest.

[0068] Reference now Figure 7, a magnetic resonance imaging (MRI) apparatus 710 is shown, wherein the MRI apparatus 710 includes a static magnetic field magnet unit 712, a gradient coil unit 713, an RF coil unit 714, an RF body or volume coil unit 715, a transmit / receive (T / R) switch 720, an RF driver unit 722, a gradient coil driver unit 723, a data acquisition unit 724, a controller unit 725, a patient examination table or examination couch 726, a data processing unit 731, an operation console unit 732, and a display unit 733. In some embodiments, the RF coil unit 714 is a surface coil, which is a local coil that is generally placed near an anatomical structure of interest of a subject 716. Here, the RF body coil unit 715 is a transmission coil that transmits RF signals, and the local surface RF coil unit 714 receives MR signals. Therefore, the transmission body coil (e.g., the RF body coil unit 715) and the surface receiving coil (e.g., the RF coil unit 714) are independent but electromagnetically coupled components. The MRI apparatus 710 transmits electromagnetic pulse signals to a subject 716 placed in an imaging space 718 forming a static magnetic field to perform scanning to obtain magnetic resonance signals from the subject 716. One or more images of the subject 716 may be reconstructed based on the magnetic resonance signals thus obtained by the scanning.

[0069] The static magnetic field magnet unit 712 includes, for example, an annular superconducting magnet installed in an annular vacuum container. The magnet defines a cylindrical space around the subject 716 and generates a constant main static magnetic field B0.

[0070] The MRI apparatus 710 further includes a gradient coil unit 713, which forms a gradient magnetic field in an imaging space 718 so as to provide three-dimensional position information for a magnetic resonance signal received by the RF coil array. The gradient coil unit 713 includes three gradient coil systems, each of which generates a gradient magnetic field along one of three spatial axes perpendicular to each other, and generates a gradient field in each of a frequency encoding direction, a phase encoding direction, and a slice selection direction according to an imaging condition. More specifically, the gradient coil unit 713 applies a gradient field in a slice selection direction (or a scanning direction) of the subject 716 to select a slice; and the RF body coil unit 715 or the local RF coil array can transmit an RF pulse to the selected slice of the subject 716. The gradient coil unit 713 also applies a gradient field in a phase encoding direction of the subject 716 to phase encode a magnetic resonance signal from a slice excited by the RF pulse. The gradient coil unit 713 then applies a gradient field in a frequency encoding direction of the subject 716 to frequency encode the magnetic resonance signal from the slice excited by the RF pulse.

[0071] The RF coil unit 714 is provided, for example, to surround a region to be imaged of a subject 716. In some examples, the RF coil unit 714 may be referred to as a surface coil or a receiving coil. In a static magnetic field space or an imaging space 718 in which a static magnetic field B0 is formed by a static magnetic field magnet unit 712, the RF coil unit 715 transmits RF pulses as electromagnetic waves to the subject 716 based on a control signal from a controller unit 725, and thereby generates a high-frequency magnetic field B1. This excites the proton spins in the slice to be imaged of the subject 716. The RF coil unit 714 receives the electromagnetic waves generated when the proton spins thus excited in the slice to be imaged of the subject 716 return to alignment with the initial magnetization vector as magnetic resonance signals. In some embodiments, the RF coil unit 714 may transmit RF pulses and receive MR signals. In other embodiments, the RF coil unit 714 may be used only to receive MR signals, and not to transmit RF pulses.

[0072] The RF body coil unit 715 is disposed, for example, to surround an imaging space 718, and generates RF magnetic field pulses orthogonal to the main magnetic field B0 generated by the static magnetic field magnet unit 712 within the imaging space 718 to excite nuclei. Compared to the RF coil unit 714, which can be disconnected from the MRI device 710 and replaced with another RF coil unit, the RF body coil unit 715 is fixedly attached and connected to the MRI device 710. In addition, although a local coil (such as the RF coil unit 714) can transmit or receive signals only from a local area of ​​the subject 716, the RF body coil unit 715 generally has a larger coverage area. For example, the RF body coil unit 715 can be used to transmit or receive signals to the whole body of the subject 716. Using a receive-only local coil and a transmission body coil provides uniform RF excitation and good image uniformity at the expense of higher RF power deposited in the subject. For a transmit-receive local coil, the local coil provides RF excitation to the region of interest and receives MR signals, thereby reducing the RF power deposited in the subject. It will be appreciated that the specific use of the RF coil unit 714 and / or the RF body coil unit 715 depends on the imaging application.

[0073] The T / R switch 720 may selectively electrically connect the RF body coil unit 715 to the data acquisition unit 724 when operating in the receive mode, and may selectively electrically connect the RF body coil unit to the RF driver unit 722 when operating in the transmit mode. Similarly, the T / R switch 720 may selectively electrically connect the RF coil unit 714 to the data acquisition unit 724 when the RF coil unit 714 operates in the receive mode, and may selectively electrically connect the RF coil unit to the RF driver unit 722 when the RF coil unit operates in the transmit mode. When both the RF coil unit 714 and the RF body coil unit 715 are used for a single scan, for example, if the RF coil unit 714 is configured to receive MR signals and the RF body coil unit 715 is configured to transmit RF signals, the T / R switch 720 may direct control signals from the RF driver unit 722 to the RF body coil unit 715 while directing received MR signals from the RF coil unit 714 to the data acquisition unit 724. The coils of the RF body coil unit 715 may be configured to operate in a transmit-only mode or a transmit-receive mode. The coils of the local RF coil unit 714 may be configured to operate in a transmit-receive mode or a receive-only mode.

[0074] The RF driver unit 722 includes a gate modulator (not shown), an RF power amplifier (not shown), and an RF oscillator (not shown), which are used to drive an RF coil (e.g., the RF coil unit 715) and form a high-frequency magnetic field in the imaging space 718. The RF driver unit 722 modulates an RF signal received from the RF oscillator into a signal of a predetermined timing having a predetermined envelope based on a control signal from the controller unit 725 and using the gate modulator. The RF signal modulated by the gate modulator is amplified by the RF power amplifier and then output to the RF coil unit 715.

[0075] The gradient coil driver unit 723 drives the gradient coil unit 713 based on a control signal from the controller unit 725, thereby generating a gradient magnetic field in the imaging space 718. The gradient coil driver unit 723 includes three driver circuit systems (not shown) corresponding to the three gradient coil systems included in the gradient coil unit 713.

[0076] The data acquisition unit 724 includes a preamplifier (not shown), a phase detector (not shown), and an analog / digital converter (not shown) for acquiring a magnetic resonance signal received by the RF coil unit 714. In the data acquisition unit 724, the phase detector uses the output of the RF oscillator from the RF driver unit 722 as a reference signal, performs phase detection on the magnetic resonance signal received from the RF coil unit 714 and amplified by the preamplifier, and outputs the analog magnetic resonance signal after the phase detection to the analog / digital converter to be converted into a digital signal. The digital signal thus obtained is output to the data processing unit 731.

[0077] The MRI apparatus 710 includes a couch 726 for placing a subject 716 thereon. By moving the couch 726 based on a control signal from the controller unit 725, the subject 716 can be moved inside and outside the imaging space 718.

[0078] The controller unit 725 includes a computer and a recording medium on which a program to be executed by the computer is recorded. The program, when executed by the computer, causes the various parts of the device to perform operations corresponding to a predetermined scan. The recording medium may include, for example, a ROM, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, or a non-volatile memory card. The controller unit 725 is connected to the operation console unit 732 and processes an operation signal input to the operation console unit 732, and also controls the examination bed 726, the RF driver unit 722, the gradient coil driver unit 723, and the data acquisition unit 724 by outputting control signals to them. The controller unit 725 also controls the data processing unit 731 and the display unit 733 based on the operation signal received from the operation console unit 732 to obtain a desired image.

[0079] The operation console unit 732 includes a user input device such as a touch screen, a keyboard, and a mouse. The operator uses the operation console unit 732 to, for example, input such data as an imaging protocol and set an area where an imaging sequence will be executed. The data on the imaging protocol and the imaging sequence execution area are output to the controller unit 725.

[0080] The data processing unit 731 includes a computer and a recording medium on which a program to be executed by the computer to perform predetermined data processing is recorded. The data processing unit 731 is connected to the controller unit 725, and performs data processing based on a control signal received from the controller unit 725. The data processing unit 731 is also connected to the data acquisition unit 724, and generates spectrum data by applying various image processing operations to the magnetic resonance signals output from the data acquisition unit 724.

[0081] In one example, the recording medium of the data processing unit 731 may include instructions for automatic image alignment of acquired 3D medical image volumes.

[0082] The display unit 733 includes a display device, and displays an image on a display screen of the display device based on a control signal received from the controller unit 725. The display unit 733 displays, for example, images regarding input items for inputting operation data by an operator from the operation console unit 732. The display unit 733 also displays a two-dimensional (2D) slice image or a three-dimensional (3D) image of the subject 716 generated by the data processing unit 731.

[0083] Although an MRI system is described by way of example, it should be understood that the present invention's techniques for automatic alignment of 3D medical image volumes may also be useful when applied to images acquired using other imaging modalities, such as CT, tomosynthesis, PET, C-arm angiography, etc. The present invention's discussion of the MRI imaging modality is provided merely as an example of one suitable imaging modality.

[0084] refer to Figure 8 , an image processing system 802 incorporated into the imaging system 800 is shown. The imaging system 800 may be configured to acquire a 3D medical image of an imaged subject using an imaging device 826, and to process the acquired 3D medical image using the image processing system 802, for example to automatically align the 3D medical image with one or more standard views. The aligned image may be displayed to a user via a display device 824. The imaging system may receive user input via a user input device 822, wherein one or more of the image acquisition and image alignment may be adjusted based on the received user input. As an example, after the 3D image acquisition, the user may specify an anatomical region of interest that the user wishes to view one or more standard views. The image processing system 802 may receive the user input from the user input device 822, and automatically determine the transformation parameters to align the anatomical region of interest to produce the requested standard view, for example using a multi-planar reformation (MPR) module 812. The image processing system 802 may then automatically apply the transformation parameters to the 3D medical image to generate an aligned subvolume of interest, and may display the aligned subvolume of interest (or one or more 2D image slices thereof) to a user via a display device 824 .

[0085] In some embodiments, at least a portion of the image processing system 802 is disposed at a device (e.g., an edge device, a server, etc.) that is communicatively coupled to the MRI system 710 via a wired connection and / or a wireless connection. In some embodiments, at least a portion of the image processing system 802 is disposed at a separate device (e.g., a workstation) that can receive images from the MRI system 710 or from a storage device storing images / data generated by the MRI system 710. The image processing system 802 can be operatively / communicatively coupled to a user input device 822 and a display device 824. The user input device 822 can be integrated into the MRI system, such as at an operating console unit 732 of the MRI system 710. Similarly, the display device 824 can be integrated into the MRI system, such as at a display unit 733 of the MRI system 710.

[0086] The image processing system 802 includes a processor 804 configured to execute machine-readable instructions stored in a non-transitory memory 806. The processor 804 may be a single-core or multi-core processor, and the program executed thereon may be configured for parallel processing or distributed processing. In some embodiments, the processor 804 may optionally include separate components distributed on two or more devices, which may be located at a distance and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 804 may be virtualized and performed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0087] The non-transitory memory 806 may store a pre-processing module 808 , a landmark detection module 810 , and a multi-planar reformatting (MPR) module 812 .

[0088] The pre-processing module 808 may include instructions for identifying and isolating one or more anatomical regions of interest, as well as instructions for dividing bilaterally symmetrical anatomical regions of interest and performing other image processing operations. In one example, the pre-processing module 808 may include a machine learning (ML) model that is trained to map and classify 2D medical images that contain or belong to anatomical regions of interest. In some embodiments, the pre-processing module 808 may include instructions for selecting a sub-volume of interest from a 3D medical image volume. The selection may be based on identifying a subset of multiple 2D images that include the anatomical region of interest. Then, the selected subset of the 2D images may be cropped to remove non-anatomical regions, thereby focusing on the relevant anatomical features. In certain embodiments, the 3D image volume may include symmetrical anatomical features. In such cases, the pre-processing module 808 may be configured to identify symmetrical anatomical feature pairs and select a sub-volume containing one of the anatomical features for further processing. This is particularly useful in cases where the anatomical features of interest are bilaterally symmetrical (such as lungs or kidneys). The pre-processing module 808 may also include a laterality separator step to separate left and right anatomical structures for separate processing. This may be beneficial in situations where the anatomical structures on one side of the body are more interesting or more relevant to medical diagnosis or treatment planning than the anatomical structures on the other side. In addition, the pre-processing module 808 may verify whether the acquired 3D image volume is suitable for alignment before continuing with further pre-processing. The verification may involve checking the quality of the 3D image volume (such as image resolution and signal-to-noise ratio) to ensure that it meets the requirements of the subsequent image alignment process. In some embodiments, the pre-processing module 808 may include a machine learning model that is trained to classify 2D medical images as containing or belonging to an anatomical region of interest. The model can be used to help select a sub-volume of interest from a 3D medical image volume. Various machine learning techniques can be used to train the model.

[0089] The non-transitory memory 806 may also store a landmark detection module 810, which may include instructions for detecting a plurality of anatomical landmarks in the subvolume of interest using a deep neural network that is trained to identify anatomical landmarks. The landmark detection module 810 may include a feature extraction component that is configured to extract features from the subvolume of interest. These features may be used to identify potential anatomical landmarks within the subvolume of interest. In some embodiments, the landmark detection module 810 may include a landmark classification component that is configured to classify anatomical landmarks based on the extracted features. The landmark classification component may use a machine learning model (such as a deep neural network) that has been trained to classify anatomical landmarks based on the features of the anatomical landmarks. In certain embodiments, the landmark detection module 810 may include a landmark encoding component that is configured to encode the classified anatomical landmarks into a feature vector. The landmark encoding component may compare the feature vector to a distribution of previously encoded feature vectors of the same landmark type to determine a probability that the landmark belongs to the distribution of the previously encoded feature vector. The confidence of the landmark may be determined based on the probability. These features may include, but are not limited to, the location of the landmarks within the subvolume of interest, the strength of the landmarks, the shape of the landmarks, and the texture of the landmarks. The extracted features may then be encoded into a feature vector for each landmark. In another embodiment, the landmark detection module 810 includes a feature normalization component that normalizes the extracted features before the extracted features are encoded into feature vectors. The feature normalization component may use various normalization techniques (such as minimum-maximum normalization, z-score normalization, or decimal scaling normalization) to ensure that the features have similar scales. This can help prevent features with larger scales from dominating the encoding process. In yet another embodiment, the landmark detection module 810 may include a feature selection component that selects a subset of the extracted features for encoding. The feature selection component may use various feature selection techniques (such as mutual information, chi-square test, or recursive feature elimination) to select the most informative features for encoding.

[0090] In another embodiment, the landmark detection module 810 may include a feature encoding component that uses a machine learning model (such as an autoencoder) to encode the selected features into a feature vector. The autoencoder can be trained to minimize the reconstruction error between the input features and the reconstructed features, thereby generating a feature vector that captures the most important information about the landmark. In another embodiment, the landmark detection module 810 may include a feature comparison component that compares the encoded feature vector with the distribution of previously encoded feature vectors of the same landmark type. The feature comparison component can use various distance metrics (such as Euclidean distance, Manhattan distance, or cosine similarity) to determine the similarity between the distribution of the encoded feature vector and the previously encoded feature vector. The confidence of the landmark can be determined based on this similarity.

[0091] The non-transitory memory 806 also stores an MPR module 812, which may include instructions for executing one or more MPR algorithms known in the field of medical imaging, and may include resampling image data from an image sequence along a landmark plane or other plane of interest to synthesize a view of an anatomical structure of interest at a desired orientation. The MPR module 812 may also include instructions for estimating transformation parameters based on detected anatomical landmarks. These transformation parameters may include rotation angles and translation vectors for adjusting the orientation and position of the subvolume of interest. The MPR module 812 may be configured to apply these transformation parameters to the subvolume of interest to produce a first aligned subvolume of interest. In some embodiments, the MPR module 812 may include instructions for determining an anatomical plane of interest relative to an image coordinate system based on a subset of landmarks detected by the landmark detection module 810. The landmark subset may include at least three landmarks detected in the subvolume of interest. The MPR module 812 may then perform multi-planar reorganization of the subvolume of interest along the anatomical plane of interest based on one or more geometric constraints imposed on the landmark subset.

[0092] The non-transitory memory 806 also stores a geometric constraint module 814, which may include instructions for determining whether the position of one or more landmarks detected by the landmark detection module 810 satisfies one or more predetermined geometric constraints, wherein the one or more geometric constraints may be stored in the geometric constraint module 814, such as in a database. The geometric constraint module 814 may include instructions for determining the relative positions of the landmarks relative to each other. For example, one landmark may be required to be above another landmark, or one landmark may be required to be to the right of another landmark. The geometric constraints may also include the distance between the landmark pairs relative to one or more axes of the image coordinate system. For example, the difference between the y coordinates of the two landmarks may be constrained to zero or within a threshold. Additionally, the geometric constraint module 814 may include instructions for determining the angle between the lines connecting the landmark pairs relative to one or more axes of the image coordinate system. For example, the angle between the line connecting the two landmarks and the point on the x-axis may be constrained to 90 degrees or within a threshold of 90 degrees.

[0093] The geometric constraint module 814 may also include instructions for determining the degree of satisfaction of the landmark with respect to the geometric constraints. This provides an indication of the accuracy of the alignment process. A higher degree of satisfaction corresponds to a higher degree of alignment confidence. In addition, the geometric constraint module 814 may include instructions for determining the geometric constraint confidence based on the degree of satisfaction of the geometric constraints. The geometric constraint confidence may be a function of the degree of satisfaction, wherein a higher degree of satisfaction corresponds to a higher degree of confidence. The geometric constraint confidence may be normalized to a range between a minimum confidence value and a maximum confidence value. For example, the geometric constraint confidence may be normalized to a range between 0 and 1, wherein 0 represents no confidence and 1 represents full confidence. The geometric constraint module 814 may also include instructions for adjusting the geometric constraint confidence based on additional factors such as the quality of the 3D image volume, the complexity of the anatomical region of interest, or the performance of the deep neural network in detecting the landmark.

[0094] The non-transitory memory also stores a confidence estimation module 816, which may include instructions for determining the confidence of one or more landmarks detected by the landmark detection module 810, and instructions for determining the confidence of the transformation parameters based on the subvolumes aligned via the transformation parameters. The confidence estimation module 816 may include instructions for determining the confidence of the transformation parameters based on the satisfaction of the landmarks in the aligned subvolumes with the geometric constraints. The confidence determination process may involve comparing the positions of the landmarks in the aligned subvolumes with one or more predetermined geometric constraints stored in the geometric constraint module 814. The satisfaction of these geometric constraints provides an indication of the accuracy of the alignment process. The confidence based on the satisfaction of the geometric constraints quantifies the confidence level of the alignment based on the satisfaction of the geometric constraints. In addition, the confidence estimation module 816 may include instructions for projecting the aligned subvolumes onto a projection plane to produce a projection image. The projection image is then mapped to a projection confidence using a confidence estimation network. The projection confidence provides a quantitative measure of the accuracy of the alignment process based on the quality of the projection image. The confidence estimation module 816 may also include instructions for determining an overall confidence of the aligned subvolumes by combining the geometric constraint confidence with the projection confidence. The overall confidence provides a comprehensive measure of the accuracy of the alignment process, taking into account both the satisfaction of the geometric constraints and the quality of the projected images.

[0095] In some embodiments, non-transitory memory 806 may include components disposed on two or more devices that may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of non-transitory memory 806 may include a remotely accessible networked storage device configured in a cloud computing configuration.

[0096] The user input device 822 may include one or more of a touch screen, keyboard, mouse, trackpad, motion sensing camera, or other device configured to enable a user to interact with and manipulate data within the image processing system 802. In one example, the user input device 822 may enable a user to select an anatomical region of interest for which the user wishes to view one or more standard views.

[0097] The display device 824 may include one or more display devices utilizing virtually any type of technology. In some embodiments, the display device 824 may include a computer monitor and may display an aligned medical image volume, such as an automatically aligned image generated according to one or more of the methods disclosed herein. The display device 824 may be combined with the processor 804, the non-transitory memory 806, and / or the user input device 822 in a common housing, or may be a peripheral display device and may include a monitor, a touch screen, a projector, or other display device known in the art that may enable a user to view the MRI generated by the MRI system and / or may interact with various data stored in the non-transitory memory 806.

[0098] It should be understood that Figure 8 The imaging system 800 shown is for illustration and not limitation. Another suitable imaging system may include more, fewer, or different components.

[0099] The present disclosure also provides support for a method for automatic image alignment of a three-dimensional (3D) medical image volume, the method comprising: preprocessing the 3D medical image volume by selecting a subvolume of interest; detecting multiple landmarks in the subvolume of interest using a deep neural network trained to identify anatomical landmarks; estimating transformation parameters to adjust a rotation angle and a translation of the subvolume of interest based on the multiple anatomical landmarks; adjusting the rotation angle and the translation of the subvolume of interest using the transformation parameters to produce a first aligned subvolume of interest; determining a confidence of the transformation parameters based on the first aligned subvolume of interest; and iteratively refining the transformation parameters if the confidence is below a predetermined threshold. In a first example of the method, the 3D medical image volume is an image sequence including multiple two-dimensional (2D) images, and wherein preprocessing the 3D medical image volume by selecting the subvolume of interest comprises: selecting a subset of the multiple 2D images including the subvolume of interest; and cropping the subset of the multiple 2D images to remove areas outside the subvolume of interest. In a second example of the method (optionally including the first example), selecting the subvolume of interest includes: identifying a bilaterally symmetrical anatomical feature pair in the 3D medical image volume, wherein the bilaterally symmetrical anatomical feature pair includes a first anatomical feature and a second anatomical feature; and selecting the subvolume that includes the first anatomical feature and does not include the second anatomical feature. In a third example of the method (optionally including one or both of the first and second examples), the plurality of anatomical landmarks include at least three different landmarks. In a fourth example of the method (optionally including one or more or each of the first to third examples), estimating transformation parameters based on the plurality of anatomical landmarks to adjust the rotation angle and translation of the subvolume of interest includes: selecting a landmark subset from the plurality of anatomical landmarks detected in the subvolume of interest, wherein the landmark subset includes at least three landmarks; and determining an anatomical plane of interest relative to an image coordinate system based on the landmark subset. In a fifth example of the method (optionally including one or more or each of the first to fourth examples), adjusting the rotation angle and translation of the subvolume of interest using the transformation parameters to generate the first aligned subvolume of interest includes performing multi-planar reformation of the subvolume of interest along the anatomical plane of interest based on one or more geometric constraints imposed on the subset of landmarks. In a sixth example of the method (optionally including one or more or each of the first to fifth examples), determining the confidence of the transformation parameters based on the first aligned subvolume of interest includes: detecting the multiple anatomical landmarks in the first aligned subvolume of interest using the deep neural network; comparing the positions of the multiple anatomical landmarks in the first aligned subvolume of interest with one or more predetermined geometric constraints to determine the degree of geometric constraint satisfaction; and determining the confidence of the transformation parameters based on the degree of geometric constraint satisfaction.In a seventh example of the method (optionally including one or more or each of the first to sixth examples), the transformation parameters include a rotation angle and a translation vector. In an eighth example of the method (optionally including one or more or each of the first to seventh examples), iteratively refining the transformation parameters includes: detecting the plurality of anatomical landmarks in the first aligned subvolume of interest using the deep neural network; estimating the refined transformation parameters based on the plurality of anatomical landmarks to adjust the rotation angle and translation of the first aligned subvolume of interest; adjusting the rotation angle and translation of the first aligned subvolume of interest using the refined transformation parameters to generate a second aligned subvolume of interest; determining a confidence level of the refined transformation parameters based on the second aligned subvolume of interest; and further iteratively refining the refined transformation parameters if the confidence level is below the predetermined threshold. In a ninth example of the method (optionally including one or more or each of the first to eighth examples), the method further includes: determining the confidence of the refined transformation parameters based on the second aligned subvolume of interest by the following steps: projecting the second aligned subvolume of interest onto multiple projection planes to generate multiple projection images; and mapping the multiple projection images to corresponding multiple confidence scores using a confidence estimation network; and determining the confidence of the refined transformation parameters by majority voting using the multiple confidence scores corresponding to the multiple projection images.

[0100] The present disclosure also provides support for an image processing system, the image processing system comprising: a display device; a non-volatile memory, the non-volatile memory comprising instructions; and a processor, wherein when executing the instructions, the processor causes the image processing system to: receive a three-dimensional (3D) image volume; pre-process the 3D image volume by selecting a subvolume of interest; detect multiple landmarks in the subvolume of interest using a deep neural network trained to identify anatomical landmarks; estimate transformation parameters based on the multiple landmarks to adjust the rotation angle and translation of the subvolume of interest; use the transformation parameters to adjust the rotation angle and translation of the subvolume of interest to produce a first aligned subvolume of interest; determine the confidence of the transformation parameters based on the first aligned subvolume of interest; and iteratively refine the transformation parameters if the confidence is below a predetermined threshold. In a first example of the system, the multiple landmarks include at least three different landmarks. In a second example of the system (optionally including the first example), the processor further causes the image processing system to: select a landmark subset from the plurality of anatomical landmarks detected in the subvolume of interest, wherein the landmark subset includes at least three landmarks; and determine an anatomical plane of interest relative to the image coordinate system based on the landmark subset. In a third example of the system (optionally including one or more of the first and second examples), the processor further causes the image processing system to: perform multi-planar reconstruction of the subvolume of interest along the anatomical plane of interest. In a fourth example of the system (optionally including one or more or each of the first to third examples), the processor further causes the image processing system to: detect the plurality of landmarks in the first aligned subvolume of interest using the deep neural network; compare the positions of the plurality of landmarks in the first aligned subvolume of interest with one or more predetermined geometric constraints to determine the degree of geometric constraint satisfaction; and determine the confidence of the transformation parameter based on the degree of geometric constraint satisfaction. In a fifth example of the system (optionally including one or more or each of the first to fourth examples), the processor also causes the image processing system to: project the first aligned subvolume of interest onto a projection plane to produce a projection image, wherein the projection plane is parallel to multiple 2D images including the first aligned subvolume of interest; and map the projection image to a confidence score using a confidence estimation network.

[0101] The present disclosure also provides support for a method comprising: receiving a three-dimensional (3D) image volume comprising a first plurality of two-dimensional (2D) images parallel to a first plane; preprocessing the 3D image volume by selecting a subset of the first plurality of 2D images to generate a subvolume of interest comprising an anatomical region of interest; detecting at least three landmarks in the subvolume of interest using a deep neural network trained to identify anatomical landmarks; estimating transformation parameters of an anatomical plane of interest intersecting the subvolume of interest based on the at least three landmarks; performing multiplanar reconstruction of the subvolume of interest along the anatomical plane of interest to generate a first aligned subvolume of interest, wherein the first aligned subvolume of interest comprises a second plurality of 2D images parallel to the anatomical plane of interest; determining confidence in the transformation parameters based on the first aligned subvolume of interest; and iteratively refining the transformation parameters if the confidence is below a predetermined threshold. In a first example of the method, detecting the at least three landmarks includes: detecting a plurality of landmarks including more than three landmarks in the subvolume of interest; for each landmark in the plurality of landmarks: encoding the landmark as a feature vector; and comparing the feature vector with a distribution of previously encoded feature vectors of the same landmark type as the landmark; determining a probability that the landmark belongs to the distribution of previously encoded feature vectors; and determining a confidence level for the landmark based on the probability. In a second example of the method (optionally including the first example), the method further includes: filtering the plurality of landmarks based on confidence levels to obtain the at least three landmarks. In a third example of the method (optionally including one or both of the first and second examples), determining the confidence of the transformation parameters based on the first aligned subvolume of interest includes: detecting the at least three landmarks in the first aligned subvolume of interest using the deep neural network; comparing the positions of the at least three landmarks in the first aligned subvolume of interest with one or more predetermined geometric constraints to determine the degree of geometric constraint satisfaction; determining a geometric constraint confidence based on the degree of geometric constraint satisfaction; projecting the first aligned subvolume of interest onto a projection plane to generate a projection image; mapping the projection image to a projection confidence using a confidence estimation network; and determining an overall confidence of the first aligned subvolume by combining the geometric constraint confidence with the projection confidence.

[0102] As used herein, the elements or steps listed in the singular and beginning with the word "one" or "an" should be understood as not excluding a plurality of the elements or steps, unless such exclusion is explicitly stated. In addition, the reference to "one embodiment" of the present invention is not intended to be interpreted as excluding the existence of additional embodiments that also include the cited features. In addition, unless explicitly stated to the contrary, "comprising", "including" or "having" an embodiment of an element or multiple elements with a specific characteristic may include additional such elements that do not have the characteristic. The terms "including" and "in..." are used as the concise language equivalents of the corresponding terms "including" and "wherein". In addition, the terms "first", "second" and "third" etc. are only used as marks, and are not intended to impose numerical requirements or specific positional order on their objects.

[0103] This written description uses examples to disclose the invention, including the best mode, and also to enable a person skilled in the relevant art to practice the invention, including making and using any devices or systems and performing any included methods. The patentable scope of the invention is defined by the claims, and may include other examples that occur to a person skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insignificant differences from the literal language of the claims.

Claims

1. A method for automatic image alignment of a three-dimensional (3D) medical image volume, the method comprising: preprocessing the 3D medical image volume by selecting a subvolume of interest (104); detecting a plurality of anatomical landmarks in the subvolume of interest using a deep neural network trained to identify anatomical landmarks (106); estimating transformation parameters based on the plurality of anatomical landmarks to adjust the rotation angle and translation of the subvolume of interest (108); Using the transformation parameters to adjust the rotation angle and translation of the subvolume of interest to generate a first aligned subvolume of interest (110); determining confidence levels for the transformation parameters based on the first aligned subvolume of interest (112); as well as If the confidence level is below a predetermined threshold, the transformation parameters are iteratively refined (114).

2. The method according to claim 1, wherein the 3D medical image volume is an image sequence comprising a plurality of two-dimensional (2D) images, and wherein pre-processing the 3D medical image volume (104) by selecting the sub-volume of interest comprises: selecting a subset of the plurality of 2D images that includes the subvolume of interest (206); as well as The subset of the plurality of 2D images is cropped to remove regions outside the subvolume of interest (210).

3. The method of claim 1 , wherein selecting the subvolume of interest (104) comprises: identifying a pair of bilaterally symmetrical anatomical features in the 3D medical image volume, wherein the pair of bilaterally symmetrical anatomical features comprises a first anatomical feature and a second anatomical feature (208); and The subvolume is selected that includes the first anatomical feature and does not include the second anatomical feature (208). The method of claim 1 , wherein the plurality of anatomical landmarks comprises at least three different landmarks.

5. The method of claim 1, wherein estimating transformation parameters based on the plurality of anatomical landmarks to adjust a rotation angle and a translation of the subvolume of interest (108) comprises: selecting a subset of landmarks from the plurality of anatomical landmarks detected in the subvolume of interest, wherein the subset of landmarks includes at least three landmarks (308); and An anatomical plane of interest relative to an image coordinate system is determined based on the subset of landmarks (402).

6. The method of claim 5, wherein adjusting the rotation angle and translation of the subvolume of interest using the transformation parameters to produce the first aligned subvolume of interest (110) includes performing multi-planar reforming of the subvolume of interest along the anatomical plane of interest based on one or more geometric constraints imposed on the subset of landmarks.

7. The method of claim 1, wherein determining the confidence level (112) of the transformation parameters based on the first aligned subvolume of interest comprises: detecting the plurality of anatomical landmarks in the first aligned subvolume of interest using the deep neural network (502); comparing the positions of the plurality of anatomical landmarks in the first aligned subvolume of interest to one or more predetermined geometric constraints to determine the extent to which the geometric constraints are satisfied (504); as well as The confidence level of the transformation parameters is determined based on the degree to which the geometric constraints are satisfied (506). The method of claim 1 , wherein the transformation parameters include a rotation angle and a translation vector.

9. The method of claim 1 , wherein iteratively refining the transformation parameters comprises: detecting the plurality of anatomical landmarks in the first aligned subvolume of interest using the deep neural network (106); estimating refined transformation parameters based on the plurality of anatomical landmarks to adjust a rotation angle and a translation of the first aligned subvolume of interest (108); adjusting a rotation angle and a translation of the first aligned subvolume of interest using the refined transformation parameters to produce a second aligned subvolume of interest (110); determining confidence levels of the refined transformation parameters based on the second aligned subvolume of interest (112); as well as If the confidence level is lower than the predetermined threshold, the refined transformation parameters are further iteratively refined (114).

10. The method according to claim 9, further comprising: The confidence level of the refined transformation parameters is determined based on the second aligned subvolume of interest by the following steps (112): projecting the second aligned subvolume of interest onto a plurality of projection planes to generate a plurality of projection images (508); as well as Mapping the plurality of projection images to a corresponding plurality of confidence scores using a confidence estimation network (510); as well as Confidence of the refined transformation parameters is determined by majority voting using the plurality of confidence scores corresponding to the plurality of projection images (512).

11. An image processing system (802), the image processing system comprising: Display device (824); non-transitory memory (806), the non-transitory memory comprising instructions; and A processor (804), wherein when executing the instructions, the processor (804) causes the image processing system (802): receiving a three-dimensional (3D) image volume (102); preprocessing the 3D image volume by selecting a subvolume of interest (104); detecting a plurality of landmarks in the subvolume of interest using a deep neural network trained to identify anatomical landmarks (106); estimating transformation parameters based on the plurality of landmarks to adjust the rotation angle and translation of the subvolume of interest (108); Using the transformation parameters to adjust the rotation angle and translation of the subvolume of interest to generate a first aligned subvolume of interest (110); determining confidence levels for the transformation parameters based on the first aligned subvolume of interest (112); as well as If the confidence level is below a predetermined threshold, the transformation parameters are iteratively refined (114).

12. The image processing system (802) of claim 11, wherein the plurality of landmarks comprises at least three different landmarks.

13. The image processing system (802) of claim 11, wherein the processor (804) further causes the image processing system (802): selecting a subset of landmarks from the plurality of anatomical landmarks detected in the subvolume of interest (308), wherein the subset of landmarks includes at least three landmarks; and An anatomical plane of interest relative to an image coordinate system is determined based on the subset of landmarks (402).

14. The image processing system (802) of claim 13, wherein the processor (804) further causes the image processing system (802): Multiplanar reformat ion of the subvolume of interest is performed along the anatomical plane of interest.

15. The image processing system (802) of claim 11, wherein the processor (804) further causes the image processing system (802): detecting the plurality of landmarks in the first aligned subvolume of interest using the deep neural network (502); comparing the positions of the plurality of landmarks in the first aligned subvolume of interest to one or more (2D) predetermined geometric constraints to determine the degree to which the geometric constraints are satisfied (504); as well as The confidence level of the transformation parameters is determined based on the degree to which the geometric constraints are satisfied (506).

Images