Automatic 3D medical image orientation determination

Abstract

A system, method, and non-transitory computer-readable storage medium for aligning a set of medical images. Operations for aligning medical images include receiving a set of medical images, selecting an anchor organ from the medical images, training a segmentation model to identify the anchor organ in the medical images based on a training data set, and generating a segmentation mask for the anchor organ based on the segmentation model. The operations further include calculating image coordinates of the anchor organ in each medical image based on a centroid of each anchor organ, determining a correlation between the image coordinates of the anchor organ in each medical image and corresponding anatomical coordinates of the anchor organ in the training data set, and aligning each medical image in the set of medical images based on the correlation between the image coordinates and the anatomical coordinates.

Description

Background Technology

[0001] There are various two-dimensional (2D) and three-dimensional (3D) medical imaging modalities. Examples include computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and ultrasound (US) scans. These medical images can be presented in various forms once stored in a database. Researchers and clinicians access scans within these medical images to determine the orientation of different parts of the body. However, medical images are not always aligned due to factors such as patient movement during imaging. Furthermore, the various formats used to store medical images do not always support the stored metadata associated with patient information, resulting in the loss of some orientation information during processing. Determining body orientation solely based on scans from images becomes difficult. Additionally, clinicians and researchers spend considerable time rotating these medical images to conform to the desired representation. Therefore, a fast and robust method is needed that will allow clinicians to properly align medical images and calculate the transition from image coordinates to anatomical coordinates to normalize these images. Furthermore, aligning medical images will allow clinicians to automatically locate desired body parts.

[0002] Existing techniques focus on landmark localization. For example, US 20080260219A1 and US 8160322 describe existing methods for landmark localization. However, these methods have many drawbacks, such as the requirements for feature engineering, dependence on morphology or body parts and their dimensions, lack of invariance to affine transformations, and low computational efficiency. Moreover, these methods require data that has been specifically annotated with landmarks. Therefore, these methods are not fully compatible with image segmentation datasets commonly used in modern medical computer vision. Summary of the Invention

[0003] Some exemplary embodiments relate to a method comprising: receiving a set of medical images from a medical imager; selecting one or more anchor organs from the medical images; training a segmentation model based on a training dataset received from a database to identify the one or more anchor organs selected from the medical image set; and generating a segmentation mask for the selected one or more anchor organs based on the segmentation model. The method may further include calculating image coordinates of the selected one or more anchor organs in each medical image of the medical image set based on the centroid of each of the selected one or more anchor organs. The method may further include determining a correlation between the image coordinates of the selected one or more anchor organs in each medical image of the medical image set and corresponding anatomical coordinates of the selected one or more anchor organs in the training dataset, and aligning each medical image in the medical image set based on the correlation between the image coordinates and the anatomical coordinates.

[0004] Other exemplary embodiments relate to a system having a memory and a processor. The memory stores a set of medical images received from a medical imager and a training dataset. The processor is configured to select one or more anchor organs from the set of medical images, train a segmentation model based on the training dataset to identify the selected one or more anchor organs in the set of medical images, and generate a segmentation mask for the selected one or more anchor organs based on the segmentation model. The processor may also be configured to compute image coordinates of the selected one or more anchor organs in each medical image in the set based on the centroid of each of the selected one or more anchor organs. The processor may also be configured to determine the correlation between the image coordinates of the selected one or more anchor organs in each medical image in the set and the corresponding anatomical coordinates of the selected one or more anchor organs in the training dataset, and align each medical image in the set based on the correlation between the image coordinates and the anatomical coordinates.

[0005] In yet another exemplary embodiment, a non-transient computer-readable storage medium having an instruction set executable by a processor is described. Executing the instructions causes the processor to receive a set of medical images from a medical imager, select one or more anchor organs from the medical images, train a segmentation model based on a training dataset received from a database to identify the selected one or more anchor organs in the set of medical images, and generate a segmentation mask for the selected one or more anchor organs based on the segmentation model. Attached Figure Description

[0006] Figure 1 A system for orientation determination of 3D medical images according to various exemplary embodiments of the present disclosure is shown.

[0007] Figure 2 A flowchart of a method for generating orientation according to various exemplary embodiments of the present disclosure is shown.

[0008] Figure 3 Exemplary 3D images for selecting an anchoring organ are shown according to various exemplary embodiments of the present disclosure.

[0009] Figure 4 Exemplary segmentation models for generating segmentation masks from 3D are shown according to various exemplary embodiments of the present disclosure.

[0010] Figure 5 Exemplary diagrams for determining organ centers are shown according to various exemplary embodiments of the present disclosure.

[0011] Figure 6Exemplary diagrams are shown illustrating the alignment of medical images with coordinate systems of interest according to various exemplary embodiments of the present disclosure. Detailed Implementation

[0012] Exemplary embodiments can be further understood with reference to the following description and accompanying drawings, wherein like elements are designated by like reference numerals. Exemplary embodiments relate to systems and methods for automatically identifying body orientation in 2D or 3D medical images and conforming the images to a standard or custom anatomical coordinate system. Exemplary embodiments provide a system and method for aligning multiple images acquired from an imaging device and conforming the aligned images to a standard or anatomical coordinate system.

[0013] Furthermore, exemplary embodiments can be used to automatically locate desired body parts. For example, if a radiologist is categorizing and analyzing an abdominal region, 2D or 3D images aligned to a specific coordinate system allow the radiologist to automatically move the viewpoint to the desired area within that coordinate system.

[0014] Exemplary embodiments may be directed to 2D or 3D images. However, for illustrative purposes, exemplary embodiments will be described with reference to 3D images. Furthermore, exemplary embodiments will be described with reference to a CT scanner used to generate 3D images. However, the use of a CT scanner is merely exemplary, and any known imaging device, such as an MRI device, PET device, etc., may be employed in exemplary embodiments.

[0015] like Figure 1 As shown, a system 100 according to various exemplary embodiments of the present disclosure determines body orientation in a 3D medical image and conforms the image to a standard or custom anatomical coordinate system. The system 100 includes a CT scanner 102 for generating 3D scans. The CT scanner 102 images a portion of a patient's body and generates multiple 3D scans of that portion of the patient's body. Portions of the patient's body that can be imaged include the chest region, abdomen, etc. Those skilled in the art will understand that these body portions are merely exemplary, and therefore, other body portions can be imaged. The resulting multiple 3D scans may vary due to various factors during the imaging process, such as motion of the object during imaging.

[0016] System 100 also includes an analysis device 104. Analysis device 104 includes a processor 106 for performing various analyses, such as selecting anchor organs in a 3D scan, training a segmentation model to identify anchor organs, and aligning the 3D scan in a coordinate system of interest. Each of these operations will be described in more detail below. Analysis device 104 also includes a memory 108 for storing the 3D scan and segmentation model. Those skilled in the art will understand that analysis device 104 and CT scanner 102 can be connected via a wired or wireless connection.

[0017] System 100 also includes a database 110 connected to analysis device 104, which contains a training dataset for training the segmentation model. The training dataset can also be used to generate target coordinates for aligning 3D scans, as will be described further below. System 100 also includes a display unit 112 for displaying the aligned 3D scans in a coordinate system of interest.

[0018] The processor 106 in the analysis device 104 receives multiple 3D scans from a CT scanner. The 3D scans may be misaligned due to factors such as patient movement during imaging. Therefore, the processor 106 normalizes the 3D scans by first determining the orientation of the image using the organ locations in each scan. The processor 106 may select one or more anchor organs in the 3D scans. Selectable anchor organs are not limited to organs of the body and may include other body parts such as bones. Examples of anchor organs include the heart, liver, spinal cord, left lung, right lung, pelvis, stomach, etc. The anchor organs selected in the 3D scans may be based on a training dataset available in database 110 and a set of organs that will be present in the 3D images during the final application.

[0019] exist Figure 3 In the example shown, processor 106 selects four anchor organs, such as the liver, spleen, right kidney, and left kidney, as the set of anchor organs in each 3D scan of the object. However, it should be understood that the number of anchor organs may include fewer than four or more than four.

[0020] Processor 106 also trains a semantic segmentation model to identify anchor organs selected in the 3D scan, which will be described in detail below. Analysis device 104 applies the segmentation model to any new 3D scan of the object to identify anchor organs. The segmentation model is generated by any machine learning model used for semantic segmentation and is trained to consume medical images obtained from CT scanner 102. The segmentation model also outputs a segmentation mask representing the location of the selected anchor organ, such as... Figure 4 As shown.

[0021] like Figure 4 As shown, the segmentation model is trained to consume the entire volume of a 3D scan. Alternatively, the segmentation model is trained to consume low-resolution 3D scans. Training the segmentation model at low resolution advantageously enables fast inference during final application and facilitates the use of deep neural networks such as U-Net. In another exemplary embodiment, the segmentation model is trained to be scaling and rotation invariant by augmenting the training data with random scaling (without aspect ratio preservation) and rotation. Alternatively, to make the segmentation model usable for different body parts, it can be trained to generate another augmentation of random cropping of various body parts from 3D images.

[0022] A segmentation model is generated by using a machine learning algorithm trained on the training dataset in database 110. For example... Figure 3 As shown, anchoring organs typically vary in size. Therefore, class balancing can be used to train the training dataset in database 110 to identify anchoring organs of different sizes in 3D scans. Moreover, due to positive augmentation, class balancing of the segmentation model can be performed at the batch level. Memory 108 stores the segmentation model generated using machine learning algorithms and applies this model to each 3D scan to generate, for example, […]. Figure 4 The segmentation and masking shown. For example... Figure 4 As shown, the segmentation model consumes 3D medical images using the liver, spleen, right kidney, and left kidney selected as anchor organs to produce a 3D segmentation mask. The segmentation mask labels each voxel as either background or as belonging to one of the selected anchor organs, which will be described further below.

[0023] However, the scans that can be applied to the segmentation model to output the segmentation mask can have arbitrary orientation. The processor 106 can also use the segmentation mask to calculate the coordinates of the center of the selected anchoring organ to determine, for example... Figure 5 and Figure 6 The alignment between the image coordinate system and the anatomical coordinate system of the anchored organ is shown, which will be described further below. Display unit 112 is configured to display the aligned 3D scan. The various functions of processor 106 can also be represented as separately incorporated components or as modular components. Processor 106 and memory 108 can be embedded in, for example... Figure 1 The analysis device 104 shown may be used as a different component connected to the analysis device 104. The analysis device 104 may be connected to the display unit 112 via a wired or wireless connection. Additionally, the functions described for the processor 106 may be distributed among two or more processors. Exemplary embodiments may be implemented in any of these or other configurations.

[0024] Figure 2 An exemplary method 200 is illustrated, in which system 100 determines body orientation in a 3D scan and conforms the image to a standard or customized anatomical coordinate system. Method 200 includes receiving medical images from a CT scanner in 210. In 220, the method further includes selecting from the 3D scan received in 210, such as... Figure 3 Multiple anchored organs are shown to ultimately align multiple 3D scans. Figure 3 In the exemplary embodiment shown, the liver, spleen, right kidney, and left kidney are selected as anchor organs for the patient from the 3D scan. Organs are selected from the 3D scan based on organs available in the training dataset in database 110.

[0025] In step 230, a segmentation model is trained to identify anchor organs in each 3D scan applied to the segmentation model. The segmentation model is stored in memory 108 and applied to each new 3D scan obtained from the CT scanner 102. Figure 4 As shown, the segmentation model identifies anchor organs in a 3D scan, the shape of the anchor organs in the 3D scan, and which 3D scan voxels belong to the anchor organs or the background. The segmentation model is trained to segment 3D scans by assigning a label to each voxel of the image and generating a 3D segmentation mask. Different training datasets can be used for image segmentation. The training dataset used to train the segmentation model typically includes images, their corresponding labels, and voxel-based masks. The segmentation mask is a label for each voxel of the image. Each voxel in the mask is assigned at least one of two categories: class 1, belonging to the anchor organ, and class 2, belonging to the surrounding area of ​​the anchor organ or not belonging to the anchor organ. Voxels belonging to the anchor organ can be represented by different colors in the segmentation mask, while the background can be left black. In an exemplary embodiment, the liver, spleen, right kidney, and left kidney are represented as shown in the example. Figure 4 The different colors shown represent different elements, while the background is represented by black.

[0026] In step 240, the segmentation model generates a segmentation mask for the anchored organ identified in step 230. The segmentation model consumes 3D scans with the selected anchored organ and generates a 3D segmentation mask for each 3D scan. As previously mentioned, the scans can have arbitrary orientations.

[0027] Therefore, in step 250, the image coordinates of the centroids of the anchoring organs are determined using the 3D segmentation mask generated in step 240. The centroid of each anchoring organ can be determined by initially determining its position in the image coordinate system and its respective mass. The mass is multiplied by its respective position, summed, and divided by the sum of its total mass to obtain the point of the image coordinates of the centroid of each anchoring organ. The point obtained for each anchoring organ is represented by Corgan1...CorganN. In an exemplary embodiment, as... Figure 5 As shown, the image coordinates of the centroids of the liver, spleen, right kidney, and left kidney are calculated for each 3D scan. The centroids are represented by circles, but circles are merely exemplary and therefore other forms of indicating the centroids may be used.

[0028] In step 260, the alignment between the anchored organ position calculated in the 3D scan and the organ position in the training dataset from database 110 is determined. Alignment between the two different coordinate systems is determined based on coordinate system transformation calculations. The anchored organ is represented in the image coordinate system. The organ position from the training dataset can be represented in the anatomical coordinate system, although the anatomical coordinate system is merely exemplary. Method 100 utilizes a decision function to consume the organ position in the training dataset and approximates it with the anchored organ position calculated in the 3D scan. Decision functions that can be used include flip, shift, and rotation cases. Those skilled in the art will understand that these cases are merely exemplary and other decision functions can be applied. Three possible cases are further described below.

[0029] In the case of flipping, the transformation can be achieved by flipping several axes of the image coordinates calculated to be the centroid of the anchoring organ. Alternatively, the coordinate system can be shifted for the case of displacement, and the transformation can rotate the coordinate system at any angle for the case of displacement.

[0030] The implementation of the flipping involves considering rules about the relative positions of organs in the human body. For example, rules such as the heart always being above the liver or the left lung being on the left and the right lung on the right are known in the art. Therefore, by implementing such rules, method 200 is able to determine whether the image coordinate axes of the anchored organs need to be flipped to align with the anatomical coordinate axes of the correctly oriented training dataset in database 110. For example, if the Z-coordinate of the image coordinate axes represents the upper-lower axis, then the sign of the projection of the center of the difference onto the upper-lower axis sign ((C)) is... heart -C liver The sign [0; 0; 1] indicates whether the axis is flipped. If the sign is negative, the image axis is flipped. Those skilled in the art will understand that the location of the organ's center will determine how the axis is flipped.

[0031] As previously mentioned, the decision function used for coordinate system transformation calculations can also be a displacement case. In an exemplary embodiment, the displacement case can be handled by determining the average value of the points of the anchoring organ determined in 250. As indicated, The centroids of each anchored organ are summed and divided by the total number of anchored organs. Although the centroids are obtained from 3D scans, the determined average point depends on the body characteristics and organ locations within the 3D scan. Therefore, the average point can be robustly used as a starting point for any further positional correlation analysis.

[0032] Alternatively, the rotation case can be implemented as the decision function in 260 of the exemplary embodiment. In this exemplary embodiment, the task at hand is to determine the number of points required to find the transformation. In a typical rotation case, at least three anchor organs are required because having at least three non-collinear points is sufficient to describe the position of the rigid body. Those skilled in the art will understand that the number of anchors used for the rotation case is merely exemplary; therefore, more than three anchor organs can be used. Implementing the rotation case requires calculating the actual organ coordinates and the target coordinates in 250. The target coordinates can be pre-calculated by averaging the organ coordinates on correctly oriented images from the training dataset in database 110.

[0033] Therefore, if the coordinates of the anchored organ calculated in 250 are presented in the anatomical coordinate system, the actual organ coordinates (the coordinates of the anchored organ) and the target coordinates can be used to calculate the 3D rotation matrix. Additionally, the actual organ coordinates and the target coordinates can be used to transform the 3D image. Those skilled in the art will understand that the anatomical coordinate system is merely exemplary, and therefore other coordinate systems of interest can be used to represent the points.

[0034] In the case of rotation, the transformation point requires a pivot for the rotation. Therefore, a set C, calculated in 240, can be chosen. organ1 ...C organN The centroid is used as the pivot of rotation. Then, the vectors in the target and organ coordinates are transformed relative to the centroid and normalized to unit length. Typically, there is no exact rotation matrix for transforming the anchor organ into the target organ. Therefore, algorithms such as the Kabsch algorithm can be used to find approximations for transforming the anchor organ into the target organ. Those skilled in the art will understand that the use of the Kabsch algorithm is merely exemplary, and other algorithms can be used to approximate the rotation matrix.

[0035] like Figure 6 As shown, an exemplary embodiment uses a decision function to determine Figure 3 The coordinate system transformation calculation for 3D scanning is performed. In an exemplary embodiment, rotation is used to align target coordinates and anchor organ coordinates to produce an aligned 3D scan. Image 602 in the decision function represents the average body structure template calculated on the training dataset. "Etalon organ position" represents the target coordinates calculated from the correctly oriented image from the training dataset. Therefore, the orientation of the images from the training data will determine the location of the target coordinates. Image 604 shows the calculated organ location as described in 230-250.

[0036] Those skilled in the art will understand that the anchoring organ selected in image 604 depends on the organs available in the images of body structures from the training data. Figure 6As shown, the calculated organ location and the etalon organ location are misaligned. Therefore, the decision function consumes the etalon organ location and approximates it with the calculated organ location from the 3D scan. Matching is performed by calculating the transformation required to align the image with the coordinate system of interest. Figure 6 The exemplary case shown illustrates a 90° rotation, representing a rotation of 260°. Those skilled in the art will understand that other exemplary cases can be applied based on the orientation of the 3D scan relative to the target coordinates in the training dataset.

[0037] The exemplary embodiments described above can be used for data-intensive research. For example, when training a learning model, users need to ensure that images are uniformly oriented or that only certain specific parts of the body are present in the image. Furthermore, the trained segmentation model can be used as a tool for external research (such as the Health and Safety Institute (HIS) platform). The exemplary embodiments can also be used in end-user applications. For example, medical personnel can use the system to align images they see with a specific coordinate system.

[0038] While the invention has been detailed and described in the accompanying drawings and foregoing description, such description and description should be considered illustrative or exemplary rather than restrictive; the invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and implemented by those skilled in the art in practicing the claimed invention by studying the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit can implement the functions of several items as described in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not imply that combinations of these measures cannot be advantageously used. Computer programs may be stored / distributed on suitable media, such as optical storage media or solid-state media provided with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. Any reference numerals in the claims should not be construed as limiting the scope.

Claims

1. A method for aligning medical images, comprising: Receive a set of medical images from a medical imager; Select one or more anchoring organs from the medical images; A segmentation model is trained based on a training dataset received from a database to identify one or more anchored organs selected from the medical image set; Based on the segmentation model, a segmentation mask is generated for the selected one or more anchored organs; Based on the centroid of each of the selected one or more anchored organs, the image coordinates of the selected one or more anchored organs in each medical image in the medical image set are calculated; Determine the correlation between the image coordinates of the selected one or more anchored organs in each medical image in the medical image set and the corresponding anatomical coordinates of the selected one or more anchored organs in the training dataset; as well as Each medical image in the medical image set is aligned based on the correlation between the image coordinates and the anatomical coordinates.

2. The method of claim 1, wherein the alignment is further based on a decision function, the decision function including one of a flip, a shift, or a rotation.

3. The method of claim 2, wherein the alignment is further based on a set of rules of human anatomical features.

4. The method of claim 1, wherein the segmentation model is trained to consume the low-resolution medical image set.

5. The method of claim 1, wherein the segmentation model is trained to be scale and rotation invariant based on the training dataset augmented with random scaling and rotation.

6. The method of claim 1, wherein the segmentation model is trained to segment each medical image in the set of medical images by assigning a label to each voxel of each medical image, wherein the label includes one of the following: the voxel belongs to one or more selected anchor organs, the voxel belongs to the vicinity of one or more selected anchor organs, or the voxel does not belong to one or more selected anchor organs.

7. A system for aligning medical images, comprising: The memory stores a collection of medical images received from the medical imager and a training dataset; A processor is configured to select one or more anchor organs from the medical image set, train a segmentation model to identify the selected one or more anchor organs in the medical image set based on the training dataset, and generate a segmentation mask for the selected one or more anchor organs based on the segmentation model. The processor is further configured to calculate the image coordinates of the selected one or more anchor organs in each medical image in the medical image set based on the centroid of each of the selected one or more anchor organs; as well as The processor is further configured to determine the correlation between the image coordinates of the selected one or more anchored organs in each medical image in the medical image set and the corresponding anatomical coordinates of the selected one or more anchored organs in the training dataset, and to align each medical image in the medical image set based on the correlation between the image coordinates and the anatomical coordinates.

8. The system of claim 7, wherein the alignment is further based on a decision function, the decision function including one of a flip condition, a shift condition, or a rotation condition.

9. The system of claim 7, wherein the segmentation model is trained to consume the low-resolution medical image set.

10. The system of claim 7, wherein the segmentation model is trained to be scale and rotation invariant based on a training dataset augmented with random scaling and rotation.

11. The system of claim 7, wherein the segmentation model is trained to segment each medical image in the set of medical images by assigning a label to each voxel of each medical image, wherein the label includes one of the following: the voxel belongs to one or more selected anchor organs, the voxel belongs to the vicinity of one or more selected anchor organs, or the voxel does not belong to one or more selected anchor organs.

12. A non-transient computer-readable storage medium comprising a set of instructions executable by a processor, wherein executing the instructions causes the processor to perform operations, the operations including: Receive a set of medical images from a medical imager; Select one or more anchoring organs from the medical images; Based on the training dataset received from the database, a segmentation model is trained to identify one or more anchored organs selected from the medical image set; Based on the segmentation model, a segmentation mask is generated for the selected one or more anchored organs; Based on the centroid of each of the selected one or more anchored organs, the image coordinates of the selected one or more anchored organs in each medical image in the medical image set are calculated; Determine the correlation between the image coordinates of the selected one or more anchored organs in each medical image in the medical image set and the corresponding anatomical coordinates of the selected one or more anchored organs in the training dataset; as well as Each medical image in the medical image set is aligned based on the correlation between the image coordinates and the anatomical coordinates.

13. The non-transient computer-readable storage medium of claim 12, wherein the alignment is further based on a decision function, the decision function including one of a flip condition, a shift condition, or a rotation condition.

14. The non-transient computer-readable storage medium of claim 12, wherein the segmentation model is trained to perform one of the following: (i) consuming the low-resolution medical image set, (ii) scaling and rotation invariant based on a training dataset enhanced with random scaling and rotation, or (iii) segmenting each medical image in the medical image set by assigning a label to each voxel of each medical image.

Images