System and method for segmentation of anatomical structures in image analysis

Abstract

To provide a system and a method for anatomic structure segmentation in image analysis, using a computer system.SOLUTION: One method includes: receiving an annotation and a plurality of keypoints for an anatomic structure in one or more images; computing distances from the plurality of keypoints to a boundary of the anatomic structure; training a model, by using data in the one or more images and the computed distances, so as to predict a boundary in the anatomic structure in an image of a patient's anatomy; receiving the image of the patient's anatomy including the anatomic structure; estimating a segmentation boundary in the anatomic structure in the image of the patient's anatomy; and predicting, by using the trained model, a boundary location in the anatomic structure in the image of the patient's anatomy by generating regression of distances from keypoints in the anatomic structure in the image of the patient's anatomy to the estimated boundary.SELECTED DRAWING: None

Description

[Technical Field]

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 62 / 503,838, filed May 9, 2017, the entire disclosure of which is incorporated herein by reference in its entirety.

[0002] Various embodiments of the present disclosure relate generally to medical imaging and related methods, and in particular, certain embodiments of the present disclosure relate to systems and methods for segmentation of anatomical structures in image analysis. [Background technology]

[0003] The problem of segmenting an image into multiple segments commonly arises in computer vision and medical image analysis. Current approaches automate this process using convolutional neural networks (CNNs), which are trained to predict class labels for each image element (e.g., pixel or voxel). CNNs typically contain multiple convolutional layers that pass the input (e.g., an image or a portion of an image) through a set of learnable filters and a nonlinear activation function. The use of convolutional operations makes CNNs equivariant to translations. For example, a translated version of the input can yield a proportionally translated version of the predicted segmentation label. A series of layers with different strides of convolutions allows CNNs to represent long-range interactions in an image in terms of local, short-range statistics.

[0004] However, the segmentation boundaries of current CNNs may be accurate down to the image element (e.g., pixel or voxel) level. In many image applications, placing the segmentation boundary at a pixel or voxel location may introduce quantization errors. It may be known (e.g., a priori) that a target structure may exist as a single connected component without holes. However, such assumptions may not be incorporated into the CNN, resulting in predicted labels that may have spurious components and holes within the segmented object. Therefore, it is desirable to build a model, such as a CNN, that can achieve sub-pixel or sub-voxel accurate segmentation and predict the label of a single connected component without holes or isolated structures.

[0005] The present disclosure is directed to overcoming one or more of the problems or concerns set forth above. Summary of the Invention [Means for solving the problem]

[0006] According to certain aspects of the present disclosure, a system and method for segmenting an anatomical structure in image analysis are disclosed. One method for segmenting an anatomical structure in image analysis includes receiving annotations of an anatomical structure and a plurality of key points in one or more images, calculating distances from the plurality of key points to boundaries of the anatomical structure, training a model using data from the one or more images and the calculated distances to predict boundaries within the anatomical structure in images of the patient's anatomy, receiving images of the patient's anatomy including the anatomical structure, estimating segmentation boundaries within the anatomical structure in the images of the patient's anatomy, and predicting boundary locations within the anatomical structure in the images of the patient's anatomy using the trained model by generating a regression of distances from the key points within the anatomical structure in the images of the patient's anatomy to the estimated boundaries.

[0007] According to another embodiment, a system for anatomical structure segmentation in image analysis is disclosed, including a data storage device storing instructions for anatomical structure segmentation in image analysis, and a processor configured to execute the instructions to perform a method including the steps of receiving annotations of an anatomical structure and a plurality of key points in one or more images, calculating distances from the plurality of key points to boundaries of the anatomical structure, training a model using data from the one or more images and the calculated distances to predict boundaries within the anatomical structure in images of the patient's anatomy, receiving images of the patient's anatomy including the anatomical structure, estimating segmentation boundaries within the anatomical structure in the images of the patient's anatomy, and predicting boundary locations within the anatomical structure in the images of the patient's anatomy by generating a regression of distances from key points within the anatomical structure in the images of the patient's anatomy to the estimated boundaries using the trained model.

[0008] According to yet another embodiment, a non-transitory computer-readable medium for use on a computer system is provided that includes computer-executable programming instructions for implementing a method for anatomical structure segmentation in image analysis, the method including receiving annotations of an anatomical structure and a plurality of keypoints in one or more images, calculating distances from the plurality of keypoints to boundaries of the anatomical structure, training a model using data from the one or more images and the calculated distances to predict boundaries within the anatomical structure in images of the patient's anatomy, receiving images of the patient's anatomy that include the anatomical structure, estimating segmentation boundaries within the anatomical structure in the images of the patient's anatomy, and predicting boundary locations within the anatomical structure in the images of the patient's anatomy by generating a regression of distances from the keypoints within the anatomical structure in the images of the patient's anatomy to the estimated boundaries using the trained model.

[0009] Additional objects and advantages of the disclosed embodiments will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the disclosed embodiments. The objects and advantages of the disclosed embodiments will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims.

[0010] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit the scope of the disclosed embodiments as claimed. The present invention provides, for example, the following items. (Item 1) 1. A computer-implemented method for segmentation of anatomical structures in image analysis, comprising: receiving annotations of anatomical structures and a plurality of key points in one or more images; calculating distances from a plurality of said key points to boundaries of said anatomical structure; training a model using data from one or more of the images and the calculated distances to predict boundaries within the anatomical structure in images of the patient's anatomy; receiving the image of the patient's anatomy including the anatomical structure; estimating segmentation boundaries within the anatomical structure in the image of the patient's anatomy; and predicting boundary locations within the anatomical structures in the images of the patient's anatomy by generating a regression of distances from keypoints within the anatomical structures in the images of the patient's anatomy to the estimated boundaries using the trained model. The computer-implemented method includes: (Item 2) Item 10. The computer-implemented method of item 1, wherein the annotation of the anatomical structure is in the form of a mesh, voxels, implicit surface representation, or point cloud. (Item 3) Item 10. The computer-implemented method of item 1, wherein the location of the received keypoints within the anatomical structure is known. (Item 4) Item 10. The computer-implemented method of item 1, further comprising fitting a shape model to the anatomical structure to determine the locations of a plurality of the key points. (Item 5) Item 1. The computer-implemented method of item 1, wherein the image coordinates are continuous. (Item 6) Item 10. The computer-implemented method of item 1, wherein the mapping is performed from multiple polar dimensions or from a polar form in one dimension and a linear form in one or more additional dimensions. (Item 7) 2. The computer-implemented method of claim 1, further comprising: determining image intensities along a ray associated with the calculated distance before training the model; and wherein the ray associated with the distance is at a fixed position in the Euclidean space. (Item 8) 2. The computer-implemented method of claim 1, wherein predicting the boundary location comprises predicting an indirect representation of the distance to the estimated boundary from key points in the anatomical structure in the image of the patient's anatomy. (Item 9) Item 10. The computer-implemented method of item 1, wherein estimating the segmentation boundary of the anatomical structure in the image of the patient's anatomy includes obtaining a set of key points within the anatomical structure. (Item 10) Item 10. The computer-implemented method of item 1, wherein the predicted boundary locations within the anatomical structures in the image of the patient's anatomy are sub-voxel accurate boundary locations. (Item 11) Item 10. The computer-implemented method of item 1, further comprising constructing a three-dimensional surface based on the predicted boundary locations. (Item 12) Item 10. The computer-implemented method of item 1, further comprising outputting the predicted boundary location to an electronic storage medium. (Item 13) Item 10. The computer-implemented method of item 1, wherein the anatomical structure includes a blood vessel and the patient's anatomy includes a blood vessel of the patient's vasculature. (Item 14) Item 10. The computer-implemented method of item 1, wherein the annotation of the anatomical structure comprises a vessel lumen boundary, a vessel lumen centerline, a vessel lumen surface, or a combination thereof. (Item 15) 2. The computer-implemented method of claim 1, further comprising: defining a mapping from a plurality of image coordinates in one or more of the images to a Euclidean space, wherein a plurality of the image coordinates lie within rays in the Euclidean space and one of a plurality of the key points lies on each of the rays; and wherein calculating the distances comprises calculating distances from a plurality of the key points on each of the rays to the boundary of the anatomical structure. (Item 16) Item 10. The computer-implemented method of item 1, wherein the generated regression is continuous. (Item 17) 1. A system for segmentation of anatomical structures in image analysis, comprising: a data storage device storing instructions for segmenting anatomical structures; Execute the instructions, receiving annotations of anatomical structures and a plurality of key points in one or more images; calculating distances from a plurality of said key points to boundaries of said anatomical structure; training a model using data from one or more of the images and the calculated distances to predict boundaries within the anatomical structure in images of the patient's anatomy; receiving the image of the patient's anatomy including the anatomical structure; estimating segmentation boundaries within the anatomical structure in the image of the patient's anatomy; and predicting boundary locations within the anatomical structures in the images of the patient's anatomy by generating a regression of distances from keypoints within the anatomical structures in the images of the patient's anatomy to the estimated boundaries using the trained model. and a processor configured to perform a method including: (Item 18) 20. The system of claim 17, wherein the anatomical structure comprises a blood vessel and the patient's anatomy comprises a blood vessel of the patient's vasculature. (Item 19) 18. The system of claim 17, wherein the predicted boundary locations within the anatomical structures in the images of the patient's anatomy are sub-voxel accurate boundary locations. (Item 20) 1. A non-transitory computer readable medium for use on a computer system comprising computer executable programming instructions for implementing a method for segmentation of anatomical structures in image analysis, said method comprising: receiving annotations of anatomical structures and a plurality of key points in one or more images; calculating distances from a plurality of said key points to boundaries of said anatomical structure; training a model using data from one or more of the images and the calculated distances to predict boundaries within the anatomical structure in images of the patient's anatomy; receiving the image of the patient's anatomy including the anatomical structure; estimating segmentation boundaries within the anatomical structure in the image of the patient's anatomy; and the non-transitory computer-readable medium comprising: predicting a boundary location within the anatomical structure in an image of the patient's anatomy by generating a regression of distances from keypoints within the anatomical structure in the image of the patient's anatomy to the estimated boundary using the trained model.

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram of an exemplary system and network for anatomical structure segmentation in image analysis, according to an exemplary embodiment of the present disclosure. [Figure 2] 2A and 2B are a flowchart of an exemplary method for segmentation of anatomical structures in image analysis, according to an exemplary embodiment of the present disclosure. [Figure 3] 3A and 3B are a flowchart of an exemplary embodiment of the method of FIGS. 2A and 2B applied to coronary artery segmentation, according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] Exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings, which illustrate the disclosure, and wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

[0014] As noted above, the accuracy of segmentation boundaries determined by current techniques may be limited to image elements, e.g., pixels or voxels. In such cases, placing the segmentation boundary at a voxel location may introduce errors. In some cases, current predictive models may not take into account certain assumptions, such as that the target structure does not contain holes or isolated structures. Therefore, it is desirable to build a model that can predict segmentation boundaries with sub-pixel or sub-voxel accuracy and / or ensure that important assumptions are incorporated into the model.

[0015] The present disclosure is directed to enabling accurate prediction of segmentation boundary locations. In one embodiment, the present disclosure may include both a training phase and a testing (or use) phase for estimating segmentation boundaries. One or more parameters of a learning system for building a trained model may be optimized during the training phase. In the testing phase, the trained model may be used to segment unseen or seen images.

[0016] For example, the disclosed systems and methods can be applied to segmenting anatomical structures in received image(s) of a patient of interest and determining the boundaries of the structures of interest at the sub-pixel or sub-voxel level. As used herein, a boundary of a structure may include a boundary of a segment of the structure. In one embodiment, the training phase can include building a model to predict the distance from a keypoint within the structure of interest to the boundary of the structure of interest or the boundary of that segment. For example, the training phase can receive multiple keypoints with known locations within the structure of interest and calculate the distance from the keypoint to the boundary of the structure of interest for that segment (e.g., based on the known locations). A model (e.g., a CNN model) can then be trained based on the keypoint locations, the calculated distances, and / or the data in the received images. The trained model can regress sample distances or predict an indirect representation of sample distances. The regressions from the trained model can be continuous, thus enabling boundary locations to be predicted based on the regressed distances with sub-pixel or sub-voxel accuracy.

[0017] In one embodiment, the testing phase may include receiving images of a patient's anatomy. The patient may be a patient of interest, e.g., a patient seeking a diagnostic test. The testing phase may estimate a boundary of the target structure based on one or more images of the patient's anatomy, and use the model built in the training phase to predict the boundary location by regressing the distance from keypoints in the target structure to the estimated boundary.

[0018] As used herein, the term "exemplary" is used in the sense of "example" rather than "ideal." While this exemplary embodiment is written in the context of medical image analysis, the present disclosure is equally applicable to any non-medical image analysis or computer vision assessment.

[0019] Referring now to the drawings, FIG. 1 illustrates a block diagram of an exemplary system and network environment for anatomical structure segmentation in image analysis. Specifically, FIG. 1 illustrates multiple physicians 102 and third-party providers 104, any of which may be connected to an electronic network 100, such as the Internet, via one or more computers, servers, and / or handheld mobile devices. The physicians 102 and / or third-party providers 104 may create or otherwise acquire images of one or more patients' cardiac, vascular, and / or organ systems. The physicians 102 and / or third-party providers 104 may also acquire any combination of patient-specific information, such as age, medical history, blood pressure, blood viscosity, etc. The physicians 102 and / or third-party providers 104 may transmit the cardiac / vascular / organ images and / or patient-specific information to a server system 106 via the electronic network 100. The server system 106 may include storage for storing images and data received from the physicians 102 and / or third-party providers 104. The server system 106 may also include a processing unit for processing the images and data stored in the storage device. Alternatively or additionally, the disclosed anatomical structure segmentation (or portions of the disclosed systems and methods) may be performed on a local processing unit (e.g., a laptop) in the absence of an external server or network.

[0020] Figures 2A and 2B describe an exemplary method for performing segmentation of anatomical structures using a learning system. Figures 3A and 3B are directed to a specific embodiment or application of the method described in Figures 2A and 2B. By way of example, Figures 3A and 3B describe an embodiment of segmenting blood vessels using a learning system. All of this method may be performed by server system 106 based on information, images, and data received from physician 102 and / or third-party provider 104 over electronic network 100.

[0021] 2A and 2B describe an exemplary method for segmenting an anatomical structure in image analysis. In one embodiment, this segmentation of an anatomical structure may include two phases: a training phase and a testing phase. The training phase may include training a learning system (e.g., a deep learning system) to predict boundary locations of a target structure or a segment thereof with sub-pixel or sub-voxel accuracy. The testing phase may include predicting boundary locations of a target structure or a segment thereof in a newly received image.

[0022] 2A is a flowchart of an exemplary training phase method 200 for training a learning system (e.g., a deep learning system) to predict boundary locations, according to various embodiments. Method 200 can provide the basis for a testing phase of method 210 of FIG. 2B for anatomical structure segmentation of imaged target structures of a particular patient. Method 200 can include one or more of steps 201-207 shown in FIG. 2A. In some embodiments, method 200 can include repeating one or more of steps 201-207, e.g., repeating steps 201-207 one or more times.

[0023] In some embodiments, step 201 of method 200 may include receiving one or more images and / or image data in an electronic storage medium (e.g., a hard drive, a network drive, a cloud drive, a mobile phone, a tablet, a database, etc.). In a medical context, these images may be from a medical imaging device, such as, for example, a computed tomography (CT), a positron emission tomography (PET), a single photon emission computed tomography (SPECT), a magnetic resonance imaging (MRI), a microscope, an ultrasound, a (multi-view) angiogram, etc. In one embodiment, multiple images may be used for a single patient. In further embodiments, the images may include the patient's anatomy. In other embodiments, the images may be of multiple individuals with similar anatomical features or multiple individuals with different anatomical features. In a non-medical context, the images may be from any source, such as a camera, a satellite, a radar, a lidar, a sonar, a telescope, a microscope, etc. In the following disclosure, the images received in step 201 may be referred to as "training images."

[0024] In some embodiments, step 202 may include receiving annotations for one or more structures of interest in one or more of the training images. In some cases, one or more of the training images may include a structure of interest, e.g., an anatomical structure of a patient. In one example, all of the training images may include the structure of interest. In one embodiment, all of the training images may be annotated. This type of embodiment may be referred to as “supervised learning.” Another embodiment may include only a portion of the training images annotated. This type of scenario may be referred to as “semi-supervised learning.” In one embodiment, the structure of interest may include a vessel or tissue of the patient. In such cases, the annotation(s) may include labels such as vessel name (e.g., right coronary artery (RCA), left anterior descending artery (LAD), left circumflex artery (LCX), etc.), vascular landmarks (e.g., aortic valve location, ostium location, bifurcation points, etc.), estimated vessel location, landmarks (e.g., portions of the image noted as fuzzy, poorly defined, or unclear), etc. In some embodiments, the annotation(s) may be in a variety of formats, including but not limited to a mesh, voxels, implicit surface representation, or point cloud.

[0025] In some embodiments, step 203 may include receiving a plurality of keypoints within the target structure. The locations of the keypoints within the target structure may be known (e.g., based on annotations for the target structure and / or data in the received image). For example, one or more of the keypoints may be known to be within the target structure, on the boundary of the target structure, or near the boundary of the target structure. In some embodiments, fitting a shape model to the target structure may determine the boundary location (e.g., approximate boundary location) of the target structure and / or the locations of the keypoints, and / or obtain keypoints with known locations within the target structure.

[0026] In some embodiments, step 204 may include defining a mapping from image coordinates in the training images to Euclidean space. The image coordinates may be continuous. The image coordinates may be intersections of 3D object structures and / or rays. The image coordinates may lie within the rays in Euclidean space. One of the keypoints received in step 203 may lie on each of the rays.

[0027] In one embodiment, a given keypoint may be known to be inside the structure of interest. In such a scenario, step 204 may involve formulating the mapping in polar form with the given keypoint at its center. In this setting, image coordinates along isometric rays emanating from the keypoint may be selected.

[0028] In another scenario, the mapping of step 204 may include mapping from multiple polar dimensions to Euclidean space. For example, it is possible to parameterize a target structure (e.g., a three-dimensional (3D) target structure) by sampling from a spherical coordinate system. In this embodiment, two rotational dimensions may be mapped to three Euclidean dimensions given by the distance along the ray and two dimensions associated with the two rotational dimensions.

[0029] In another scenario, the mapping of step 204 may involve mapping from a one-dimensional polar form and a linear form in one or more additional dimensions to Euclidean space. For example, a tubular structure may be represented by landmarks on a series of closed curves. In that case, the closed curves may be represented in polar form but linear in the direction along the tube. This mapping therefore results in a state of three-dimensional Euclidean space, where the first dimension corresponds to the distance along the sampled ray, the second dimension corresponds to the rotational dimension, and the third dimension corresponds to the linear dimension along the tube.

[0030] In one embodiment, a set of keypoints near the image surface may be received in step 203. For example, the set of keypoints may be located along a set of closed curves on a given 3D mesh or 3D implicit representation. In that case, step 204 may include defining a set of equal-length rays, each containing a keypoint and directed perpendicular to the 3D mesh, to further refine the boundary of the target structure. Then, when mapping to Euclidean space in step 204, the distance along the defined rays may represent one of the Euclidean dimensions, while the keypoints on the closed curves may represent a second dimension.

[0031] In some embodiments, step 205 may include calculating, for each ray mapped in step 204, the distance from a keypoint on the ray to the boundary of the target structure. This calculated distance may be a target value for the training system.

[0032] In some embodiments, step 206 may include determining, for each target value, image intensity along the ray defined in step 204. The defined ray is associated with a key point on the ray and therefore associated with the target value. Additionally, step 206 may include ensuring that the ray associated with the target value is at a fixed position. For example, a first ray associated with a first target value may be at a fixed first position. Other rays associated with other target values ​​may have coordinates relative to the first ray, and therefore the positions of these other rays may also be fixed based on the position of the first ray. In one embodiment, the inputs for each of the target value distances may be cyclic transforms of each other.

[0033] In some embodiments, step 207 may include training a model (e.g., a CNN model) to predict segmentation boundary locations of structures of interest in newly received images. For example, the model may be trained to predict the distance from keypoints to the segmentation boundary of the structure of interest. In one embodiment, the model may be trained to regress the distance. The regression value may be continuous so that boundary locations can be predicted with sub-pixel or sub-voxel accuracy. In certain embodiments, the model may be trained to predict an indirect representation of distance. For example, the model may quantize rays into multiple small bins and / or predict bins that correspond to distances.

[0034] 2B is a block diagram of an exemplary test-phase (or use-phase) method 210 for predicting segmentation boundaries of a structure of interest in a particular patient image, according to an exemplary embodiment of the present disclosure. Method 210 may include one or more of steps 211-215. In one embodiment, the segmentation boundaries may be predicted using a trained model (e.g., provided by method 200).

[0035] In some embodiments, step 211 may include receiving one or more patient images in an electronic storage medium (e.g., a hard drive, a network drive, a cloud drive, a mobile phone, a tablet, a database, etc.). In one embodiment, the images may include medical images, e.g., the images may be provided by any medical imaging device, e.g., CT, MR, SPECT, PET, microscope, ultrasound, (multi-view) angiography, etc. In one embodiment, training images (e.g., of method 200) may include images acquired from a patient, and step 211 may also include receiving images of the patient. Alternatively, or additionally, step 211 may include receiving one or more images from a non-medical imaging device, e.g., a camera, a satellite, radar, lidar, sonar, telescope, microscope, etc. In the following steps, the images received during step 211 may be referred to as “test images.”

[0036] In some embodiments, step 212 may include estimating a segmentation boundary of the target structure or the location or boundary of another object (e.g., an object different from the target structure) in the test image. The estimated boundary or the location or boundary of the other object may be used to initialize an automated segmentation system (e.g., a centerline). A set of keypoints may be obtained from this initial segment. In one embodiment, the set of keypoints may include keypoints within the target structure.

[0037] In some embodiments, step 213 may include defining a mapping from image coordinates in the test image to Euclidean space. The image coordinates may be continuous. The image coordinates may lie within rays in Euclidean space. A keypoint (e.g., one of the keypoints obtained in step 202) may be present in each ray that may contain a keypoint. In one embodiment, the input for this mapping may be similar to the input for the mapping in step 204.

[0038] In some embodiments, step 214 may include predicting a boundary of the target structure using the model trained by method 200. In one embodiment, this prediction may include regressing the distance from keypoints on the ray defined in step 213 to the estimated boundary. In some cases, this regression may be continuous, thus allowing the boundary to be predicted with sub-pixel or sub-voxel accuracy. In one embodiment, step 214 may further include obtaining a surface from the predicted boundary (e.g., boundary point cloud). This surface may be obtained using a surface reconstruction method such as Poisson surface reconstruction.

[0039] In some embodiments, step 215 may include outputting the predicted boundaries of the target structures (e.g., complete segmentation boundaries) to an electronic storage medium (e.g., a hard drive, a network drive, a cloud drive, a mobile phone, a tablet, a database, etc.) Additionally, step 215 may include displaying the output results on an interface.

[0040] Figures 3A and 3B are directed to specific embodiments or applications of the exemplary method described in Figures 2A and 2B, for example, Figures 3A and 3B describe exemplary training and testing phases, respectively, for coronary artery segmentation in image analysis.

[0041] The accuracy of patient-specific segmentation of blood vessels, e.g., coronary arteries, can affect medical assessments such as blood flow simulations and calculations of vessel geometric properties. If the segmentation accuracy is insufficient, e.g., if it is limited to the level of image elements (e.g., pixels or voxels), the medical assessment may produce erroneous results. Figures 3A and 3B illustrate an exemplary method for segmenting coronary arteries with sub-pixel or sub-voxel accuracy. While the exemplary method herein uses coronary arteries, the method illustrated in Figures 3A and 3B may also be used to segment other types of blood vessels or anatomical structures other than blood vessels.

[0042] 3A is a flowchart of an exemplary method 300 of a training phase designed to provide a foundation for sub-voxel segmentation of coronary arteries, according to various embodiments. Method 300 may include one or more of steps 301-308 shown in FIG. 3A. In some embodiments, method 200 may include repeating one or more of steps 301-308, e.g., repeating steps 301-308 one or more times.

[0043] In some embodiments, step 301 may include receiving images of one or more coronary arteries in an electronic storage medium (e.g., a hard drive, a network drive, a cloud drive, a mobile phone, a tablet, a database, etc.). These images may be from a medical imaging device such as CT, MR, SPECT, PET, ultrasound, (multi-view) angiography, etc. These images may be referred to as "training images."

[0044] In some embodiments, step 302 may include receiving annotations for the coronary arteries in one or more of the training images. For example, the annotations may include vessel lumen boundaries and / or vessel lumen centerline(s). In one embodiment, step 303 may include receiving or generating a geometric mesh of the coronary vessels represented in the received images. The geometric mesh may be specified as a set of vertices and edges. Alternatively, or additionally, step 303 may include receiving centerlines of the coronary vessels. The centerlines may also be represented as a set of vertices that may be connected by edges.

[0045] In some embodiments, step 303 may include converting the training image data (e.g., geometric mesh, vertices, edges, centerlines, etc.) into a curved planar representation (CPR). This conversion may allow for a simplified vessel segmentation process. For example, a set of planes (e.g., frames) may be extracted along the centerline (e.g., orthogonal to the centerline) to construct a 3D volume. In one embodiment, the 3D volume may include a CPR, where a coordinate system frame of reference defines two dimensions and the length of the centerline defines the third dimension. In one embodiment, the curved planar representation may exclude degrees of freedom (e.g., centerline curvature), which may not be relevant to predicting one or more parameters of the coronary vessels. For example, the centerline curvature may be irrelevant to determining parameters related to the location of the luminal boundary of the coronary vessels.

[0046] In some embodiments, step 304 may include defining key points based on the image data. For example, step 304 may include defining points on the centerline of a coronary artery as key points. Key points defined in this manner may be interpreted as being inside the vessel. These key points may not necessarily be centrally located. Nevertheless, in some cases, these key points may be at the center of each frame, depending on the structure.

[0047] In some embodiments, step 305 may include defining, for each frame, a mapping from image coordinates in the test image to Euclidean space. In one embodiment, this mapping may be defined using polar coordinate sampling within the frame. In particular embodiments, defining the mapping may include determining CPR intensity values ​​at a set of angular directions around the keypoints defined in step 304. The determined CPR intensity values ​​may be arranged, for example, such that radial and angular coordinates are mapped to a two-dimensional (2D) image. For example, a discrete set of samples specified by radial and angular components of the frame may be mapped to rows and columns of the 2D image that point to the radial and angular components. Each row of CPR intensity values ​​may be defined as a radial coordinate, and each column of CPR intensity values ​​may be defined as an angular coordinate.

[0048] In some embodiments, step 306 may include defining a target regression value. The target regression value may be defined as the distance from a given keypoint to the boundary of the vessel lumen in each angular direction. In one embodiment, step 306 may include defining a target regression value for r angular directions. For example, for a given target distance value in r angular directions, the 2D image created in step 305 may be circularly rotated so that the column associated with the r angular direction associated with the given target regression value becomes the first column. For example, when predicting all r target values, each associated with a different column in the input image, the columns may be circularly rotated. If the image is rotated by r columns, the rth column becomes the first column, and the same model used to predict the target value for the first column can be applied to the target value for the rth column, which is the first column after the circular rotation.

[0049] In some embodiments, step 307 may include training a model (e.g., a CNN model) to predict the distance from a given keypoint to the boundary of the vessel lumen. In one embodiment, the trained model may predict a mapping from each of the 2D images created in step 305 to an associated target distance value. A loss function may be specified to minimize the mean squared error between the predicted distance and the target distance. As used herein, a loss function may specify the error between the predicted value and the target value and is the integral portion of the objective function that is optimized to learn appropriate model parameters. For example, the loss function may be the mean squared error, e.g., the mean square of the difference between the predicted value and the target value.

[0050] 3B is a block diagram of an exemplary method 310 for a testing phase that may provide sub-voxel segmentation of a patient's blood vessels (e.g., coronary arteries), according to one embodiment. In some embodiments, step 311 may include receiving image data of the patient's coronary arteries in an electronic storage medium (e.g., a hard drive, a network drive, a cloud drive, a mobile phone, a tablet, a database, etc.).

[0051] In some embodiments, step 312 may include receiving a prediction of the vessel centerline, for example, using a centerline detection algorithm. In one embodiment, step 312 may include converting the received image or image data to CPR. This conversion may allow the vessel segmentation process to be simplified. For example, a set of planes (e.g., frames) may be extracted along the centerline of the vessel lumen (e.g., orthogonal to the centerline) to construct a 3D volume (e.g., CPR). In one embodiment, the 3D volume may include the CPR, where a coordinate system frame of reference defines two dimensions and the length of the centerline defines the third dimension. Transformation parameters (e.g., translation, scaling, rotation) may be stored.

[0052] In some embodiments, step 313 may include defining points on the centerline of the vessel as key points.

[0053] In some embodiments, step 314 may include defining a mapping of image coordinates within the patient's image to Euclidean space for each of the frames defined in step 312. For example, this mapping may be defined using polar coordinate sampling within the frame. This step may be similar to one or more steps of method 300.

[0054] In some embodiments, step 315 may include determining CPR intensity values ​​at a set of angular directions around the keypoints defined in step 313. The determined CPR intensity values ​​may be arranged such that radial and angular coordinates are mapped to a two-dimensional (2D) image. The 2D image may be circularly rotated such that the column associated with the r angular direction associated with a given target distance value is the first column. Additionally, step 315 may include creating a circularly rotated (in the r angular directions) version of the 2D image.

[0055] In some embodiments, step 316 may include predicting segmentation boundaries of the patient's coronary arteries using the model trained in method 300. In one embodiment, step 316 may include predicting distances associated with each first column of the rotated images created in step 315, and thus predicting landmarks for the boundaries of the CPR representation. In one embodiment, step 316 may include generating an anatomical model of the patient's imaged coronary arteries. The anatomical model may include a final lumen segmentation with sub-pixel or sub-voxel accuracy. For example, step 317 may include transforming the predicted landmark(s) from the CPR representation back to the original 3D image space. The orientation and position of each frame along the centerline may be determined from the CPR representation. For example, this orientation and position may be determined and stored in step 312. In one embodiment, 3D points can be calculated from the CPR and any 3D surface reconstruction method (e.g., Poisson surface reconstruction) can be applied to the point cloud of landmark(s) to construct an anatomical model or final lumen segmentation of the patient's coronary arteries.

[0056] In some embodiments, step 317 may include outputting the anatomical model and / or the complete segmentation boundary of the vessel to an electronic storage medium (e.g., a hard drive, a network drive, a cloud drive, a mobile phone, a tablet, a database, etc.) and / or a display.

[0057] Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the invention being indicated by the appended claims.

Claims

1. 1. A computer-implemented method for performing anatomical structure segmentation of input image data, the method comprising: receiving the input image data from a medical imaging device including one or more images of a patient's anatomy, the patient's anatomy including a blood vessel; obtaining annotations of the anatomical structure, the annotations including a vessel lumen boundary comprising a boundary of the anatomical structure; determining one or more key points in the input image data based on the annotations, the one or more key points being points on a centerline of the anatomical structure; determining a respective distance between each keypoint and the boundary of the anatomical structure at each of a plurality of angular directions at equally spaced angles around each keypoint by calculating the distance from each keypoint to the boundary in polar form with each keypoint at its center; determining respective intensities of the input image data comprising image elements at each of the plurality of angular directions around each keypoint, the respective intensities being sampled at each of the plurality of angular directions from each keypoint, the image elements being pixels or voxels; training a convolutional neural network (CNN) using the respective intensities and the respective distances of each keypoint as inputs to predict sub-pixel or sub-voxel locations of the boundary of the anatomical structure, the sub-pixel or sub-voxel locations being finer than the image elements of the input image data, said training including training the CNN to regress continuous distance values ​​by using the respective distances as target values ​​and the respective intensities as input data, the continuous distance values ​​enabling prediction of the boundary at the sub-pixel or sub-voxel locations; generating sub-pixel or sub-voxel boundaries of the anatomical structures using the trained CNN; and A method comprising:

2. The method of claim 1 , wherein the generated sub-pixel or sub-voxel boundaries of the anatomical structure are continuous boundaries.

3. 2. The method of claim 1, wherein the input image data comprises a plurality of consecutive frames orthogonal to the centerline of the anatomical structure, each frame of the plurality of consecutive frames being in a plane orthogonal to the centerline of the anatomical structure.

4. 4. The method of claim 3, wherein generating the sub-pixel or sub-voxel boundary of the anatomical structure comprises generating predicted boundary points of the anatomical structure in each frame of the plurality of consecutive frames.

5. 5. The method of claim 4, wherein generating the sub-pixel or sub-voxel boundaries of the anatomical structure further comprises reconstructing a continuous three-dimensional surface representation of the anatomical structure based on the respective boundary portions.

6. 1. A system for performing anatomical structure segmentation of input image data, comprising: at least one memory that stores instructions and a convolutional neural network (CNN); at least one processor operatively connected to said at least one memory; wherein the at least one processor is configured to execute the instructions to perform operations, the operations comprising: receiving the input image data from a medical imaging device including one or more images of a patient's anatomy, the patient's anatomy including a blood vessel; obtaining annotations of the anatomical structure, the annotations including a vessel lumen boundary comprising a boundary of the anatomical structure; determining one or more key points in the input image data based on the annotations, the one or more key points being points on a centerline of the anatomical structure; determining a respective distance between each keypoint and the boundary of the anatomical structure at each of a plurality of angular directions at equally spaced angles around each keypoint by calculating the distance from each keypoint to the boundary in polar form with each keypoint at its center; determining respective intensities of the input image data comprising image elements at each of the plurality of angular directions around each keypoint, the respective intensities being sampled at each of the plurality of angular directions from each keypoint, the image elements being pixels or voxels; training the CNN using the respective intensities and the respective distances of each keypoint as inputs to predict sub-pixel or sub-voxel locations of the boundary of the anatomical structure, the sub-pixel or sub-voxel locations being finer than the image elements of the input image data, said training including training the CNN to regress continuous distance values ​​by using the respective distances as target values ​​and the respective intensities as input data, the continuous distance values ​​enabling prediction of the boundary at the sub-pixel or sub-voxel locations; generating sub-pixel or sub-voxel boundaries of the anatomical structures using the trained CNN; and Including, the system.

7. The system of claim 6 , wherein the generated sub-pixel or sub-voxel boundaries of the anatomical structure are continuous boundaries.

8. 7. The system of claim 6, wherein the input image data comprises a plurality of consecutive frames orthogonal to the centerline of the anatomical structure, each frame of the plurality of consecutive frames being in a plane orthogonal to the centerline of the anatomical structure.

9. 9. The system of claim 8, wherein generating the sub-pixel or sub-voxel boundary of the anatomical structure comprises generating predicted boundary points of the anatomical structure in each frame of the plurality of consecutive frames.

10. 10. The system of claim 9, wherein generating the sub-pixel or sub-voxel boundaries of the anatomical structure further comprises reconstructing a continuous three-dimensional surface representation of the anatomical structure based on the respective boundary portions.

11. 1. A non-transitory computer-readable medium comprising instructions for performing anatomical structure segmentation of input image data, the instructions being executable by one or more processors to perform operations, the operations comprising: receiving the input image data from a medical imaging device including one or more images of a patient's anatomy, the patient's anatomy including a blood vessel; obtaining annotations of the anatomical structure, the annotations including a vessel lumen boundary comprising a boundary of the anatomical structure; determining one or more key points in the input image data based on the annotations, the one or more key points being points on a centerline of the anatomical structure; determining a respective distance between each keypoint and the boundary of the anatomical structure at each of a plurality of angular directions at equally spaced angles around each keypoint by calculating the distance from each keypoint to the boundary in polar form with each keypoint at its center; determining respective intensities of the input image data comprising image elements at each of the plurality of angular directions around each keypoint, the respective intensities being sampled at each of the plurality of angular directions from each keypoint, the image elements being pixels or voxels; training a convolutional neural network (CNN) using the respective intensities and the respective distances of each keypoint as inputs to predict sub-pixel or sub-voxel locations of the boundary of the anatomical structure, the sub-pixel or sub-voxel locations being finer than the image elements of the input image data, said training including training the CNN to regress continuous distance values ​​by using the respective distances as target values ​​and the respective intensities as input data, the continuous distance values ​​enabling prediction of the boundary at the sub-pixel or sub-voxel locations; generating sub-pixel or sub-voxel boundaries of the anatomical structures using the trained CNN; and 1. A non-transitory computer-readable medium, comprising:

12. The non-transitory computer-readable medium of claim 11 , wherein the generated sub-pixel or sub-voxel boundaries of the anatomical structure are continuous boundaries.

13. 12. The non-transitory computer-readable medium of claim 11, wherein the input image data comprises a plurality of consecutive frames orthogonal to the centerline of the anatomical structure, each frame of the plurality of consecutive frames being in a plane orthogonal to the centerline of the anatomical structure.

14. Generating the sub-pixel or sub-voxel boundaries of the anatomical structure comprises: generating predicted boundary points of the anatomical structure in each frame of the plurality of consecutive frames; reconstructing a continuous three-dimensional surface representation of the anatomical structure based on the respective boundary portions; 14. The non-transitory computer-readable medium of claim 13, comprising:

Images