System and method for segmentation of anatomical structures in image analysis

Abstract

To provide a system and method for segmenting anatomical structures in image analysis using a computer system. [Solution] One method includes receiving annotations of anatomical structures and multiple keypoints in one or more images, calculating the distance from the multiple keypoints to the boundary of the anatomical structure, training a model using the data from one or more images and the calculated distance to predict the boundary within the anatomical structure in images of the patient's biological structure, receiving images of the patient's biological structure including the anatomical structure, estimating the segmentation boundary within the anatomical structure in images of the patient's biological structure, and predicting the boundary location within the anatomical structure in images of the patient's biological structure by using the trained model to generate a regression of the distance from keypoints within the anatomical structure to the estimated boundary in images of the patient's biological structure.

Description

Technical Field

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 62 / 503,838, filed May 9, 2017, the entire disclosure of which is hereby incorporated by reference in its entirety.

[0002] Various embodiments of the present disclosure generally relate to medical imaging and related methods. In particular, certain embodiments of the present disclosure relate to systems and methods for segmentation of anatomical structures in image analysis.

Background Art

[0003] The problem of dividing an image into multiple segments generally occurs in computer vision and medical image analysis. Currently used methods automate this process using convolutional neural networks (CNNs), which are trained to predict class labels for each image element (e.g., pixel or voxel). A CNN typically includes multiple convolutional layers that pass an input (e.g., an image or a part of an image) through a set of learnable filters and a non-linear activation function. By using convolutional operations, a CNN becomes equivariant to translations. For example, a translated version of the input can result in a proportionally translated version of the predicted segmentation labels. A series of layers with convolutional operations of different strides allows a CNN to represent long-range interactions within an image from a local, short-range statistical perspective.

[0004] However, current CNN segmentation boundaries may be accurate down to the level of image elements (e.g., pixels or voxels). In many image applications, quantization errors can be introduced by placing segmentation boundaries at pixel or voxel locations. It may be known (e.g., a priori) that a target structure may not contain holes and exist as a single connected component. However, such assumptions may not be incorporated into CNNs, and as a result, the predicted labels may have false components and holes within the segmented object. Therefore, it is desirable to build a model such as a CNN that can achieve accurate segmentation at the sub-pixel or sub-voxel level and predict labels for a single connected component without holes or isolated structures.

[0005] This disclosure aims to overcome one or more of the above-mentioned problems or concerns. [Overview of the project] [Means for solving the problem]

[0006] In certain aspects of this disclosure, systems and methods for segmenting anatomical structures in image analysis are disclosed. One method for segmenting anatomical structures in image analysis includes receiving annotations and keypoints of an anatomical structure in one or more images; calculating distances from the keypoints to the boundaries of the anatomical structure; training a model using data from one or more images and the calculated distances to predict boundaries within the anatomical structure in images of a patient's biological structure; receiving images of a patient's biological structure containing the anatomical structure; estimating segmentation boundaries within the anatomical structure in images of the patient's biological structure; and predicting boundary locations within the anatomical structure in images of the patient's biological structure by using the trained model to generate regressions of distances from keypoints within the anatomical structure to the estimated boundaries in images of the patient's biological structure.

[0007] According to another embodiment, a system for segmentation of anatomical structures in image analysis is disclosed. The system includes a data storage device that stores instructions for segmentation of anatomical structures in image analysis, and a processor configured to execute instructions and perform a method that includes the steps of receiving annotations and a plurality of keypoints of an anatomical structure in one or more images, calculating the distance from the plurality of keypoints to the boundary of the anatomical structure, training a model using the data from one or more images and the calculated distance to predict the boundary within the anatomical structure in an image of the patient's biological structure, receiving an image of the patient's biological structure containing the anatomical structure, estimating the segmentation boundary within the anatomical structure in the image of the patient's biological structure, and predicting the boundary location within the anatomical structure in the image of the patient's biological structure by using the trained model to generate a regression of the distance from the keypoints within the anatomical structure to the estimated boundary in the image of the patient's biological structure.

[0008] In yet another embodiment, a non-temporary computer-readable medium for use on a computer system is provided, which includes computer-executable programming instructions for performing a method of segmenting anatomical structures in image analysis. The method includes receiving annotations of an anatomical structure and a plurality of keypoints in one or more images; calculating distances from the plurality of keypoints to the boundaries of the anatomical structure; training a model using data from one or more images and the calculated distances to predict boundaries within the anatomical structure in images of a patient's biological structure; receiving images of a patient's biological structure containing the anatomical structure; estimating segmentation boundaries within the anatomical structure in images of a patient's biological structure; and predicting boundary locations within the anatomical structure in images of a patient's biological structure by using the trained model to generate regressions of distances from keypoints within the anatomical structure to the estimated boundaries in images of a patient's biological structure.

[0009] Further objectives and advantages of the disclosed embodiments are partially stated in the following description, partially evident from the description, or can be learned by practicing the disclosed embodiments. The objectives and advantages of the disclosed embodiments will be realized and achieved by the elements and combinations specifically indicated in the appended claims.

[0010] Please understand that the above summary and the following detailed explanation are examples only and are for illustrative purposes, and do not limit the scope of the disclosed embodiments claimed in the patent application. The present invention provides, for example, the following items: (Item 1) A computer-based method for segmenting anatomical structures in image analysis, Receiving annotations of anatomical structures and multiple key points in one or more images. Calculating the distance from multiple keypoints to the boundary of the anatomical structure, To train a model using data from one or more images and the calculated distances in order to predict the boundaries within the anatomical structures in images of a patient's biological structures, Receiving the image of the patient's biological structure, including the anatomical structure. To estimate the segmentation boundaries within the anatomical structure in the image of the patient's biological structure, and Using the trained model, predict the boundary position within the anatomical structure in the image of the patient's biological structure by generating a regression of the estimated distance from keypoints within the anatomical structure in the image of the patient's biological structure to the estimated boundary. The computer implementation method, including the above. (Item 2) The computer implementation method described in item 1, wherein the annotation of the anatomical structure is in the form of a mesh, voxel, implicit surface representation, or point cloud. (Item 3) The computer-aided method according to item 1, wherein the location of the received keypoint within the anatomical structure is known. (Item 4) The computer-aided method according to item 1, further comprising fitting a shape model to the anatomical structure and determining the positions of a plurality of key points. (Item 5) The computer implementation method described in item 1, wherein the image coordinates are continuous. (Item 6) The computer implementation method according to item 1, wherein the mapping is performed from multiple polar dimensions, or from a polar form of one dimension and a linear form of one or more additional dimensions. (Item 7) The computer implementation method according to item 1, further comprising determining the image intensity along the ray associated with the calculated distance before training the model, wherein the ray associated with the distance is at a fixed position in the Euclidean space. (Item 8) The computer implementation method according to item 1, wherein predicting the boundary position includes predicting an indirect representation of the distance from key points within the anatomical structure in the image of the patient's biological structure to the estimated boundary. (Item 9) A computer-aided method according to item 1, wherein estimating the segmentation boundary of the anatomical structure in the image of the patient's biological structure includes obtaining a set of keypoints within the anatomical structure. (Item 10) The computer-aided method according to item 1, wherein the predicted boundary location within the anatomical structure in the image of the patient's biological structure is the exact boundary location of the subvoxel. (Item 11) The computer implementation method according to item 1, further comprising constructing a three-dimensional surface based on the predicted boundary positions. (Item 12) The computer implementation method according to item 1, further comprising outputting the predicted boundary position to an electronic storage medium. (Item 13) The computer-assisted method according to item 1, wherein the anatomical structure includes blood vessels, and the patient's biological structure includes blood vessels in the patient's vascular structure. (Item 14) The computer-aided method described in item 1, wherein the annotations for the anatomical structures include vascular lumen boundaries, vascular lumen centerlines, vascular lumen surfaces, or combinations thereof. (Item 15) A computer implementation of item 1, further comprising defining a mapping from a plurality of image coordinates in one or more images to Euclidean space, wherein the plurality of image coordinates lie in rays in the Euclidean space, one of the plurality of keypoints lies on each of the rays, and calculating the distances includes calculating the distance from the plurality of keypoints on each of the rays to the boundary of the anatomical structure. (Item 16) The computer implementation method described in item 1, wherein the generated regression is a continuous value. (Item 17) A system for segmenting anatomical structures in image analysis, A data storage device that stores instructions for segmenting anatomical structures, Execute the aforementioned instruction, Receiving annotations of anatomical structures and multiple key points in one or more images. Calculating the distance from multiple keypoints to the boundary of the anatomical structure, To train a model using data from one or more images and the calculated distances in order to predict the boundaries within the anatomical structures in images of a patient's biological structures, Receiving the image of the patient's biological structure, including the anatomical structure. To estimate the segmentation boundaries within the anatomical structure in the image of the patient's biological structure, and Predicting the boundary position within the anatomical structure in the image of the patient's biological structure by generating a regression of the distance from the keypoints within the anatomical structure in the image of the patient's biological structure to the estimated boundary using the trained model A processor configured to execute a method including, and a system comprising the same (Item 18) The system according to item 17, wherein the anatomical structure includes blood vessels and the patient's biological structure includes blood vessels of the patient's vasculature (Item 19) The system according to item 17, wherein the predicted boundary position within the anatomical structure in the image of the patient's biological structure is the exact boundary position of a sub-voxel (Item 20) A non-transitory computer-readable medium for use on a computer system including computer-executable programming instructions for implementing a method of segmenting an anatomical structure in image analysis, the method comprising Receiving annotations of anatomical structures and a plurality of keypoints in one or more images Calculating the distance from the plurality of keypoints to the boundary of the anatomical structure Training a model using the data of one or more of the images and the calculated distance to predict a boundary within the anatomical structure in an image of a patient's biological structure Receiving the image of the patient's biological structure including the anatomical structure Estimating a segmentation boundary within the anatomical structure in the image of the patient's biological structure, and Predicting the boundary position within the anatomical structure in the image of the patient's biological structure by generating a regression of the distance from the keypoints within the anatomical structure in the image of the patient's biological structure to the estimated boundary using the trained model, the non-transitory computer-readable medium

[0011] The attached drawings are incorporated into and constitute part of this specification, illustrating various exemplary embodiments and are useful in illustrating the principles of the embodiments disclosed in conjunction with this description. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram of an exemplary system and network for anatomical structure segmentation in image analysis, according to an exemplary embodiment of the present disclosure. [Figure 2] Figures 2A and 2B are flowcharts illustrating exemplary methods for segmenting anatomical structures in image analysis according to exemplary embodiments of the present disclosure. [Figure 3] Figures 3A and 3B are flowcharts of exemplary embodiments of the method shown in Figures 2A and 2B, applied to coronary artery segmentation, according to exemplary embodiments of the present disclosure. [Modes for carrying out the invention]

[0013] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings illustrating the present disclosure. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts.

[0014] As described above, the accuracy of segmentation boundaries determined by current methods may be limited to image elements, such as pixels or voxels. In such cases, errors can be introduced by locating the segmentation boundaries at the voxel positions. In some cases, current prediction models may not take into account several assumptions, such as the target structure not containing holes or isolated structures. Therefore, it is desirable to construct a model that can predict segmentation boundaries with sub-pixel or sub-voxel accuracy and / or ensure that important assumptions that should be incorporated into the model are met.

[0015] This disclosure aims to enable accurate prediction of segmentation boundary locations. In one embodiment, this disclosure may include both a training phase and a test (or use) phase for estimating segmentation boundaries. One or more parameters of the learning system for building the trained model may be optimized during the training phase. In the test phase, the trained model can be used to segment invisible or visible images.

[0016] For example, the disclosed systems and methods can be applied to segment anatomical structures in received images (may be multiple) of a patient under consideration and to determine the boundaries of the structures at the subpixel or subvoxel level. As used herein, the boundaries of a structure may include the boundaries of segments of the structure. In one embodiment, the training phase may include building a model for predicting the distance from keypoints within the structure to the boundaries of the structure or its segments. For example, the training phase may involve receiving multiple keypoints along with their known locations within the structure and calculating the distance from these keypoints to the boundaries of the structure of their segments (e.g., based on their known locations). A model (e.g., a CNN model) can then be trained based on the keypoint locations, the calculated distances, and / or data in the received images. The trained model can perform regression analysis on sample distances or predict an indirect representation of sample distances. Regressions from the trained model can be continuous values, and thus, based on the regressed distances, it is possible to predict boundary locations with subpixel or subvoxel precision.

[0017] In one embodiment, the test phase may include receiving images of the patient's biological structure. The patient may be a target patient, for example, a patient who wishes to undergo a diagnostic examination. The test phase may estimate the boundaries of the target structure based on one or more images of the patient's biological structure and predict the boundary location by performing a regression analysis of the distance from keypoints in the target structure to the estimated boundaries using a model built in the training phase.

[0018] As used herein, the term “exemplary” is used to mean “example” and not “ideal.” Although this exemplary embodiment is written in the context of medical image analysis, this disclosure is equally applicable to any non-medical image analysis or computer vision assessment.

[0019] Next, referring to the drawings, Figure 1 shows a block diagram of an exemplary environment of a system and network for anatomical structure segmentation in image analysis. Specifically, Figure 1 shows multiple physicians 102 and third-party providers 104, each of which may be connected to an electronic network 100, such as the Internet, via one or more computers, servers, and / or handheld mobile devices. Physicians 102 and / or third-party providers 104 may create or otherwise acquire images of the cardiac, vascular, and / or organ systems of one or more patients. Physicians 102 and / or third-party providers 104 may also acquire any combination of patient-specific information, such as age, medical history, blood pressure, and blood viscosity. Physicians 102 and / or third-party providers 104 may transmit cardiac / vascular / organ images and / or patient-specific information to a server system 106 via the electronic network 100. The server system 106 may include a storage device for storing images and data received from physicians 102 and / or third-party providers 104. The server system 106 may also include a processing unit for processing images and data stored in a storage device. Alternatively or additionally, the anatomical structure segmentation of the Disclosure (or any part of the system and method of the Disclosure) may be performed on a local processing unit (e.g., a laptop computer) in the absence of an external server or network.

[0020] Figures 2A and 2B describe exemplary methods for performing anatomical structure segmentation using a learning system. Figures 3A and 3B cover specific embodiments or applications of the methods described in Figures 2A and 2B. As an example, Figures 3A and 3B describe an embodiment of vascular segmentation using the learning system. All of these methods can be performed by a server system 106 based on information, images, and data received from a physician 102 and / or a third-party provider 104 via an electronic network 100.

[0021] Figures 2A and 2B describe an exemplary method for segmenting anatomical structures in image analysis. In one embodiment, this segmentation of anatomical structures may include two stages: a training stage and a test stage. The training stage may include training a learning system (e.g., a deep learning system) to predict the boundary location of a target structure or its segment with sub-pixel or sub-voxel accuracy. The test stage may include predicting the boundary location of a target structure or its segment in a newly received image.

[0022] Figure 2A is a flowchart of an exemplary training stage of Method 200 for training a learning system (e.g., a deep learning system) to predict boundary locations, according to various embodiments. Method 200 can provide a basis for the test stage of Method 210 in Figure 2B for anatomical structural segmentation of an imaged target structure of a particular patient. Method 200 may include one or more of the steps 201-207 shown in Figure 2A. In some embodiments, Method 200 may include repeating one or more of the steps 201-207, for example, repeating steps 201-207 one or more times.

[0023] In some embodiments, step 201 of method 200 may include receiving one or more images and / or image data in an electronic storage medium (e.g., a hard drive, network drive, cloud drive, mobile phone, tablet, database, etc.). In a medical context, these images may be provided by medical imaging devices such as computed tomography (CT), positron emission tomography (PET), single-photon emission computed tomography (SPECT), magnetic resonance imaging (MRI), microscope, ultrasound, (multiview) angiography, etc. In one embodiment, multiple images may be used for a single patient. In further embodiments, the images may include the biological structure of a patient. In other embodiments, the images may be of a number of individuals with similar anatomical features or of a number of individuals with different anatomical features. In non-medical contexts, these images may be from any input source, e.g., a camera, satellite, radar, lidar, sonar, telescope, microscope, etc. In the following disclosure, the images received in step 201 may be referred to as “training images”.

[0024] In some embodiments, step 202 may include receiving annotations for one or more target structures in one or more training images. In some cases, one or more training images may include a target structure, e.g., an anatomical structure of a patient. In one example, all training images may include a target structure. In one embodiment, all training images may be annotated. This type of embodiment is sometimes called “supervised learning”. In another embodiment, only a portion of the training images may be annotated. This type of scenario is sometimes called “semi-supervised learning”. In one embodiment, the target structure may include a blood vessel or tissue of a patient. In such a case, the annotation(s) may include labels such as the name of the blood vessel (e.g., right coronary artery (RCA), left anterior descending artery (LAD), left circumflex artery (LCX), etc.), the target of the blood vessel (e.g., location of the aortic valve, location of a foramen, bifurcation, etc.), the estimated location of the blood vessel, and markings (e.g., portions noted as being ambiguous or having unclear or indistinct boundaries). In some embodiments, annotations may be in various forms, including but not limited to meshes, voxels, implicit surface representations, or point clouds.

[0025] In some embodiments, step 203 may include receiving a number of keypoints within the target structure. The locations of the keypoints within the target structure may be known (for example, based on annotations to the target structure and / or data in the received image). For example, it may be known that one or more keypoints are inside the target structure, on the boundary of the target structure, or near the boundary of the target structure. In some embodiments, the boundary locations of the target structure (e.g., approximate boundary locations) and / or the locations of the keypoints can be determined and / or keypoints with known locations within the target structure can be obtained by fitting a shape model to the target structure.

[0026] In some embodiments, step 204 may include defining a mapping from image coordinates in the training image to Euclidean space. The image coordinates may be continuous. The image coordinates may be intersections of 3D object structures and / or rays. The image coordinates may be within rays in Euclidean space. One of the keypoints received in step 203 may be located on each of the rays.

[0027] In one embodiment, it may be known that a given keypoint is located inside the structure of interest. In such a scenario, step 204 may include formulating the mapping in polar form with the given keypoint at the center. In this setting, image coordinates can be selected along isometric isometric rays emanating from the keypoint.

[0028] In an alternative scenario, the mapping in step 204 may involve mapping from multiple polar dimensions to Euclidean space. For example, it is possible to sample from a spherical coordinate system and parameterize the object structure (e.g., a three-dimensional (3D) object structure). In this embodiment, two rotational dimensions can be mapped to three Euclidean dimensions given by the distance along the ray and two dimensions associated with the two rotational dimensions.

[0029] In an alternative scenario, the mapping in step 204 may include mapping from a one-dimensional polar form and a linear form of one or more additional dimensions to Euclidean space. For example, a tubular structure can be represented by a series of marked points on a closed curve. In this case, the closed curve can be represented in polar form, while it can be linear in the direction along the tube. Thus, this mapping ultimately arrives at a state in three-dimensional Euclidean space, where the first dimension corresponds to the distance along the sampled rays, the second dimension corresponds to the rotational dimension, and the third dimension corresponds to the linear dimension along the tube.

[0030] In one embodiment, a set of keypoints close to the image surface can be received in step 203. For example, the set of keypoints may be arranged along a set of closed curves on a given 3D mesh or 3D implicit representation. In this case, step 204 may include defining a set of isometric rays, each containing a keypoint and oriented perpendicular to the 3D mesh, and further refining the boundaries of the target structure. When mapped to Euclidean space in step 204, the distances along the defined rays may represent one of the Euclidean dimensions, while the keypoints on the closed curves may represent a second dimension.

[0031] In some embodiments, step 205 may include calculating the distance from a keypoint on each of the rays mapped in step 204 to the boundary of the target structure. This calculated distance can be the target value of the learning system.

[0032] In some embodiments, step 206 may include determining the image intensity along the ray defined in step 204 for each target value. The defined ray is associated with a keypoint on the ray and therefore with the target value. Furthermore, step 206 may include ensuring that the ray associated with the target value is at a fixed position. For example, a first ray associated with a first target value may be at a fixed first position. Other rays associated with other target values ​​may have coordinates with respect to the first ray, and therefore the positions of these other rays can also be fixed based on the position of the first ray. In one embodiment, the inputs for each of the target value distances may be cyclic transformations of each other.

[0033] In some embodiments, step 207 may include training a model (e.g., a CNN model) to predict the segmentation boundary location of the target structure in a newly received image. For example, the model may be trained to predict the distance from a keypoint to the segmentation boundary of the target structure. In one embodiment, the model may be trained to perform a regression analysis of the distance. This regression value may be continuous so that the boundary location can be predicted with sub-pixel or sub-voxel accuracy. In certain embodiments, the model may be trained to predict an indirect representation of the distance. For example, the model may quantize a ray into several small bins and / or predict bins corresponding to the distance.

[0034] Figure 2B is a block diagram of an exemplary test (or use) stage of Method 210 for predicting segmentation boundaries of a target structure in a specific patient image, according to an exemplary embodiment of the present disclosure. Method 210 may include one or more of steps 211-215. In one embodiment, the segmentation boundaries may be predicted using a trained model (for example, provided by Method 200).

[0035] In some embodiments, step 211 may include receiving images of one or more patients in an electronic storage medium (e.g., a hard drive, network drive, cloud drive, mobile phone, tablet, database, etc.). In one embodiment, the images may include medical images, for example, images provided by any medical imaging device, such as CT, MR, SPECT, PET, microscope, ultrasound, (multiview) angiography, etc. In one embodiment, the training images (e.g., of method 200) may include images acquired from one patient, and step 211 may also include receiving images of that one patient. Alternatively or additionally, step 211 may include receiving one or more images from non-medical imaging devices, such as cameras, satellites, radar, lidar, sonar, telescopes, microscopes, etc. In the following steps, the images received during step 211 may be referred to as “test images”.

[0036] In some embodiments, step 212 may include estimating the segmentation boundary of the target structure, or the location or boundary of another object in the test image (e.g., an object different from the target structure). An automated segmentation system (e.g., a centerline) can be initialized using the estimated boundary or the location or boundary of the other object. A set of keypoints can be obtained from this initial segment. In one embodiment, the set of keypoints may include keypoints inside the target structure.

[0037] In some embodiments, step 213 may include defining a mapping from image coordinates in a test image to Euclidean space. The image coordinates may be continuous. The image coordinates may be in rays in Euclidean space. Each ray that can contain keypoints may have a keypoint (e.g., one of the keypoints obtained in step 202). In one embodiment, the input to this mapping may be similar to the input to the mapping in step 204.

[0038] In some embodiments, step 214 may include predicting the boundary of the structure of interest using the model trained by method 200. In one embodiment, this prediction may include regression analysis of the distance from keypoints on the rays defined in step 213 to the estimated boundary. In some cases, this regression can be continuous, and thus the boundary can be predicted with sub-pixel or sub-voxel accuracy. In one embodiment, step 214 may further include obtaining a surface from the predicted boundary (e.g., boundary point cloud). This surface can be obtained using a surface reconstruction method such as Poisson surface reconstruction.

[0039] In some embodiments, step 215 may include outputting the predicted boundaries of the target structure (e.g., complete segmentation boundaries) to an electronic storage medium (e.g., a hard drive, network drive, cloud drive, mobile phone, tablet, database, etc.). Furthermore, step 215 may include displaying the output results on an interface.

[0040] Figures 3A and 3B illustrate specific embodiments or applications of the exemplary methods described in Figures 2A and 2B. For example, Figures 3A and 3B describe exemplary training and test phases for coronary artery segmentation in image analysis, respectively.

[0041] The accuracy of patient-specific segmentation of blood vessels, such as coronary arteries, can impact medical evaluations such as blood flow simulations and calculations of vascular geometric properties. If segmentation accuracy is insufficient, for example, limited to the level of image elements (e.g., pixels or voxels), medical evaluations may produce erroneous results. Figures 3A and 3B illustrate exemplary methods for segmenting coronary arteries with sub-pixel or sub-voxel accuracy. While the exemplary methods described herein use coronary arteries, the methods shown in Figures 3A and 3B can also be used for segmenting other types of blood vessels or non-vascular anatomical structures.

[0042] Figure 3A is a flowchart of an exemplary method 300 of training stages designed to provide a basis for sub-voxel segmentation of coronary arteries in various embodiments. Method 300 may include one or more of the steps 301-308 shown in Figure 3A. In some embodiments, Method 200 may include repeating one or more of the steps 301-308, for example, repeating steps 301-308 one or more times.

[0043] In some embodiments, step 301 may include receiving images of one or more coronary arteries in an electronic storage medium (e.g., a hard drive, network drive, cloud drive, mobile phone, tablet, database, etc.). These images may be from medical imaging devices such as CT, MR, SPECT, PET, ultrasound, or (multiview) angiography. These images may be referred to as “training images”.

[0044] In some embodiments, step 302 may include receiving annotations for coronary arteries in one or more training images. For example, the annotations may include vascular lumen boundaries and / or vascular lumen centerlines. In one embodiment, step 303 may include receiving or generating a geometric mesh of the coronary vessels represented in the received images. The geometric mesh can be specified as a set of vertices and edges. Alternatively, or additionally, step 303 may include receiving the centerlines of the coronary vessels. The centerlines can also be represented as a set of vertices that can be connected by edges.

[0045] In some embodiments, step 303 may include converting the training image data (e.g., geometric mesh, vertices, edges, centerlines, etc.) into a curved planar representation (CPR). This conversion may allow for simplification of the vascular segmentation process. For example, a set of planes (e.g., frames) can be extracted along (e.g., perpendicular to) a centerline to constitute a 3D volume. In one embodiment, the 3D volume may include a CPR where the reference coordinate system frame defines two dimensions and the length of the centerline defines a third dimension. In one embodiment, the curved planar representation may exclude degrees of freedom (e.g., curvature of the centerline) which may not be relevant to predicting one or more parameters of the coronary vessel. For example, the curvature of the centerline may be irrelevant to determining parameters related to the position of the lumen boundary of the coronary vessel.

[0046] In some embodiments, step 304 may include defining keypoints based on image data. For example, step 304 may include defining points on the midline of the coronary arteries as keypoints. Keypoints thus defined may be interpreted as being located inside the blood vessels. These keypoints do not necessarily have to be centrally located. Nevertheless, in some cases, these keypoints may be centrally located in each frame by structure.

[0047] In some embodiments, step 305 may include defining a mapping from image coordinates in the test image to Euclidean space for each frame. In one embodiment, this mapping can be defined using polar coordinate sampling within the frame. In certain embodiments, defining the mapping may include determining CPR intensity values ​​in a set of angular directions around keypoints defined in step 304. The determined CPR intensity values ​​may be arranged such that, for example, radial and angular coordinates are mapped to a two-dimensional (2D) image. For example, a discrete set of samples specified by the radial and angular components of a frame can be mapped to rows and columns of a 2D image pointing to the radial and angular components. Each row of CPR intensity values ​​may be defined as radial coordinates, and each column of CPR intensity values ​​may be defined as angular coordinates.

[0048] In some embodiments, step 306 may include defining a target regression value. This target regression value can be defined as the distance from a given keypoint to the boundary of the vascular lumen in each angular direction. In one embodiment, step 306 may include defining a target regression value in the r angular direction. For example, for a given target distance value in the r angular direction, the 2D image created in step 305 may be cyclically rotated such that the column associated with the r angular direction, which is associated with a given target regression value, becomes the first column. For example, when predicting all r target values, if each is associated with a different column in the input image, the columns may be cyclically rotated. If the image is rotated by r columns, the r-th column becomes the first column, and the same model used to predict the target value of the first column can be applied to the target value of the r-th column, which is in the first column after the cyclic rotation.

[0049] In some embodiments, step 307 may include training a model (e.g., a CNN model) for predicting the distance from a given keypoint to the boundary of a vascular lumen. In one embodiment, the trained model can predict a mapping from each of the 2D images created in step 305 to a corresponding target distance value. The loss function may be specified to minimize the mean squared error between the predicted distance and the target distance. As used herein, the loss function can specify the error between the predicted and target values ​​and is the integral portion of the objective function that is optimized to learn appropriate model parameters. For example, the loss function may be the mean squared error, e.g., the mean squared difference between the predicted and target values.

[0050] Figure 3B is a block diagram of an exemplary method 310 for a test stage that may provide subvoxel segmentation of a patient's blood vessels (e.g., coronary arteries) according to one embodiment. In some embodiments, step 311 may include receiving image data of the patient's coronary arteries in an electronic storage medium (e.g., a hard drive, network drive, cloud drive, mobile phone, tablet, database, etc.).

[0051] In some embodiments, step 312 may include receiving a prediction of the vascular centerline, for example, using a centerline detection algorithm. In one embodiment, step 312 may include converting the received image or image data into a CPR. This conversion may allow for simplification of the vascular segmentation process. For example, a set of planes (e.g., frames) can be extracted along the centerline of the vascular lumen (e.g., perpendicular to the centerline) to constitute a 3D volume (e.g., a CPR). In one embodiment, the 3D volume may include a CPR where the reference coordinate system frame defines two dimensions and the length of the centerline defines a third dimension. Conversion parameters (e.g., translation, scaling, rotation) can be saved.

[0052] In some embodiments, step 313 may include defining a point on the midline of a blood vessel as a keypoint.

[0053] In some embodiments, step 314 may include defining a mapping of image coordinates in the patient's image to Euclidean space for each of the frames defined in step 312. For example, this mapping can be defined using polar coordinate sampling within the frame. This step may be analogous to one or more steps of method 300.

[0054] In some embodiments, step 315 may include determining CPR intensity values ​​in a set of angular directions around keypoints defined in step 313. The determined CPR intensity values ​​may be arranged so that radial and angular coordinates are mapped to a two-dimensional (2D) image. The 2D image can be cyclically rotated such that the column associated with the r-angle direction associated with a given target distance value becomes the first column. Furthermore, step 315 may include creating a cyclically rotated (in the r-angle direction) version of the 2D image.

[0055] In some embodiments, step 316 may include predicting the segmentation boundaries of the patient's coronary arteries using a model trained in method 300. In one embodiment, step 316 may include predicting the distances associated with each first column of the rotational images created in step 315, and thus predicting the landmarks of the boundaries of the CPR representation. In one embodiment, step 316 may include generating an anatomical model of the patient's imaged coronary arteries. The anatomical model may include a final lumen segmentation with sub-pixel or sub-voxel precision. For example, step 317 may include converting the predicted landmark(s) from the CPR representation to the original 3D image space. The orientation and position of each frame along the centerline may be determined from the creation of the CPR. For example, this orientation and position may be determined and stored in step 312. In one embodiment, 3D points can be calculated from CPR, and any 3D surface reconstruction method (e.g., Poisson surface reconstruction) can be applied to the point cloud of the identified point(s) to construct an anatomical model or the final lumen segmentation of the patient's coronary arteries.

[0056] In some embodiments, step 317 may include outputting the complete segmentation boundaries of the anatomical model and / or blood vessels to an electronic storage medium (e.g., a hard drive, network drive, cloud drive, mobile phone, tablet, database, etc.) and / or display.

[0057] Other embodiments of the present invention will be apparent to those skilled in the art from the discussion herein and the practices of the present invention disclosed herein. This specification and the examples are intended to be illustrative only, and the true scope and spirit of the invention are shown in the appended claims.

Claims

[Claim 1] The invention described herein.

Images