How to detect landmarks in large numbers of medical images

Abstract

Detecting landmarks in medical images is an important step in developing various diagnostic methods. The present invention relates to a method for detecting landmarks in a volume of medical images, which can be selected from a computed tomography volume or a magnetic resonance volume. The present invention includes a residual U-Net architecture for generating a heat map describing the probability of finding a landmark that can be combined with a 3D spatial numerical differential transform (DSNT) to obtain the landmark coordinates. The present invention is evaluated on two datasets, one containing 157 coronary computed tomography (CCTA) scans and the other containing 77 computed tomography (CTA) scans. The proposed method significantly improves the accuracy of landmark detection, especially ostial landmark detection, compared to known methods.

Description

[Technical field]

[0001] The present invention relates to the detection of landmarks in medical images, such as volumes of computed tomography (CT) images and volumes of magnetic resonance (MR) images. The landmarks include, among others, coronary artery ostia. The detection is performed using a deep learning model that includes a neural network, specifically a residual U-Net architecture. The transformation of probabilities to coordinates is performed using a non-learning 3D differential space-based numeric transform (DSNT). [Background technology]

[0002] Automated ostium detection is used in various medical image analysis techniques. It is often combined with other tasks such as image segmentation and registration. The main challenges in automated ostium detection are the anatomical variations between patients and the image acquisition variations. The regression of landmark coordinates is performed globally and does not require local refinement, whereas other most common methods do [4, 17, 18, 26, 31]. In particular, we achieve a state-of-the-art (SOTA) performance with a median error of Euclidean distance of 1.12 mm for the left femoral head and 1.05 mm for the right femoral head.

[0003] The first methods that addressed the problem of detecting coronary ostia from coronary computed tomography angiography (CCTA) were based on discovering intensity patterns. Waechter et al.

[26] and Elattar et al. {4} used a prior aortic valve segmentation to perform intensity-based landmark detection on the surface of the aorta. Elattar et al. achieved a Euclidean error of 2.81 ± 2.07 mm. These methods were designed strictly to detect the coronary ostia and required a two-stage setup to perform the detection. The required two-stage setup was time-consuming, making these methods time-inefficient. Furthermore, the accuracy of these methods is limited by anatomical anomalies and anatomical differences between patients.

[0004] The introduction of machine learning has been beneficial in terms of both robustness and time consumption. Classical machine learning methods are divided into two types: classification-based [2, 3, 7, 8, 12, 13, 19, 25, 31] and regression-based [1, 3, 5, 6, 19, 22, 25]. Classification-based methods generate a probability that a given pixel / voxel is a landmark of interest. Lonasec et al. [8] proposed the use of probabilistic boosting trees for the application to coronary artery ostium detection and achieved a Euclidean error of 2.28 mm. Zheng et al.

[31] used global-to-local marginal space learning and obtained a Euclidean error of 2.11 ± 1.34 mm.

[0005] The regression-based methods are designed to output continuous values ​​that define the sub-resolution coordinates of the landmarks of interest. Al et al. [1] adopted a colonial walk-based approach to the problem of coronary ostium detection and achieved a Euclidean error of 2.01 ± 1.04 mm. In recent years, as the field of artificial neural networks has developed, many deep neural network (DNN)-based approaches to the landmark detection problem have been introduced. The first networks that addressed the problem of landmark detection employed a convolutional neural network (CNN) model with a fully connected head to regress landmark coordinates [9, 24]. Mostafa et al.

[14] proposed to use a U-Net architecture with a fully connected head

[21] to solve the problem of coronary ostium detection. They reported SDRs (successful detection rates) of 68.3% and 74.1%, respectively, at a threshold of 2 mm.

[0006] Another approach, using heatmap matching, was introduced in a subsequent study

[23] . In this approach, landmarks are represented as Gaussians centered on their coordinates. Models constructed from this approach generate a heatmap that matches the Gaussian landmark representation and are penalized by a mean squared error (MSE) loss. This landmark representation is similar to classification-based MLML (ML-machine learning) methods that predict for each pixel / voxel the probability that a landmark is present at that location. Current SOTA performance levels for landmark detection have been achieved by models based on pyramid / hourglass architectures [15, 28]. Heatmap matching has also been used in medical imaging by Payer et al.

[20] for landmark detection in x-ray images. Wolterink et al.

[27] used heatmap matching for coronary ostia detection, obtaining a Euclidean error of 1.8 ± 1.0 mm on a test set of 36 images.

[0007] Currently, the best performing methods for coronary ostium detection employ a two-stage hybrid approach. Noothout et al.

[18] developed an approach that performs slice-wise image analysis in 2018. The approach consists of a vector pointing from the center of each slice to the landmark of interest. This vector is regressed for each slice to predict the probability of the presence of a given landmark on a given slice. A two-headed CNN was implemented for this task, and the final landmark location is found by weighting the prediction vector by the predicted presence probability. The backbone of the CNN was responsible for finding the rough landmark location, which was then individually refined by subsequent networks of the same design as the main network. They reported Euclidean errors of 2.19 mm and 2.88 mm for the right and left coronary ostium, respectively, on a test set of 40 CCTA volumes. Nootout's approach was further improved by its authors and extended by Nootout et al.

[17] in 2020. The slice-wise approach was changed to a patch-wise approach, and the CCTA dataset was expanded to 672 volumes, with 200 volumes constituting the test set. With the larger dataset and greater anatomical variability, they achieved Euclidean errors of median (IQR-interquartile range) 1.45 (1.20) and 1.55 (0.97) mm for the right and left coronary arteries, respectively. However, such results were only achieved due to the use of subsequent local refinement networks. The lowest median (IQR) Euclidean errors achieved using only the global network were 2.90 (1.93) and 3.31 (2.23) mm for the right and left coronary arteries, respectively.

[0008] From this review of the prior art development, it has been determined that there is a need for a reliable and accurate method for quickly finding landmark coordinates, which the present invention addresses.

[0009] The following description is for better understanding of the principles and advantages of the invention described in the appended claims. Therefore, it is not meant to be limiting in any sense. In particular, the specific explanations for the meaning or function of, for example, equations used in the present invention cannot be understood as limiting or exhaustive. They are provided for ease of understanding and cannot replace the complete understanding provided by the general knowledge of a person skilled in the art in this field.

[0010] The present invention relates to the problem of landmark detection, and in particular to automatic stoma detection. The present invention learns supervised regression models of landmark coordinates, e.g., from computed tomography (CT) volumes and human labels. The term supervised has the typical definition as used in this application and refers to the use of examples to teach an algorithm.

[0011] We refer to the input volume as x, the human labels as y, and the model predictions as y. Within the scope of this disclosure, bold symbols used in formulas refer to vectors and non-bold symbols refer to scalars.

[0012] The present invention includes heatmap matching regression that can be combined with extraction of heatmap center coordinates by a Differential Spatial to Numerical Transform (DSNT) layer. The present invention can include a combination of Residual U-Net and 3D DSNT for landmark detection, especially for coronary artery ostium detection.

[0013] For ease of explanation, a residual block can be used synonymously with a block. For example, a coded residual block can also be called a coded block. A volume is the result of a medical scan. For example, a computed tomography volume is the result of a CT scan.

[0014] In one aspect, the invention relates to a method for localizing a landmark in a volume of a medical image. The method includes generating at least one heat map using a trained residual U-Net applied to a volume of a medical image. In the method, the medical image volume is comprised of a computed tomography volume or a magnetic resonance volume. The medical image volume is comprised of voxels. The method can be realized as a computer-implemented invention. The localization includes, but is not limited to, indicating a likely location on the heat map, providing coordinates of the landmark, providing estimated coordinates of the landmark, or providing a range of coordinates where the landmark is located. The range of coordinates can be a range of only one coordinate, a range of two or three coordinates.

[0015] The at least one heatmap according to the present invention is generated using a trained Residual U-Net applied to the volume of the medical image, each heatmap representing the probability of finding the localization of the at least one landmark in a voxel.

[0016] The residual U-Net of the present invention is composed of a contraction pass and an expansion pass, where the contraction pass is composed of a residual coding block for coding the volume of the medical image, and the expansion pass is composed of a residual decoding block for decoding the volume of the medical image coded by the contraction pass.

[0017] The method of the present invention provides faster and significantly improved accuracy of landmark localization compared to known methods, and furthermore, once trained, the method is a one-step method that is relevant in terms of application of the method in clinical practice.

[0018] In one embodiment, the method further includes transforming the at least one heatmap into the coordinates of at least one landmark using a 3D differentiable spatial-to-numerical transform, which performs better than a method that uses only residual U-Net.

[0019] In one embodiment, each said heatmap is normalized, which reduces the computational burden of the method.

[0020] In one embodiment, the volume of medical images is a volume of pre-processed medical images. Pre-processing can be different for training and for using the method of the present invention. For example, the volume of re-processed medical images is obtained using at least one of the steps of center cropping, clipping, and setting the value of selected voxels to a defined value, normalizing the value of voxels, compacting the volume of medical images, and growing. Pre-processing is performed to remove unnecessary information to make the method computationally less intensive.

[0021] The clipping and setting the values ​​of selected voxels to defined values ​​may include setting voxel values ​​less than -300 HU to -300 HU and setting voxel values ​​greater than 1100 HU to 1100 HU.

[0022] Normalizing the voxel values ​​may include setting voxel values ​​equal to -300 HU to 0, setting voxel values ​​equal to 1100 HU to 1, and linearly scaling values ​​between -300 HU and 1100 HU to corresponding values ​​on a scale from 0 to 1.

[0023] The downsizing step may include reducing the volume of the medical image volume by a factor of 3 to 4. In particular, the downsizing step may include reducing the volume of the medical image volume by a factor of 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8 or 3.9.

[0024] The augmentation process can include rotations of up to π / 6 radians and translations of up to 10 voxels in each dimension.

[0025] In one embodiment, the encoding step comprises downsampling a volume of the medical image, increasing the number of channels to generate a downsampled output, and sending the downsampled output to a corresponding encoded residual block and a corresponding decoded residual block.

[0026] In one embodiment, the downsampling step is performed by a factor of two.

[0027] In another embodiment, the decoding step comprises upsampling the input of the medical image volume to a residual decoding block to provide an upsampling with a reduced number of channels. The upsampling may be performed by a factor of 2.

[0028] In one embodiment, the training of the trained residual U-Net comprises at least one of the following: minimizing discrepancies between the at least one heatmap and a labeled heatmap comprising at least one label, the at least one label indicating a location of a landmark on the at least one heatmap; Minimizing the Euclidean distance between the localization on the at least one heatmap and the localization on the at least one heatmap containing the label.

[0029] In one embodiment, the training process of the learned residual U-Net includes both minimizing the discrepancy between the at least one heatmap and a labeled heatmap including at least one label (wherein the at least one label refers to the localization of a landmark on the at least one heatmap) and minimizing the Euclidean distance between the localization on the at least one heatmap and the localization on the heatmap including at least one label. Training that includes minimizing both can obtain the most accurate results.

[0030] In an embodiment, the probability of finding the landmark has a normal distribution.

[0031] In one embodiment, the minimization of the Euclidean distance comprises:

[0032]

number

[0033] and minimizing said differences includes:

[0034]

number

[0035] D js (·||·) is the Jensen-Shannon divergence.

[0036] In an embodiment, the training step of the trained residual U-Net includes a linear combination of the minimizing step of the dissimilarity and the minimizing step of the Euclidean distance.

[0037] In an embodiment, the at least one heatmap and the labelled heatmap are generated using medical images of the same volume.

[0038] In an embodiment, the 3D differentiable spatial-to-numeric transformation includes:

[0039]

number

[0040] where H = φ(H) and H is the heatmap H output from the residual U-Net, normalized by a normalization function, where "" F represents the Frobenius dot product of two matrices, selected from H, X, Y, and Z. X, Y, and Z are grids of size m x n x l defined as follows:

[0041]

number

[0042] The 3D DSNT outperforms known methods in landmark detection, especially in coronary artery landmark detection.

[0043] In an embodiment, each residual block has: Y l =h(x l )+F(x l ,W l ) X l+1 =f(y l ) where x l and x l+1 are the input and output of the l-th residual block, f(·) is the residual function, f(y l ) is the activation function, h(x l ) is the identity mapping function.

[0044] In an embodiment, the at least one landmark is selected from a list comprising:

[0045] Inflow and outflow fossa surrounding the ostioma of the left coronary artery, inflow and outflow fossa surrounding the ostioma of the right coronary artery, apex of the left ventricle, base / bottom of the left ventricle, 17 segments of the left ventricle according to the American Heart Association, junction of the left and right ventricles, origin of the left coronary artery (LCA), origin of the right coronary artery (RCA), left main coronary artery, origin of the first diagonal artery (DG1), origin of the second diagonal artery (DG2), origin of the Ramus Intermedius artery, origin of the first blunt marginal artery (OM1), origin of the second blunt marginal artery (OM2), origin of the RCA of the right coronary artery, origin of the AM of the acute side branch, origin of the PDA of the posterior descending branch, origin of the PL of the posterior branch of the left ventricle, origin of the superior vena cava Inlet to the right atrium, Inlet of the inferior vena cava to the right atrium, Inlet of the coronary sinus to the right atrium, Right adnexal / right adnexal entrance to the right atrium, Right adnexal / right adnexal apex, Inlet of the left upper / right lower and right upper / right lower pulmonary veins to the left atrium, Right adnexal / right adnexal entrance to the right atrium, Left adnexal / left adnexal apex, Origin of the pulmonary artery, Tricuspid valve (including leaflets: anterior, posterior and septum, apex of the right ventricle, Origin of the ascending aorta, Aortic valve with leaflets: right, left, posterior anterior, right anterior, right posterior, anterior coronary, left coronary, right coronary, non-coronary, Bicuspid valve with leaflets: anterior and posterior , Mitral valve: anterior and posterior, Anterior cerebral artery, Horizontal or anterior communicating segment, Vertical, posterior communicating segment or infraumbilical segment, Landmarks located at the beginning of the preumbilical, supraumbilical and postumbilical segments, Anterior communicating artery, Superior cranial artery, Anterior posterior cranial artery, Posterior posterior cranial artery, Intermediate cranial artery, Sphenoid-horizontal segment, Insular segment, Landmarks located at the beginning of the palatine segment, Cortical segment, Posterior cranial artery, Anterior commissural segment, Posterior commissural segment, Tetrageminal segment, Cortical segment, Posterior commissural artery, OPHTALMIC artery, Left and right choroidal arteries, Basilar artery, Anterior spinal artery, Brachiocephalic artery / trunk, Left and right subclavian arteries, Left and right common carotid arteries, Left Right internal carotid artery, left and right external carotid arteries, left and right vertebral arteries, left and right thoracic and cervical trunks, right and left internal thoracic arteries, right and left costocervical trunks, right and left suprascapular arteries, right and left transverse cervical arteries, right and left axillary arteries, right and left brachial arteries, right and left ulnar arteries, right and left radial arteries, left and right superior thyroid arteries, left and right lingual arteries, left and right facial arteries, left and right ascending pharyngeal arteries, left and right occipital arteries, left and right posterior auricular arteries, left and right superficial temporal arteries, left and right maxillary arteries, posterior intercostal arteries, subcostal arteries, upper pneumatic arteries, pericardial arteries, esophageal arteries, esophageal branches of the inferior thyroid artery (upper third), thoracic esophageal branches of the aorta (middle third), esophageal branches of the left gastric artery (lower third), inferior phrenic arteries, lumbar arteries, celiac arteries, upper central arteries, lower central arteries,Middle superior renal artery, renal artery, testicular artery / internal spermatic artery / ovarian artery, common iliac artery, internal iliac artery, external iliac artery, femoral artery, popliteal artery, anterior tibial artery, dorsalis pedis artery, posterior tibial artery, medial plantar artery, lateral plantar artery, v. superior vena cava, v. brachiocephalicae, v. subclavian vein, v. Ilugulares, v. inferior vena cava, vv. Iliacae. Communes, external iliac vein, internal iliac vein, vv. lateral body, vv. cardiac vein, great cardiac vein, minor cardiac vein, v. Cardiaca medium, v. Posterior Ventriculi Sinister, v. Obliqua Atril Sinister, vv. Minimae, vv. Greater cerebral veins, Inferior cerebral veins, v. Veins of the anterior septum pellucidum, v. Septi Pellucidi Posterior, v. Thalamostriata Superior, vv. Thalamus-striata Inferior, Thalamus-striatae Inferiores, v. Choroidea Superior, v. Choroidea Inferior, Greater cerebral veins, v. Superior cerebral veins, vv. Inferior cerebellum, Anterior cerebellum, Deep cerebellum, Superficial cerebellum, Part of the ... anastomosis, superior anastomosis, inferior anastomosis, base, anterior commissure, posterior commissure, cerebral venous circulation, posterior commissure, superior ventricle, vv. superior cerebral hemisphere, inferior cerebral hemisphere, vv. inferior cerebral hemisphere, vv. Diploicae, v. Diploica Frontails, v. Diploica Anterior, v. Diploica Temporalis, v. Diploica Anterior, v. Diploica Temporalis, occipitalis, vv. Emissariae, v. Emissaria Palitaris, v. Mastoid process, v. Emissaria Condillaris, occipitalis, oval venous plexus, internal jugular venous plexus, sublingual venous plexus, vv. meninges, vv. medial sinus, thenar venous sinus, superior sagittal sinus, inferior sagittal sinus, rectal venous sinus, transverse sinus, sinus Portal sinus, occipital sinus, sinus junction, cavernous sinus, intercavernous sinus, basilar plexus, superior palpebral sinus, lacrimal sinus, vv. ethmoid, anterior and posterior ethmoid, nasofrontalis muscle, infraophthalmic muscle, facial muscles, angular oris muscle, supraorbital muscle, orbicularis oris muscle, palatine plexus, palatine plexus, palatine plexus, palatine plexus, palatine plexus, palatine plexus, palatine plexus, palatine plexus, palatine plexus, palatine plexus, palatine plexus, uvula, vv. external nose, vv. lips,Deep palate, parotid gland, inferior magnus muscle, lateral palate, posterior mandible, mylohyoid ... transverse muscle, anterior tibial muscle, pterygoid plexus, sphenopalatine plexus, septal plexus, occipital plexus, suprascapular plexus, vv. media, vv. deep temporal, vv. upper alveolar, vv. masseter, inferior alveolar, vertebral, v. subglossal comitans n, superior turbinate, subfascia, deep palmar vein, radius, ulna, humerus, brachialis, coracoid, ulnar collateral, deep humerus, axilla, subclavian, calcaneus, phalangeal, fibula, interfibularis, peroneal muscle, peroneal fascia ... Antebrachi, v. intermediate basilicas, v. intermedia cephalica, superior vena cava, cerebral herpes, Angulus varicosus, v.thyroidea inferior, vv.thyroideae imae, v.vertebralis, deep cervical, v.Iugularis Ext, Vertebral basement membrane, Intervertebral membrane, Spinal cord, Intraspinal membrane, Extraspinal membrane, Radius, Dorsal foot, Plantar, Venous arc, Plantar aponeurosis, Plantar aponeurosis plexus ... Medial, Lateral cricoid of femur, Upper gluteus maximus, vv. Inferior gluteus, Dorsal phalanges, Dorsal metatarsus, Dorsal vein of pedis, Phalanges, Dorsal vein of pedis, Dorsal vein of pedis, Dorsal vein of pedis, Dorsal vein of pedis, Dorsal vein of pedis, Dorsal vein of pedis, Safena, Safena parva, Inferior vena cava, Inferior sternum, Lumbar bone, Hepatic bone, Renal bone, Superior renal bone, Testis, Ovary, Iliac commissure, Middle sacrum, Lateral ilium, Lateral ilium, Lateral sacrum, Upper renal bone, Testis, Ovary, Ilium, Middle sacrum, External ilium, Deep ilium, Inferior ilium, Internal ilium, Iliopsoas, Lateral sacrum, Superior gluteus maximus, Inner clitoris, Deep penis, Clitoris resp., Caruncle of penis (resp.),Vestibular caruncle (resp.) Vestibule of the vagina, posterior surface of the scrotum, posterior surface of the labia. Posterior labia, inferior rectum, dorsal surface of the penis, dorsal surface of the clitoris ... rectal venous plexus, penile venous plexus, prostatic venous plexus, uterine venous plexus, vaginal venous plexus, v. Portal venous plexus of the liver, superior mesenteric venous plexus, mesenteric-ileal venous plexus, ileal villous venous plexus, appendiceal venous plexus, Colic venous plexus, pancreatic venous plexus, dilated gastrostomy venous plexus, pancreaticoduodenal venous plexus, pancreatic venous plexus, splenic venous plexus, pancreatic venous plexus, stomach venous plexus, gastric fundus venous plexus, submesenteric venous plexus, large intestine venous plexus, upper rectum venous plexus, gastric fundus venous plexus, foregut venous plexus, upper and posterior pancreatic venous plexus, bladder venous plexus, umbilical venous plexus, large duodenal papilla / Vater's papilla, bladder duct to common bile duct Entrance, exit of the bladder duct from the gallbladder, junction of the right and left hepatic ducts, junction of the right anterior and right posterior hepatic ducts, carina, subdivision to right upper lobe bronchus, subdivision to right middle lobe bronchus, subdivision to right lower lobe bronchus, subdivision to left lower lobe bronchus, subdivision to right 1st segmental bronchus, subdivision to right upper lobe bronchus, subdivision to right middle lobe bronchus, subdivision to right lower lobe bronchus, subdivision to left upper lobe bronchus, subdivision to left lower lobe bronchus, subdivision to right 1st bronchiole, subdivision to right 2nd bronchiole, subdivision to right 3rd bronchiole, subdivision to right 4th bronchiole, subdivision to right 5th bronchiole, subdivision to right 6th bronchiole division, subdivision to right 7th bronchiole, subdivision to right 8th bronchiole, subdivision to right 9th bronchiole, subdivision to right 10th bronchiole, subdivision to right 1st bronchiole, subdivision to left 1st bronchiole, subdivision to left 2nd bronchiole, subdivision to left 3rd bronchiole, subdivision to left 4th bronchiole, subdivision to left 5th bronchiole, subdivision to left 6th branch bronchus, subdivision to left 7th branch bronchus, subdivision to left 8th branch bronchus, anterior nose, entry / drainage of right posterior ethmoid cells into right sphenoidal recess, entry / drainage of left posterior ethmoid cells into left sphenoidal recess, exit of right sphenoid sinus into right sphenoidal recess Entrance / drainage, entrance / drainage of right sphenoid recess to right superior turbinate, left sphenoid recess to left superior turbinate, right frontal recess to right frontal recess, left frontal recess to left frontal recess, right frontal recess to right middle turbinate, drainage / drainage of left frontal recess to left nasal septum, drainage / drainage of right frontal ethmoid cells to right turbinate complex, drainage / drainage of right frontal ethmoid cells to right turbinate complex, drainage / drainage of right right maxillary sinus to right turbinate complex, drainage / drainage of left maxillary sinus to left turbinate complex, drainage / drainage of right turbinate complex to right middle turbinate, drainage / drainage of left turbinate complex to left middle turbinate,Outlet / outlet of right nasolacrimal duct to right inferior turbinate, outlet / outlet of left nasolacrimal duct to left inferior turbinate, anterior nares / posterior nares / maxillary opening, uvula, pharyngeal tonsils, left and right auditory tube openings, tip of epiglottis, left and right piriform recesses / turbinates / nasal sinuses, posterior turbinate region, cricoid cartilage, interarticular fold, left and right epiglottic fold, left and right sphenoid cartilage, left and right maxillary fold, outlet of left and right laryngeal pouch, vocal cords, left and right vocal cords, first tracheal cartilage, kidney, renal papilla, lesser curvature, renal pyramid, renal pelvis, inlet ureter, abdominal aortic plexus, abducens nerve, accessory nerve, accessory sinus nerve, Alderman's nerve, innominate sinus nerve, tibial nerve, anterior interosseous nerve, anterior superior alveolar nerve, Auerbach's plexus, ear axillary nerve, axillary nerve, brachial plexus, buccal branch of facial nerve, buccal nerve, cardiac plexus, cavernous nerve, cavernous plexus, celiac ganglion, cervical branch of facial nerve, cervical plexus, chorda tympani nerve, ciliary ganglion, coccygeal nerve, cochlear nerve, common peroneal nerve, median nerve common palmodigital nerve, deep branch of radial nerve, deep peroneal nerve, deep palpebral nerve, deep temporal nerve, Broca's diagonal band, digital branch of facial nerve, dorsal branch of ulnar nerve, dorsal nerve of clitoris, dorsal nerve of penis, dorsal nerve of scapula, esophageal plexus, ethmoid nerve, external laryngeal nerve, external nasal nerve, facial nerve, femoral nerve, frontal nerve, gastric plexus, cerebellum ganglion, femoral genital nerve branch, femoral genital nerve, glossopharyngeal nerve, great auricular Nerve, greater occipital nerve, greater tarsal nerve, hepatic plexus, hypoglossal nerve, iliolumbar nerve, inferior alveolar nerve, inferior anal nerve, inferior cardiac nerve, inferior cervical ganglion, inferior gluteal nerve, inferior inferior gastric plexus, inferior mesenteric plexus, inferior turbinate nerve, inferior orbital nerve, inferior orbital plexus, inferior turbinate nerve, intercostal nerve, intercostal brachial nerve, intermediate cutaneous nerve, internal carotid plexus, internal laryngeal nerve, internal ganglion, jugular ganglion, lacrimal nerve, lateral cord, lateral cutaneous nerve of forearm, lateral cutaneous nerve of thigh, lateral cutaneous nerve of thigh, lateral plantar nerve of thigh, lateral pterygoid nerve of thigh, lesser occipital nerve, lingual nerve, long ciliary nerve, long root of ciliary ganglion, long thoracic nerve, subscapular nerve, lumbar nerve, lumbar plexus, lumbosplenic nerve, lumbosacral nerve , lumbosacral plexus, lumbosacral trunk, mandibular nerve, mandibular branch of facial nerve, masseteric nerve, maxillary nerve, medial cord, medial cutaneous nerve of arm, medial cutaneous nerve of forearm, medial cutaneous nerve, medial greater thoracic nerve, medial plantar nerve, medial pterygoid nerve, median nerve, Meissner's plexus, mental nerve, central nerve, middle cervical ganglion, median nerve, motor nerve, muscular branch of radial nerve, musculocutaneous nerve, myocutaneous nerve, nasociliary nerve, nasopalatine nerve, pterygoid nerve, vara nerve, quadriceps nerve, piriformis nerve, xiphoid nerve, subclavian nerve, nerve intermedius, spinous nerve, nodose ganglion, greater trochanteric nerve, oculomotor nerve, olfactory nerve, ophthalmic nerve, optic nerve, auricular ganglion, ovarian plexus, palatine nerve,Palmar branch of the median nerve, Palmar branch of the ulnar nerve, Pancreatic plexus, Patellar plexus, Pelvic splenic nerve, Perforating cutaneous nerve, Perineal branch of the posterior femoral cutaneous nerve, Pharyngeal branch of the vagus nerve, Pharyngeal branch of the glossopharyngeal nerve, Pharyngeal nerve, Pharyngeal plexus, Respiratory nerve, Respiratory plexus, Posterior auricular nerve, Posterior branch of the spinal nerve, Posterior cord, Posterior cutaneous nerve of the arm, Posterior cutaneous nerve of the arm, Posterior cutaneous nerve of the forearm, Posterior cutaneous nerve of the thigh, Posterior scrotal nerve, Posterior superior alveolar nerve, Proprioceptive palmar digital nerve of the median nerve, Anterior palpate, Glandular plexus (neurogenic), pterygopalatine ganglion, saphenous nerve, saphenous plexus, pulmonary branch of the vagus nerve, radial nerve, recurrent laryngeal nerve, renal plexus, sacral plexus, sacrosplenic nerve, saphenous nerve, sciatic nerve, semilunar ganglion, sensory nerve, short ciliary nerve, sphenopalatine nerve, splenic plexus, hyoid branch of the facial nerve, subcostal nerve, submandibular ganglion, suboccipital nerve, superior cardiac nerve, superior cervical ganglion, superior glossopharyngeal ganglion, superior vagal ganglion, superior gluteal nerve, superior subgastric plexus, superior labial nerve, superior laryngeal nerve, superior lateral cutaneous nerve of the upper arm, superior mesenteric plexus, superior rectal plexus, supraclavicular nerve, superior orbital nerve, Superior renal plexus, suprascapular nerve, middle supratympanic nerve, auricular nerve, sympathetic trunk, temporal branch of the facial nerve, third occipital nerve, thoracic aortic plexus, thoracic splenic nerve, thoracic abdominal nerve, thoracic dorsal nerve, tibial nerve, transverse cervical nerve, trigeminal nerve, auricular nerve, tympanic nerve, ulnar nerve, suprascapular nerve, utero-vaginal plexus, vagus nerve, abdominal oblique nerve, cerebellar plexus, vestibular nerve, vestibulocochlear nerve, zygomatic branch of the facial nerve, zygomatic nerve, zygomaticofacial nerve, zygomaticotemporal nerve, cecal foramen, optic canal, superior orbital fissure, foramen rotundum, foramen ovale, foramen spinosum, hiatus, carotid artery, cochlear foramen, jugular foramen, hypoglossal foramen, foramen magnum.

[0046] Another aspect of the invention pertains to a computer readable (storage) medium comprising instructions that, when executed by a computer, cause the computer to perform steps according to any embodiment of the invention.

[0047] Another aspect of the invention relates to a system for defining coordinates of landmarks in a volume of a medical image, the system comprising measurement means for acquiring a computed tomography or magnetic resonance volume of a human patient and a computer system adapted to carry out the steps of any embodiment of the method of the invention.

[0048] In one embodiment, the system according to the invention comprises a computer system adapted to carry out the steps of the method, including the embodiment with training. In this embodiment, the acquisition of coordinates can be performed more quickly. In particular, in this embodiment, the operator who collects CT or MR images can provide a label indicating a landmark for at least a part of the image, so that the method of the invention can learn with the part of the image and define coordinates for the remaining image. This also allows the method to be adjusted to a specific landmark, since the label has a specific landmark.

[0049] Further features and advantages of the present invention will become more apparent from the detailed description of the non-limiting embodiments and the accompanying figures. It should be understood that the embodiments shown in the specification and claims can be combined in any order and number to create new embodiments that form part of this disclosure, unless otherwise stated. This specification consists of numerous citations to prior art documents, especially scientific journals, the entire disclosures of which are part of this disclosure. [Brief description of the drawings]

[0050] The invention will now be described in more detail with reference to the accompanying drawings. [Figure 1] FIG. 1 illustrates an application of an embodiment of the method of the present invention involving a trained residual U-Net. [Diagram 2] FIG. 2 is a detailed diagram of an embodiment of the method of the present invention that includes a training step. [Diagram 3] FIG. 3 illustrates the architecture of a Residual U-Net according to one embodiment of the method of the present invention. [Figure 4] FIG. 4 shows the architecture of a coding block according to an embodiment of the invention. [Diagram 5] FIG. 5 shows the architecture of a decoding block according to an embodiment of the invention. [Figure 6] FIG. 6 shows more detailed steps of training the residual U-Net according to one embodiment of the method of the present invention. [Figure 7] Figure 7 shows example slices from two datasets with coronary ostium labels: the first row shows examples from the CCTA dataset, and the second row shows examples from the ImageTBAD CTA dataset. [Figure 8] FIG. 8 is an example 3D rendering from the CCTA dataset showing the localization of the left and right coronary artery periosteum. [Figure 9] FIG. 9 shows the successful detection rate (SDR) of landmark localization in experiments on the CCTA dataset and the ImageTBAD CTA dataset. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0051] The above description has focused on embodiments involving computed tomography volumes, although magnetic resonance (MR) image volumes may also be used in the embodiments, subject to adaptations of the embodiments.

[0052] At the core of one embodiment of the present invention is a deep learning model with 3D Differential Space Numerical Transformation (DSNT) enhanced with a Residual U-Net architecture 103. In other embodiments, the Residual U-Net architecture can be used without DSNT. To use the model, it must first be trained. Various embodiments of the method, including training and architecture details, are shown in Figures 2, 3, 4, 5, and 6 and described herein. In one embodiment, the output is the coronary periosteal landmark coordinates 104. Once the model 103 is trained, in one embodiment, it infers two coordinate predictions with sub-voxel accuracy for a given pre-processed input CT volume: one coordinate prediction for the left coronary ostium and one for the right coronary ostium. Each coordinate includes three numerical values, respectively for sagittal, coronal, and axial positions. The use of the present invention with the learned model is called the inference phase. The training of the model is called the training phase.

[0053] The present invention is described in detail for the landmarks of the coronary ostia. However, the present invention is applicable to other modalities and other landmarks. To this end, different data sets are used and, if necessary, pre-processing steps or procedures are required. A person skilled in the art will understand that different data sets require different data pre-processing. The principle of pre-processing remains the same, although it can be defined as obtaining a data set that is easy to use for a particular detection. To perform localization of a different number of points, Cout of the decoding block (310) must be set to a different number. The output (311) then contains the landmark predictions of said different number. A list of landmarks whose coordinates can be detected is given in this specification and in the appended claims.

[0054] FIG. 1 shows the application of the method adapted to the detection of coronary ostia in a CT volume 101 according to the invention consisting of a trained residual U-Net. A CT volume is a 3D guard made up of 3D pixels named voxels. Each voxel contains information of relative radiodensity. A voxel usually represents the average attenuation in Hounsfield units (HU) of the tissue to which it corresponds. Values ​​range from +3071 HU to -1024 HU on the Hounsfield scale. The volume is expected to be calibrated to water with attenuation of 0 HU. A typical volume is 512 voxels wide, 512 voxels high, and z number of voxels long. The number of voxels with length z depends on the number of slices. For example, the CTA volume of FIG. 7 shown in 730, 740 and 750 images has 330 slices and therefore contains a total of 512x512x330 voxels with a length of 330 voxels. For example, the CCTA volume in Figure 7 shown in slices 700, 710 and 720 is 0.3281X0.3281x0.4mm 3 The voxel size is

[0055] The input CT volume is normalized to remove unnecessary information and reduce computational load. In the pre-processing step, voxels in the volume are clipped to the range of -300HU to 1100HU. That is, all values ​​less than -300HU are set to -300HU, and all values ​​greater than 1100HU are set to 1100HU. That is, each voxel at -300HU is set to 0, each voxel at 1100HU is set to 1, and voxels with values ​​between -300HU and 1100HU are set to the corresponding value on the scale from 0 to 1. Other normalization techniques can be used. In the pre-processing step, the volume is downsized to retain the most information and to solve memory issues. In one embodiment, the volume is downsized to a spatial size of 144x144x144 voxels using trilinear interpolation. Other interpolation techniques can be used. In other embodiments, the input CT volume is downsized by a factor of 3-4, in particular by a factor of 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8 or 3.9. Despite the loss of detail and spatial precision of the data due to downsizing, the accuracy of the results obtained using the method according to the invention is not reduced and has even been shown to outperform known methods using local refinement.

[0056] Fig. 2 details the training stage of the model for a single input volume. The trained model used in the present invention does not require 207, 208, 209 and 210 from Fig. 2 and does not require increased CT volume.

[0057] The training stage consists of an input volume x 201. The same input can be used in the inference stage. In one embodiment, the training stage also comprises inputting the coordinates y of the two coronary ostia, the coordinates being provided by a trained radiologist (CT landmark labels 209). In the same embodiment, the training stage comprises providing the localizations y of the two coronary ostia (landmark coordinate predictions 206).

[0058] In general, the method combines matching regression of heatmaps with extraction of heatmap center coordinates (shown in step 210) using a Differential Spatial Numerical Transform (DSNT) layer shown in step 205.

[0059] Preprocessing and reinforcement for training Volumes can be pre-processed in the same way in the training and inference phases. In one embodiment, pre-processing in the training phase additionally includes a volume augmentation step as disclosed herein. The input volume is pre-processed and then augmented 202. Voxels can be clipped to the range of -300HU to 1100HU. In that embodiment, all values ​​less than -300HU are set to -300HU, and all values ​​greater than 1100HU are set to 1100HU. That is, each voxel at -300HU is set to 0, each voxel at 1100HU is set to 1, and HU values ​​between -300HU and 1100HU are set to corresponding values, for example on a scale from 0 to 1. Finally, the volume can be resized to a spatial size of 144x144x144 voxels using trilinear interpolation. Other spatial sizes and other interpolation techniques can also be used. Additionally, the input volume can be augmented with a rotation of up to π / 6 radians in each dimension and a translation of up to 10 voxels.

[0060] During the training and inference phases of the ImageTBAD CTA dataset, an additional pre-processing step in the form of center cropping can be applied to each volume. The center cropping can, as an example, include cropping 112 voxels from a volume of size 512x512xZ voxels on a slice along the Z axis to arrive at a volume of 288x288xZ. The center cropping removes unnecessary regions from the volume, leaving only the cardiac region.

[0061] Heatmap Generation In general, the unnormalized heatmap H-fΘ(x) is the output from a model fΘ that is parameterized by parameters (also called weights) Θ that can be learned using an input volume x. In step 203, the fΘ model that serves as the backbone feature extraction network in the present invention is a residual U-Net [10, 30] that adopts the classical U-Net architecture

[21] and combines it with a residual neural network. The residual U-Net of the present invention discovers a heatmap that can be called the features discovered by the residual U-Net. To avoid any misunderstanding, fΘ represents the residual U-Net. Θ is discovered during training and does not change in application (inference stage). Those skilled in the art know that there are many residual U-Net architectures, and the difference between them is the structure of the encoded residual block and the decoded residual block contained in the architecture. The structure of the residual block affects the application of a particular residual U-Net architecture. As a result, the specific structure of the residual U-Net architecture can be defined according to its purpose. Nevertheless, and in order to provide a complete disclosure of the present invention, details of embodiments of the present Residual U-Net architecture are set forth in the description and claims.

[0062] Figure 3 illustrates the residual U-Net architecture according to an embodiment of the present invention. It receives an input volume 301. The input 301 is passed to a series of coded residual blocks (302, 303, 304, 305) that successively reduce the dimensionality and resolution size of the volume until a final coded residual block 306 is reached. In Figure 3, for simplicity, encoder and decoder blocks are used instead of coded residual and decoded residual blocks, respectively.

[0063] In an embodiment, the coded residual block is in ) channel input volume and output (C out) channel volume. It also reduces the size of the volume according to the value of stride (s). If stride is equal to 2, for example, the output volume reduces its size by 2. If stride is equal to 1, the volume size is not reduced. Usually, all convolution kernels of the coding convolution layer of the coding residual block have dimensions 3x3x3, and k s = 3. As in the coded residual block 306, s=1 means that the kernel dimension of the coding convolution layer of the mapping function of the coding residual block is 1x1x1 in size. The coding residual block is also connected to the corresponding decoded residual block (307, 308, 309, 310) to share the information lost during the spatial size reduction. Sharing should be understood as making a copy of the information. In other words, each coding residual block connected to another coding residual block outputs the same information twice, one copy is forwarded to another coding residual block and one copy is forwarded to the corresponding decoded residual block.

[0064] Each volume contains dimensions X, Y, Z, and C. X, Y, and Z are called spatial or shape dimensions. C is the number of channels. Volumes in the architecture of the residual U-Net according to the embodiment are transferred as follows: An input volume 301 with one channel (grayscale Hounsfield scale) is sent to an encoding block 302, which produces a volume with 16 channels and half the size of the shape (stride=2). A copy of the signal follows encoding block 303 and decoding block 310. The encoding block 303 produces a 32-channel volume downsampled by a factor of 2 (stride=2). A copy of the signal follows encoding block 304 and decoding block 309. The encoding block 304 produces a 64-channel volume downsampled by a factor of 2 (stride=2). A copy of the signal follows encoding block 305 and decoding block 308. The encoding block 305 produces a 128-channel volume downsampled by a factor of 2 (stride=2). A copy of the signal follows the encoding block 306 and the decoding block 307. The encoding block 305 produces a volume with 128 channels and does not change the volume size (stride=1) in contrast to the other encoding blocks. The output from 306 is sent to the decoding block 307. The decoding block 307 concatenates the signals from the encoding blocks 305 and 306 and produces a volume with 384 channels as input. The decoding block 307 produces a volume with 64 channels upsampled by a factor of 2. The output from 307 is sent to the decoding block 308. The decoding block 308 concatenates the signals from the encoding blocks 304 and 307 and produces a volume with 128 channels at input. The decoding block 308 produces a volume with 32 channels upsampled by a factor of 2. The output from the decoding block 308 is sent to the decoding block 309. The decoding block 309 concatenates the signals from the encoding blocks 303 and 308 and produces a volume with 64 channels at input. The decoding block 309 produces a 16 channel, 2x upsampled volume.The output from the decoding block 309 is sent to a decoding block 310, which concatenates the signals from the encoding blocks 302 and 309 and generates a volume with 32 channels at its input. The decoding block 310 generates a two-channel upsampled volume by a factor of two. The final step of the architecture is the output volume 311 of the same spatial size as the input volume 301, but with two channels. In an embodiment, each channel corresponds to an estimated probability of landmark localization; one for the probability of left coronary ostium localization and one for the probability of right coronary ostium localization. In other embodiments, the number of channels, downsampling factor, upsampling factor and / or stride value may be different.

[0065] Each residual block, both for encoding and decoding, can be defined as follows: y l =h(x l )+F(x l ,W l ) x l+1 =f(y l ),(l) where x l and x l+1 are the input and output of the lth residual unit, F(·) is the residual function, f(y l ) is the activation function, h(x l ) is usually the identity mapping function. The residual function F(·) depends on the residual block structure. For each coding block it is defined as the product: 402, 403, 404, 405, 406, 407. For each decoded block it is defined as the product: 506, 507, 508.

[0066] FIG. 4 illustrates the architecture of a coded residual block according to an embodiment of the present invention. Each coded residual block is composed of coding layers. To simplify the diagram, the word "coding" is omitted. The volumes that enter a coded residual block can be called coded residual block inputs. A coded residual block is a block that receives a certain input (C in) channel number of input volume 401 and a certain output (C out ) channel volume 410. 402 is a first encoding convolutional layer in the residual function F(-) of the encoding residual block, and 405 is a second encoding convolutional layer. In 408, a mapping function h(xl)=conv(x l ,K,C in ,C out ,s) applies.

[0067] For the coded residual block according to the embodiment of the present invention, h(xl)=conv(x l ,K,C in ,C out ,s), defined as follows:

[0068]

number

[0069] where * is the 3D cross-correlation operator or its strided version. C in is the number of input channels, C out is the number of output channels. K is a set of learnable weights in the form of convolution kernels, and s is the stride value. The 3D cross-correlation is defined as:

[0070]

number

[0071] where K is the kernel, defined as a 3D matrix of size NxNxN, and I(x,y,z) is a voxel in the input volume I with coordinates x,y,z. The strided version of the 3D cross-correlation is defined as:

[0072]

number

[0073] where K is the kernel defined as a 3D matrix, I are the examples of the input volume, and s is the stride.

[0074] The first encoding convolutional layer 402 and the subsequent encoding convolutional layer 408 have a number of channels C in From C out whereas the second encoding convolutional layer 405 does not change the number of channels, and therefore in =C out The first encoding convolutional layer 402 and the subsequent encoding convolutional layer 408 use a strided 3D cross-correlation operator, while the second encoding convolutional layer 405 does not use striding (i.e., the stride is equal to 1). The first encoding convolutional layer 402 and the second encoding convolutional layer 405 use a kernel size K , which means that the kernel size is of dimensions 3x3x3. s The subsequent encoded convolutional layer 408 has a kernel size K = 3 for the encoded residual blocks 302, 303, 304, 305 and 1 for the encoded residual block 306. s has.

[0075] The encoding convolutional layers 402 and 405 are followed by encoding norm layers 403 and 406, respectively. Examples of encoding norm layers include batch normalization

[33] , instance normalization

[36] , layer normalization

[37] , group normalization

[35] , etc. In one embodiment, 3D batch normalization

[33] is used, which is defined as follows:

[0076]

number

[0077] where x is the input and y is the output of the coding norm layer. γ and β are learnable parameters, and ε is a small constant value added to the denominator for numerical stability.

[0078] The coding norm layers 403 and 406 are followed by coding activation layers that include activation functions 404 and 407, respectively. Examples of activation functions include tanh, rectified linear unit (ReLU), parametric rectified linear unit (PReLU), etc. In one embodiment, PReLU is used, which is defined as follows:

[0079]

number

[0080] where y i is the i-th element of the input y, and α is a learnable parameter.

[0081] The outputs from 407 and 408 are merged with an element-wise addition function defined as regular matrix addition 409, resulting in a matrix of shape C out This results in the output volume 410.

[0082] FIG. 5 illustrates the architecture of a decoded residual block in an embodiment of the present invention. Each decoded residual block is composed of a decoding layer. To simplify the diagram, the word "decoded" is omitted. The volume that enters the decoded residual block can be called the decoded residual block input. The decoded residual block receives a certain input (C in ) channel number of inputs 501 and outputs (C out ) outputs a volume with 510 channels. First, the input volume goes to 502, the first decoded convolutional layer, as defined for 402, 405, or 408. This decodes the number of channels to C in From C out to, which uses a non-strided 3D cross-correlation operator. The decoding convolutional layer 502 uses a constant kernel size k s= 1. 504 is a decoding norm layer, where we apply the 3D batch normalization defined in 403 or 406. 505 is a first decoding activation layer, where we apply the PReLU defined in 404 or 407. 506 is a second decoding convolutional layer in the residual function of the residual decoding block, defined similarly to 402, 405, 408. The number of channels C out The second decoding convolutional layer 506 uses a constant kernel size k s = 3. This is followed by a second decoding norm layer 507, here a 3D batch normalization as defined in 403 or 406. The second decoding norm layer 507 is followed by a second decoding activation layer 508, here a PReLU as defined in 404 or 407. The outputs from 505 and 508 are merged with an element-wise addition function defined as ordinary matrix addition 509, resulting in a matrix of shape C out For the decoded residual block, the mapping function is h(x l )=X l and is the connection between 505 and 509 in FIG.

[0083] The learnable parameters are the parameters that are changed during the training phase: the kernel weight K of the convolutional layer, the parameters γ and β of the batch normalization layer, and the parameter a of the activation layer.

[0084] Heatmap normalization function Figure 2 The normalization function φ(·) is L l This is a rectification operation following normalization. The 1-channel heatmap H output from the residual U-Net is normalized by the normalization function to generate a normalized 1-channel heatmap H. H is output to 208 and 205.

[0085] 3D DSNT The 3D Differentiable Spatial Numerical Transform (3D DSNT)

[16] 205 in Figure 2 is a fully differentiable and non-trainable layer. Due to this fact, as backpropagation passes through the 3D DSNT, numerical coordinates can be obtained during training and used as a learning reference.

[0086] We define the 3D extension of DSNT as follows:

[0087]

number

[0088] Here, H = φ(H), and H is the one-channel heatmap H output from the residual U-Net normalized by the normalization function (step 204). F denotes the Frobenius dot product of two matrices, selected from H, X, Y, and Z. H, X, Y, and Z are grids in the form of m x n x l matrices used to calculate the expected values ​​of the landmark coordinates, defined as follows:

[0089]

number

[0090] Each X, Y, and Z entry contains its own coordinates scaled so that the top-left corner of the volume is at (-l,-l,-l) and the bottom-right corner is at (l,l,l).

[0091] Loss function Coronary ostium localizations that have been manually labeled by a trained radiologist are provided in step 209. Conversion of the manually labeled coronary ostium localizations into a labeled heatmap occurs in step 207. This can be done by assuming a normal distribution of the coronary ostium localizations on the heatmap.

[0092] In 208, the minimization of the dissimilarity between H and the labeled heatmap is performed. This is done by using the Jensen-Shannon dissimilarity regularization L D Using H and a suitable target normal distribution N(y,σ 2 ) where σ is a hyperparameter that controls the spread of the desired distribution.

[0093] At 210, we minimize the Euclidean distance between the manually labeled coronary ostia locations and the coronary ostia locations provided by the 3D DSNT. The general Euclidean distance L between the network output y=DSNT(H) and the ground truth y is E It has been shown that having a network generate heatmaps with a normal (e.g., Gaussian) distribution of probabilities improves the overall quality of predictions

[16] .

[0094] The full loss is defined as a linear combination of the divergence loss function (208) and the regression loss function (210):

number

[0095] The Euclidean loss LE (available at 210) can be defined as:

[0096]

number

number

[11] .

[0097] Training Procedure FIG. 6 shows an exemplary training procedure of the method in more detail. Step 602 requires a training dataset, which may be a dataset requiring landmark prediction. The training dataset may be the same dataset requiring landmark detection. In an embodiment, a CCTA or CTA dataset is used. The training dataset is split into a training data split and a validation data split. The training dataset split is used to train the weights of the model, and the validation dataset split is used to validate the performance of the model after each epoch of training. An epoch includes one iteration over all volumes of the training data split.

[0098] For the CCTA dataset, we can use 3-fold cross-validation, which results in 104 volumes per fold for the training data split and 53 volumes per fold for the validation data split. For the Image TBAD dataset, we use 4-fold cross-validation, which results in 58 volumes per fold for the training data split and 19 volumes per fold for the validation data split. Each volume x in the dataset has a corresponding landmark label y.

[0099] In step 601, a set of training parameters (weights) is initialized and the training parameters are selected. There are two sets of parameters: learnable parameters and learning parameters (hyperparameters). The weights W of the deep learning model are composed of all the learnable parameters. That is, W is composed of the kernel weight K of the convolutional layer, the parameters γ and β of the batch normalization layer, and the parameter a of the PReLU activation layer. In one embodiment, the model weights are initialized as follows: The kernel weight K of the convolutional layer is initialized by sampling from Kaiming Uniform Distribution as described in

[32] . The parameters γ and β of the batch normalization layer are set to 1 and 0, respectively. The parameter α of the PReLU activation layer is initialized to 0.25. These weights are used as initialization weights in step 601 and are updated after each iteration of the learning phase.

[0100] The following learning parameters can be used: 1) Number of epochs: The maximum number of epochs to use in the learning phase. For example, it can be 1,000 epochs. 2) Value of training batch size: The number of training volumes in one iteration of training, usually limited by the GPU memory on which the training is performed. It can be set to four. That is, four volumes are concatenated and passed to the network in one training iteration. Other values ​​can also be used. 3) Learning rate: Controls the strength of the weight updates and is a scalar that is multiplied with the derivatives of the weights during learning. It can be set to 0.00025. 4) The hyperparameter to be selected can be the optimization algorithm. The Adam optimization algorithm defined in

[34] can be used as the hyperparameter. 5) Then, the hyperparameter is used as the evaluation metric. This hyperparameter is defined to monitor the model quality and is used for evaluation after each epoch. In one embodiment, the hyperparameter is defined as the localization accuracy in the validation data split. 6) Stopping criterion: The hyperparameter used for early stopping of the learning phase. In one embodiment, this is defined as reaching a maximum number of epochs, which may be 1,000. Other numbers may also be used. 7) Finally, the hyperparameters specific to our architecture are the parameter σ, which controls the spread of the distribution of localized labels in the dissimilarity loss, and the regularization rate parameter λ, which controls the strength of the dissimilarity loss effect. In one embodiment, the parameters are set to σ=3 and λ=1.

[0101] 3D DSNT augmented with residual U-Net (deep learning model) 604, 103 includes residual U-Net 203, heatmap regularization function 204, and 3D DNT 205. At each iteration, a data batch 603 is received from a training data split 602 consisting of a training CT input volume 201 and corresponding CT landmark labels 209. The deep learning model 604 uses the current values ​​of the weights in a forward propagation step 605 to infer predictions from the batch of data 603. These predictions are used in a loss calculation step 606 along with the corresponding data labels of the batch of data 603. The loss calculation step 606 is composed of the dissimilarity loss function from step 208 and the regression loss function from step 210. As shown in step 207, in order to calculate the dissimilarity loss 208, the CT landmark labels 209 need to be converted to a heatmap.

[0102] Once the loss is calculated, after each epoch our model infers a prediction for the entire validation split of data. A decision to continue learning is made in step 607 after each epoch. If the requirement is not met, a backpropagation step 608 is performed to calculate weight gradients with respect to the calculated loss 606, for example using the chain rule. Specifically, given the complete loss function L=L(H,y) 606 for the current learning iteration, each deep learning learnable parameter weight W i (W i is the i-th learnable parameter). If the model has m learnable parameters, the gradient of the loss function is defined as:

[0103]

number

[0104] In one embodiment, a stochastic gradient descent version of backpropagation is used over the training batches. The gradient is calculated by minimizing the loss L606 using the parameter W i should be changed in a positive or negative direction.

[0105] The weights are updated according to the loss gradient from 608 and an optimizer in 609 and its learning rate. The optimizer is an algorithm that modifies the attributes of the neural network such as the weights and the learning rate, thus helping to reduce the overall loss and improve the accuracy. In one embodiment, the Adam optimization algorithm is used as defined in

[34] .

[0106] The batch of data 603 for the next iteration is updated with new volumes and labels from the training data 603 for each iteration.

[0107] The training continues, batch after batch, at each iteration across all epochs, until requirement 6607 is met.

[0108] If the requirements of step 607 are met, the training procedure stops at step 610 and the model weights are trained and ready to be used in the 3D DSNT enhanced residual U-Net in the inference phase, as shown in FIG. 1.

[0109] Dataset CCTA Dataset The CCTA dataset has 157 CCTA volumes collected from eight different medical centers. It has 67 volumes of female patients and 90 volumes of male patients with a mean age of 69.7 ± 11.1 years. Images were acquired from six different scanners: Siemens SOMATOM Force, Siemens SOMATOM Definition Flash, Siemens Sensation Cardiac 64, Canon Medical Systems Aquilion ONE, GE Medical Systems Discovery CT750HD, and GE Medical Systems Optima CT660. Volume size is 512x512x(130-420), spacing is 0.4-0.6mm, typical voxel size is 0.25mm. Landmarks include the location of the heads of the left and right coronary arteries and were manually labeled by trained radiologists.

[0110] ImageTBAD CTA Dataset The ImageTBAD CTA dataset

[29] is a computed tomography (CTA) image dataset of type B aortic dissection. The dataset consists of 100 CTA images collected from Guangdong Provincial People's Hospital in China. It has 31 volumes of female patients and 69 volumes of male patients with a mean age of 52.5 ± 11.3 years. Images were acquired from two types of scanners: a Siemens SOMATOM Force and a Philips 256slice Brilliance iCT system. The volume size is 512x512x(135-416), with a spacing of 0.75 mm and a typical voxel size of 0.25 mm. This dataset consists of segmentation labels of type B aortic dissection, which were augmented for the purposes of the present invention by trained radiologists. Similar to the CCTA dataset, landmarks include the location of the left and right coronary ostia and were manually labeled by trained radiologists. During labeling, 23 volumes were discarded due to insufficient data quality and reconstruction artifacts near the periosteum, resulting in 77 volumes with coronary periosteal labels. We have made these labels publicly available for further exploitation.

[0111] Comparison with known methods Using the two datasets mentioned above, we compared our method with three common regression methods. To ensure that all methods were comparable, the common regression methods used in the comparison comprised the same residual U-Net backbone used in the present invention. The three common regression methods were CNN (consisting of the U-Net coding part) + FC for coordinate regression, residual U-Net + FC for coordinate regression, and residual U-Net heatmap regression with argmax. We compared the methods using standard evaluation criteria

[17] for the task of landmark detection. The standard evaluation criteria consist of the Euclidean distance (EUC) and its median, mean, standard deviation (STD), interquartile range (IQR), and detection success rate (SDR). The SDR is defined as the proportion of predicted landmarks whose Euclidean distance with the reference landmark is less than or equal to a given threshold.

[0112] Table 1 shows a comparison of our method with three common methods. The SDR histogram plot is shown in Figure 9. The trends are similar for both datasets. The architecture using fully connected (FC) layers at the end of the backbone performed poorly in the comparison. This proves that in the task of osteria regression, especially landmark regression, utilizing heatmap generation is more beneficial than exact coordinate regression. In general, the methods using argmax over heatmap and the methods using DSNT according to our invention performed much better. By applying triple cross-validation, the method using DSNT achieved a median Euclidean distance error of 1.11 mm and 1.05 mm for the left and right mouths, respectively, a mean Euclidean distance error of 1.32 ± 0.94 mm and 1.48 ± 1.66 mm, and an IQR of 0.76 mm and 0.79 mm for all 157 scans. Of the total 314 landmarks (157 scans with 2 landmarks labeled), only 14 had EUC errors above 3.5 mm, which corresponds to an SDR of 95.5%. At the clinically accepted SDR range of 3.5 mm, our invention outperforms other methods on both datasets by achieving accuracies of 95.5% and 65% on the CCTA and ImageTBAD CTA datasets, respectively.

[0113] Table 1: Euclidean distance error (mm). Comparison of our method with three common methods.

[0114] [Table 1]

[0115] Ablation Study The Jensen-Shannon difference regularization L, which controls the desired heatmap spread over the volume of the CCTA dataset. DThe effect of σ was evaluated. See Table 2 for the evaluation results. If σ is too large or too small, the heat map is shaped to an unnatural size. As a result, the heat map has too much or too little context, which affects the prediction accuracy of the periosteum position. In the preferred embodiment, the best results were obtained with σ=3.

[0116] Table 2: Euclidean distance error (mm). Ablation study of standard deviation parameters in heatmap generation from ground truth of the CCTA dataset.

[0117] The method according to the invention outperformed other methods, achieving mean Euclidean distance errors of 1.32±0.94 mm and 1.48±1.66 mm for the left and right femoral heads, respectively, on the CCTA dataset, and 3.19±2.06 mm and 3.09±2.12 mm for the left and right femoral heads, respectively, on the ImageTBAD CTA dataset. For a clinically accepted SDR range of 3.5 mm, the method according to the invention achieved an accuracy of 95.5% on the CCTA dataset and 65% on the ImageTBAD CTA dataset.

[0118] FIG. 8 shows a three-dimensional rendering of a CCTA volume with a left coronary artery periosteum 801 and a right coronary artery periosteum 802. The present invention was tested and evaluated on a 157-volume CCTA dataset and a 77-volume CTA dataset. FIG. 7 shows slices from two exemplary volumes from these datasets. Images 700, 710, and 720 are exemplary slices from the CCTA dataset in sagittal, coronal, and axial views, respectively. Points 701, 711, and 712 indicate the location of the left coronary artery ostium. Images 730, 740, and 750 are exemplary slices from the CCTA dataset in sagittal, coronal, and axial views, respectively. Points 731, 741, and 752 indicate the location of the left coronary artery ostium.

[0119] [Table 2]

[0120] [1] Al, WA, Jung, HY, ID, Jang, Y., Park, HB, Chang, HJ..: Automated location of aortic valve landmarks in coronary CT angiograms using colonial walk. Plos one 13(7), e0200317(2018).[2] Dabbah, MA, Murphy, S., Pello, H., Courbon, R., Beveridge, E., Wiseman, S., Wyeth, D., Poole, I.: Detection and location of 127 antomical landmarks in diverse ct datasets. In: Image Processing. International Society for Optics and Photonics(2014). Such image analysis is of great importance in performing image analysis. [Such image analysis is also useful for automated detection of aortic root landmarks in CTA images for preoperative planning of aortic valve replacement. The international journal of cardiovascular imaging 32(3),501-511(2016).[5]Gao,Y.,Shen,D....:Collaborative regression-based anatomical landmark detection.Physics in Medicine & Biology 60(24),9377(2015).[6]Han,D., Gao,Y.,Wu,G.,Yap,PT.,Shen,D:Robust anatomical landmark detection for brain image registration. In:International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer(2014).[7]In such cases, robust anatomical landmark detection for brain image registration is required.In:International Symposyum on Biomedical imaging (ISBI)(2015).[8]Ionasec,RI,Georgescu,B.,Gassner,E.,Vogt,S.,Kutter,O.,Sheuering,M.,Nvab,N.,Comaniciu,D...:Dynamic model-driven quantitative and visual assessment of the aortic valve. In:International Conference on Medical Image Computing and Computer-Aided Interventions. Springer(2008).[9]Jaderberg,M.,Simonyan,K.,Zisserman,A.,et al:Spatial transformation networks. Advances in neural information processing systems 28,2017-2025(2015),

[10] Khanna,A.,Londhe,ND,Gupta,S.,Semwal,A.:Deep residual U-Net convolutional neural network for automatic lung segmentation in computed tomography images. With the advancement of image processing technology, such image analysis has become more accurate.

[11] Lin, J.: Dissimilarity measure based on Shannon entropy. IEEE Transactions on Information theory 37(1), 145-151(1991).

[12] Lu, X., Jolly, M. P.: Discriminative context modeling with auxiliary markers for LV landmark detection from single MR images. In: International Workshop on Statistical Atlases and Computational Models of the Heart. Springer(2012).

[13] Mahapatra, D.: Landmark detection in cardiac MRL using learned local image statistics. In: International Workshop on Statistical Atlases and Computational Models of the Heart. Springer(2012).

[14] Mostafa, A., Ghanem, A. M., El-Shatoury, M. M., Basha, T.: Improved centerline extraction in a fully automated coronary artery periosteal localization and centerline extraction framework using deep learning.In: 43rd IEEE Annual International Conference on Medical and Biological Engineering 2021 (EMBC) P. 3846-3849(2021).

[15] Newell, A., Yang, K., Deng., J.: Stacked hourglass networks for human pose estimation. In: European Conference on Computer Vision pp 483-499. Springer(2016).

[16] Nibali, A., He, Z., Morgan, S., Prendergast, A., He, Z., Morgan, S., Prendergast, L.: Numerical coordinate regression with convolutional neural networks. arXiv preprint arXiv:1801.07372(2018).

[17] Noothout, JM, De Vos, BD, Wolterink, JM, Psotma, EM, Smeets, PA, Takx, RA, Leiner, T., Viergever, MA, Isgum, I.: Automatic landmark localization in medical images by deep learning based regression and classification. IEEE transactions on medical imaging 39(12), 4011-4022(2020).

[18] Noothout, JM, De vos, BD, Wolterink, JM, Leiner, T., Isgum, I.: CNN-based landmark detection in cardiac CT scans. arXiv preprint arXiv:1804.04963(2018).

[19] Oktay, O.,Bai, W.,Guerrero, R.,Rajchl, M.,deMarvao, A.,O'Rega, DP,Cook, SA,Heinrich, MP,Glocker, B.,Rueckert, D.:Stratified decision forests for accurate anatomical landmark localization in cardiac images. IEEE transactions on medical imaging 36(1), 332-342(2016).

[20] Payer, C.Stern, D.,Bischof, H.,Urschler, M.:Integrating spatial configuration into heatmap regression-based CNNs for landmark localization.Medical image analysis 54,207-219(2019)

[21] Ronneberger,O.,Fischer,P.,Brox,T.:U-net: Convolutional networks for biomedical image segmentation. In:International Conference on Medical Image Computing and Computer-Assisted Intervention. pp.234-241.Springer(2015)

[22] Stern,D.,Ebner,T.,Urschler,M.:From local to global random regression forests: Exploring the localization of anatomical landmarks. In:International Conference on Medical Image Computing and Computer-Assisted Intervention. pp.221-229 Springer(2016).

[23] Tompson,JJ,Jain,A.,LeCun,Y.,Bregler,C.:Joint learning of convolutional networks and graphical models for human pose estimation. In:Advances in Neural Information Processing Systems 27,1799-1807(2014).

[24] Toshev,A.,Szergedy,C.:Deeppose: Human pose estimation with deep neural networks. In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pp.1653-1660(2014).

[25] Urshler,M.,Ebner,T.,Stern,D.:Integrating geometric and appearance information to build a unified framework for anatomical landmark positioning. Medical image analysis 43,23-36(2018).

[26] Wachter,I.,Kneser,R.,Korosoglou,G.,Peters,J.,Bakker,N.,Weese,J.,et al.K:Patient-specific models for planning and guidance of minimally invasive aortic valve implantation. In:International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 526-533. Springer(2010).

[27] Wolterink, JM, van Hamersvelt, RW, Viergever, MA, Leiner, T., Isgum, I.: Extraction of coronary artery centerlines in cardiac CT angiography using a CNN-based orientation classifier.Medical image analysis 51,46-60(2019).

[28] Yang,W.,Li,S.,Ouyang,W.,Li,H.,Wang,X.:Learning feature pyramids for human pose estimation. In:Proceedings of the IEEE International Conference on Computer Vision. pp.1281-1290(2017).

[29] Yao,Z.,Xie,W.,Zhang,J.,Dong,Y.,Qiu,H.,H.,Yuan,H.,Jia,Q.,Wang,T.,Shi,Y.,Zhuang,j.,et al.:Imagetbad: A 3D computed tomography angiography image dataset for automated segmentation of type b aortic dissection. Frontiers in Physiology 12(2021).

[30] Zhang,Z.,Liu,Q.,Wang,Y.:Road extraction with deep residual U-Net. IEEE Geoscience and Remote Sensing Letters 15(5), 749-753(2018).

[31] Zheng,Y.,John,M.,Liao,R.,Nottling,A.,Boese,J.,Kempfert,J.,Walther,T.,Brockmann,G.,Gomaniciu,D.:Automatic segmentation of the aorta and valve landmark detection in C-arm CT for transcatheter aortic valve implantation. IEEE Medical Imaging 31(12),2307-2321(2012).

[32] He,Kaiming,et al.:Deep diving into the rectifier: Exceeding human-level performance in imagenet classification. Proceedings of the IEEE International Conference on Computer Vision 2015.

[33] Loffe,Sergey,Christian Szegedy "Batch normalization: Speeding up deep network training by reducing internal covariate shift."” International Conference on Machine Learning PMLR, 2015.

[34] Kingma, Diederik P., Jimmy Ba. “ADAM: A Stochastic Optimization Method.” arXiv prepring arXiv:1412.6980(2014).

[35] Wu, Yuxin, Kaiming He. “Group Normalization.” Proceedings of the European Conference on Computer Vision (ECCV) 2018.

[36] Ulyanov, Dmitry, Andrea Vedaldi, Victor Lempitsky. “Instance Normalization: A Necessary Ingredient for Fast Stylization.” arXiv prepring arXiv:1607.08022(2016).

[37] Ba, Jimmy Lei, Jamie Ryan Kiros, Geoffrey E. Hinton. “Layer Normalization.” arXiv prepring arXiv:1607.06450(2016).

[0121] Any computational steps or sub-steps of the method according to the invention can be implemented using a computer or a computer program. In some specific embodiments, some or all computations are performed using a computer program stored in a computer or any type of storage device or both. In another embodiment, some or all computations for the purposes of the method can be performed remotely, for example using a cloud-based infrastructure, which can include the use of the Internet or a local network.

[0122] All measurement methods and algorithms described herein are intended to define the parameters of the invention and provide specific results that can be compared to clinical trials, but are by no means limiting and are exemplary. Words such as "comprises" or "consists of" are not limiting, e.g., if element A contains another element B, element A may contain other elements or elements in addition to element B. The use of the singular or plural is not intended to limit the scope of the disclosure, e.g., a portion of a description indicating that element A contains element B, which also discloses that multiple elements B are contained in one element A and multiple elements A are contained in one element B, as well as embodiments in which multiple elements A contain multiple elements B. The scope of the invention should therefore be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. A method for detecting a landmark (104, 206) in a volume (101, 201) of a medical image, comprising: The method includes generating at least one heat map using a trained U-Net applied to the volume of medical images; The medical image volume comprises a computed tomography volume (101, 201) or a magnetic resonance volume, the medical image volume comprises voxels, the at least one heat map comprises a probability of finding the localization of the landmark (104, 206) within the voxel, and the trained U-Net is a trained residual U-Net (203) having a contraction path and a dilation path, The method of claim 1, wherein the contraction pass has an encoding residual block (302, 303, 304, 305) for encoding a volume of the medical image, and the dilation pass has a decoding residual block (307, 308, 309, 310) for decoding the volume of the medical image encoded by the contraction pass.

2. 10. The method of claim 1, further comprising: Transforming the at least one heat map into coordinates of at least one landmark (104, 206) using a three-dimensional spatial numerical differential transform (103, 206).

3. The method according to any of claims 1 to 2, wherein the volume of medical images is a volume of pre-processed medical images (102).

4. 4. The method of claim 3, wherein the pre-processed medical image volume is obtained using at least one of the steps of center cropping, clipping, and setting values ​​of selected voxels to a defined value, normalizing voxel values, compacting the medical image volume, and growing.

5. 5. The method of claim 4, wherein the steps of clipping and setting values ​​of selected voxels to defined values ​​include: setting voxel values ​​less than −300HU to −300HU; setting the value of voxels greater than 1100HU to 1100HU; The method comprising:

6. 6. The method of claim 5, wherein normalizing (204) the voxel values ​​comprises: setting the value of voxels equal to −300HU to 0; setting the value of voxels equal to 1100HU to 1; Linearly scaling values ​​between -300HU and 1100HU to corresponding values ​​on a scale from 0 to 1; The method comprising:

7. The method according to any one of claims 4 to 6, wherein the step of miniaturizing comprises reducing the volume of the medical image by a factor of 3 to 4.

8. 8. The method of claim 7, wherein the step of downsizing includes reducing the volume of the medical image volume by 3.1 times, 3.2 times, 3.3 times, 3.4 times, 3.5 times, 3.6 times, 3.7 times, 3.8 times, or 3.9 times.

9. The method of any of claims 4 to 8, wherein the step of increasing (202) includes rotations of up to π / 6 radians and translations of up to 10 voxels in each dimension.

10. In the method according to any one of claims 1 to 9, the encoding step comprises: - downsampling said volume of medical images; increasing the number of channels to produce a downsampled output; sending said downsampled outputs to corresponding coded residual blocks (302, 303, 304, 305) and corresponding decoded residual blocks (307, 308, 309, 310); The method comprising:

11. The method of claim 10, wherein the downsampling step is performed by a factor of two.

12. 12. The method of claim 1, wherein the decoding step comprises upsampling an input to a residual decoding block of the medical image volume to produce an upsampled output having a reduced number of channels.

13. The method of claim 12, wherein the upsampling step is performed by a factor of two.

14. In the method according to any one of claims 1 to 13, the training step of the trained residual U-Net comprises: minimizing discrepancies between the at least one heatmap and a labeled heatmap having at least one label, the at least one label indicating the location of the landmark (104, 206) on the at least one heatmap; minimizing a Euclidean distance between the location on the at least one heatmap and the location on the heatmap having at least one label; The method of claim 1, further comprising at least one of:

15. The method of any one of claims 1 to 14, wherein the probability of finding the landmark (104, 206) has a normal distribution.

16. In the method according to any one of claims 14 to 15, the minimization of the Euclidean distance comprises: [0010] and said minimizing the difference comprises: [0025] Including D js (·||·) is the Jensen-Shannon divergence, H is the heatmap generated by the residual U-Net (203) and normalized by a normalization function, y is the expected localization, y is the predicted localization, N(y,σ 2 ) is the expected heat map generated by sampling from a normal distribution N with standard deviation σ centered at the expected location y.

17. 17. The method of claim 16, wherein σ is equal to 3.

18. In the method according to any of claims 14 to 17, the training step of the trained residual U-net (203) comprises a linear combination of the minimization of the dissimilarity and the minimization of the Euclidean distance: ##EQU00012## Here, λ is the regularization coefficient hyperparameter,method.

19. 20. The method of claim 18, wherein λ is equal to 1.

20. The method of any of claims 14 to 19, wherein the at least one heatmap and the labeled heatmap are generated using the same volume of the medical image or using different volumes.

21. 21. The method of claim 1, wherein the transformation from the three-dimensional differentiable space to a numerical value comprises: [0030] where H=φ(H), H is the heatmap H generated by the residual U-Net (203) and normalized by a normalization function, and "" F denotes the Frobenius dot product of two matrices, selected from H, X, Y, and Z, where X, Y, and Z are grids of the form m×n×l matrices defined as follows: ##EQU00011## method.

22. A method according to any one of claims 1 to 21, wherein each coded residual block (302, 303, 304, 305) comprises: a first encoding convolutional layer (402) for receiving an input of the encoded residual block (302, 303, 304, 305); a first encoding norm layer (403) for receiving an output from the first encoding convolutional layer (402); a first coding activation layer (404) for receiving an output from the first coding norm layer (403); a second encoding convolutional layer (405) for receiving an output from the first encoding activation layer (404); a second coding norm layer (406) for receiving an output from the second coding convolution layer (405); and a second coding activation layer (407) for receiving an output from the second coding norm layer (406); a subsequent coding convolutional layer (408) for receiving the coded residual block (302, 303, 304, 305) input (301); having the subsequent encoding convolutional layer (408) comprises a three-dimensional matrix of size 3x3x3 or 1x1x1, the second encoding convolutional layer (405) comprises a three-dimensional matrix of size 3x3x3, and the first encoding convolutional layer (402) comprises a three-dimensional matrix of size 3x3x3; the output from the subsequent coding convolutional layer (408) has a greater number of channels than the coding residual block (302, 303, 304, 305) input (301), and the output from the first coding convolutional layer (402) has a greater number of channels than the coding residual block (302, 303, 304, 305) input; The output from the subsequent convolutional coding layer and the output from the second coding activation layer (407) are merged (409) by regular matrix addition; the first encoding convolutional layer (402) and the subsequent encoding convolutional layer (408) have strided 3D convolution operators; The strided 3D convolution operator is: [0070] where K is a kernel with a 3D matrix, I is an example input volume, and s is the stride; where K is a kernel of a 3D matrix, I is an example input volume, and s is the stride. The first coding norm activation and the second coding activation layer (407) have an activation function, the activation function is selected from tanh, a rectified linear unit, or a parametric rectified linear unit; The parametric rectified linear unit is: [0097] having y i is the i-th element of the input volume y, α is a learnable parameter, and The first coding norm layer (403) and the second coding norm layer (406) include batch normalization, instance normalization, layer normalization, or three-dimensional batch normalization, and the three-dimensional batch normalization includes: [0080] where x is the input, y is the output of the batch normalization layer, γ and β are learnable parameters, and ε is a small constant value added to the denominator for numerical stability.

23. In a method according to any one of claims 1 to 22, each decoded residual block (307, 308, 309, 310) comprises: a decoded residual block (307, 308, 309, 310) having a first decoding convolutional layer (502) for receiving an input (501) and a decoding unsample layer (503) for receiving an output of the first decoding convolutional layer (502); the decoding unsample layer (503) is configured to spatially unsample the output of the first decoding convolution layer (502) by a factor of 2 using nearest neighbor interpolation, and is composed of a first decoding norm layer (504) for receiving an output of the decoding unsample layer (503), a first activation layer (505) for receiving an output of the first decoding norm layer (504), and a second decoding convolution layer (506) for receiving an output of the first activation layer (505); the second decoding convolutional layer (506) has a kernel having a three-dimensional matrix of size 3x3x3, a second decoding norm layer (507) for receiving an output of the second decoding convolutional layer (506), and a second decoding activation layer (508) for receiving an output of the second decoding norm layer (507); The three-dimensional cross-correlation operator is: [006] where K is a kernel with a 3D matrix of size N x N x N, and I(x,y,z) is a voxel of an exemplary input volume I with coordinates x,y,z; the first decoding norm layer (504) and the second decoding norm layer (507) include batch normalization, instance normalization, layer normalization, or 3D batch normalization; The 3D batch normalization is: [0080] where x is the input, y is the output of the batch normalization layer, γ and β are learnable parameters, and ε is a small constant value added to the denominator for numerical stability, the first decoding activation layer (505) and the second decoding activation layer (508) have an activation function, the activation function being selected from tanh, a rectified linear unit, or a parametric rectified linear unit; The parametric rectified linear unit is: [0097] having y i is the i-th element of the input y, and α is a learnable parameter. method.

24. In a method according to any one of claims 22 to 23, each residual block (302, 303, 304, 305, 307, 308, 309, 310) comprises: Y l =h(x l )+F(x l ,W l ) X l+1 =f(y l ) where x l and x l+1 are the inputs and outputs of the l-th residual block, F(·) is the residual function, f(y l ) is the activation function, h(x l ) is the identity mapping function, For each coded residual block (302, 303, 304, 305), the residual function is The first encoding convolutional layer (402); the first coding norm layer (403); the first coding activation layer (404); The second encoding convolutional layer (405); and the second coding norm layer (406); the second coding activation layer (407); having For each decoded residual block (307, 308, 309, 310), the residual function is the second decoding convolutional layer (506); the second decoding norm layer (507); the second decryption activation layer (508); The method comprising:

25. 25. The method of claim 24, for each coded residual block (302, 303, 304, 305): [0018] where * is the cubic cross-correlation operator or the strided cubic cross-correlation operator, and C ｉｎ is the number of input channels, and C ｏｕｔ is the number of output channels, K is a set of learnable weights in the form of convolution kernels, s is the stride value, and the 3D cross-correlation operator is: [006] where K is a kernel with a three-dimensional matrix of size N×N×N, and I(x, y, z) is a voxel in input volume I with coordinates x, y, z.

26. In a method according to any one of claims 22 to 25, for each decoded residual block (307, 308, 309, 310), h(x l )=x l That is, the method.

27. The method according to any one of claims 1 to 26, wherein the at least one landmark (104, 206) comprises: Inflow and outflow fossa surrounding the ostium of the left coronary artery, inflow and outflow fossa surrounding the ostium of the right coronary artery, apex of the left ventricle, base / bottom of the left ventricle, 17 segments of the left ventricle according to the American Heart Association, junction of the left and right ventricles, origin of the left coronary artery (LCA), origin of the right coronary artery (RCA), left main coronary artery, origin of the first diagonal artery (DG1), origin of the second diagonal artery (DG2), origin of the Ramus Intermedius artery, origin of the first blunt marginal artery (OM1), origin of the second blunt marginal artery (OM2), origin of the RCA of the right coronary artery, origin of the AM of the acute side branch, origin of the PDA of the posterior descending branch, origin of the PL of the posterior branch of the left ventricle, of the superior vena cava Inlet to the right atrium, Inlet of the inferior vena cava to the right atrium, Inlet of the coronary sinus to the right atrium, Right adnexal / right adnexal entrance to the right atrium, Right adnexal / right adnexal apex, Left upper / right lower and right upper / right lower pulmonary vein entrance to the left atrium, Right adnexal / right adnexal entrance to the right atrium, Left adnexal / left adnexal apex, Origin of the pulmonary artery, Tricuspid valve (including leaflets: anterior, posterior and septum, apex of the right ventricle, Origin of the ascending aorta, Aortic valve with leaflets: right, left, posterior anterior, right anterior, right posterior, anterior coronary, left coronary, right coronary, non-coronary, Bicuspid valve with leaflets: anterior and posterior , Mitral valve: anterior and posterior, Anterior cerebral artery, Horizontal or anterior intercommunicating segment, Vertical, Postcommunicating or subumbilical segment, Landmarks located at the beginning of the preumbilical, supraumbilical and postumbilical segments, Anterior communicating artery, Superior cranial artery, Anterior posterior cranial artery, Posterior posterior cranial artery, Intermediate cranial artery, Sphenoid and horizontal segments, Insular segment, Landmarks located at the beginning of the palatine segments, Cortical segments, Posterior cranial artery, Anterior commissural segment, Posterior commissural segment, Tetrageminal segment, Cortical segments, Posterior commissural artery, OPHTALMIC artery, Left and right choroidal arteries, Basilar artery, Anterior spinal artery, Brachiocephalic artery / trunk, Left and right subclavian arteries, Left and right common carotid arteries, Left Right internal carotid artery, left and right external carotid arteries, left and right vertebral arteries, left and right thoracic and cervical trunks, right and left internal thoracic arteries, right and left costocervical trunks, right and left suprascapular arteries, right and left transverse cervical arteries, right and left axillary arteries, right and left brachial arteries, right and left ulnar arteries, right and left radial arteries, left and right superior thyroid arteries, left and right lingual arteries, left and right facial arteries, left and right ascending pharyngeal arteries, left and right occipital arteries, left and right posterior auricular arteries, left and right superficial temporal arteries, left and right maxillary arteries, posterior intercostal arteries, subcostal arteries, upper pneumatic arteries, pericardial arteries, esophageal arteries, esophageal branches of the inferior thyroid artery (upper third), thoracic esophageal branches of the aorta (middle third), esophageal branches of the left gastric artery (lower third), inferior phrenic arteries, lumbar arteries, celiac arteries, upper central arteries, lower central arteries,Middle and upper renal artery, renal artery, spermatic artery / internal spermatic artery / ovarian artery, iliac artery, internal iliac artery, external iliac artery, femoral artery, popliteal artery, anterior tibial artery, dorsalis pedis artery, posterior tibial artery, medial plantar artery, lateral plantar artery, v. superior great vein, vv. brachiocephalicae, vv. subclavian vein, vv. Ilugulares, v. inferior great vein, vv. Iliacae Commununes, external iliac vein, internal iliac vein, vv. lateral body of heart, vv. cardiac vein, great cardiac vein, small cardiac vein, v. Cardiac a medius, v. Posterio ventricular sinister, v. Obliquua Atrium,vv. Apical Anterior, Apical Lateral, Apical Lateral, Apical Lateral, Apical Lateral, Apical Lateral, Apical Lateral Cardiacaceae Minimae,vv. Great Vein, Inferior Great Vein,v. Anterior Septal Vein,v. Septimum Pellucidi Posterio,v. Thalamostriatum Superior,vv. Optic Linear Body Inferior, Thalamstriatum Inferior,v. Choroidea Superior,v. Choroidea Inferior, large vein, v. Upper large vein, vv. The lower part of the small chest, the front part of the small chest, the deep part of the small chest, the superficial part of the small chest, the anastomotic part of the small chest, the anastomotic part of the small chest, the anastomotic part of the small chest, the anastomotic part of the small chest, the anastomotic part of the small chest, the anastomotic part of the small chest, the anastomotic part of the small chest, the anastomotic part of the small chest. , anastomosis, superior anastomosis, inferior anastomosis, base, anterior commissure, posterior commissure, cerebral venous circulation, posterior commissure, superior ventricle, vv. superior cerebral hemisphere, inferior cerebral hemisphere, vv. inferior cerebral hemisphere, vv. Diploicae, v. Diploica Frontails, v. diplotemporalis anterior, v. temporalis anterior, v. temporalis posterior, occipital, vv. Emissariae, v. Emissaria paritalis, v. mastoid process, v. Emissaria condillaris, occipital, oval venous plexus, internal jugular venous plexus, sublingual venous plexus, vv. meninges, vv. v. medial sinus, thenar sinus, superior sagittal sinus, inferior sagittal sinus, rectal sinus, transverse sinus, portal sinus, occipital sinus, sinus junction, cavernous sinus, intercavernous sinus, basilar plexus, superior palpebral sinus, lacrimal sinus, v. ethmoid bone, anterior and posterior ethmoid bones, nasofrontalis muscle, subophthalmic muscle, facial muscles, angular oris muscle, supraorbital muscle, orbicularis oris muscle, palatine plexus ... Lips, Deep palate, Parotid gland, Inferior magnus muscle, Lateral palate, Posterior jaw, Mylohyoid ... maxilla, lateral auricle, posterior auricle, occipital lobe, scapulohumeral bone, parietal lobe ... v. comitans sublingualis, v. superior turbinate, v. subfascial, v. deep palmar vein, v. radial, v. ulnar, v. humerus, v. brachialis intermedius, v. coracoid, v. ulnar collateral, v. deep humerus, v. axillary, v. subclavian, v. calcaneus, v. phalangeal, v. fibulae, v. peroneal, v. peroneal fascia, v. peroneal fascia, v. peroneal fascia, v. peroneal fascia, v. cephalicus, v. thyroidea inferior, v. thyroideae imae, v. vertebralis, deep neck, v. IugularisExt, vertebral basement membrane, intervertebral membrane, spinal cord, intraspinal membrane, extraspinal membrane, radius, dorsum of foot, plantar, venous arc, plantar aponeurosis, plantar aponeurosis plexus, plantar aponeurosis plexus, plantar aponeurosis plexus, plantar aponeurosis plexus, plantar aponeurosis plexus, plantar aponeurosis plexus, plantar aponeurosis plexus, plantar venous arc, medial plantar, plantar midfoot, plantar perforator, anterior tibia, posterior tibia, peroneal muscle, popliteal fossa, genitalia, gastrocnemius muscle, femoral muscle, superficial epigastric, superficial periliac, epigastric abdominal, vulva, anterior scrotum, anterior labia, dorsal penis, dorsal clitoris, deep thigh, perforator, medial femoral crest, lateral femoral crest, superior gluteus maximus, vv. Inferior gluteal muscle, Dorsal phalanges, Dorsal metatarsal, Dorsal vein of the foot, Dorsal vein of the toes, Dorsal vein of the foot, Dorsal vein of the toes, Dorsal vein of the foot, Dorsal vein of the foot, Dorsal vein of the foot, Safena, Safena parva, Inferior vena cava, Inferior sternum, Lumbar, Hepatic bone, Renal bone, Superior renal bone, Testis, Ovary, Iliac commissure, Middle sacrum, Lateral ilium, Lateral ilium, Lateral sacrum, Superior renal bone, Testis, Ovary, Ilium, Middle sacrum, External ilium, Deep ilium, Inferior ilium, Internal ilium, Iliopsoas, Lateral sacrum, Superior gluteal muscle, Inferior gluteal muscle, Gluteus maximus, Internal clitoris, Deep penis, Clitoris resp., Clitoris, Caruncle of penis (resp.), Caruncle of vestibule (resp.), Vestibule of vagina, Posterior scrotum, Posterior labia. Posterior labia, inferior rectum, dorsal penis, dorsal clitoris, dorsal clitoris, dorsal clitoris, dorsal clitoris, dorsal clitoris, dorsal clitoris, dorsal clitoris, rectal venous plexus, penile venous plexus, prostatic venous plexus, uterine venous plexus, vaginal venous plexus, v. Portal venous plexus, superior mesenteric venous plexus, mesenteric-ileal venous plexus, ileal villous venous plexus, appendix venous plexus, large intestine venous plexus, pancreatic venous plexus, dilated gastrostomy venous plexus, pancreatic venous plexus, splenic venous plexus, splenic vein plexus, pancreatic vein plexus, stomach vein plexus, gastric fundus vein plexus, inferior mesenteric vein plexus, large intestine vein plexus, superior rectum vein plexus, gastric fundus vein plexus, gastric fundus vein plexus, foregut vein vein plexus, superior and posterior pancreatic vein plexus, bladder vein plexus, umbilical vein plexus, great duodenal papilla / Vater's papilla, entrance of the bladder duct into the common bile duct, exit of the bladder duct from the gallbladder, junction of the left and right hepatic ducts, junction of the right anterior and right posterior hepatic ducts, carina, subdivision into the right upper lobe bronchus, subdivision into the right middle lobe bronchus, right lower lobe Subdivision to bronchus, Subdivision to left lower lobe bronchus, Subdivision to right 1st segmental bronchus, Subdivision to right right upper lobe bronchus, Subdivision to right middle lobe bronchus, Subdivision to right lower lobe bronchus, Subdivision to left upper lobe bronchus, Subdivision to left lower lobe bronchus, Subdivision to right 1st bronchiole, Subdivision to right 2nd bronchiole, Subdivision to right 3rd bronchiole, Subdivision to right 4th bronchiole, Subdivision to right 5th bronchiole, Subdivision to right 6th bronchiole, Subdivision to right 7th bronchiole, Subdivision to right 8th bronchiole, Subdivision to right 9th bronchiole, Subdivision to right 10th bronchiole, Right 1st bronchiole Subdivision to bronchus, subdivision to left 1st bronchiole, subdivision to left 2nd bronchiole, subdivision to left 3rd bronchiole, subdivision to left 4th bronchiole, subdivision to left 5th bronchiole, subdivision to left 6th branch bronchus, subdivision to left 7th branch bronchus, subdivision to left 8th branch bronchus, anterior nose, entrance / exit of right posterior ethmoid cells to right sphenoidal recess / drainage, entrance / exit of left posterior ethmoid cells to left sphenoidal recess / drainage, entrance / exit of right sphenoid sinus to right sphenoidal recess / drainage of right sphenoidal recess to right superior turbinate, left sphenoidal sinus recess to left superior turbinate, right frontal sinus recess to Right frontal sinus recess, left frontal sinus recess to left frontal sinus recess, right frontal sinus recess to right middle turbinate, left frontal sinus recess to left nasal septum, right frontal lobe ethmoid cells to right turbinate complex, right frontal lobe ethmoid cells to right turbinate complex, right maxillary sinus to right turbinate complex, left maxillary sinus to left turbinate complex, right turbinate complex to right middle turbinate, left turbinate complex to left middle turbinate, right nasolacrimal duct to right inferior turbinate, left nasolacrimal duct to left inferior turbinate,Anterior nares / posterior nares / maxillary opening, uvula, pharyngeal tonsils, left and right auditory tube openings, tip of epiglottis, left and right piriform recesses / turbinates / nasal sinuses, posterior turbinate region, cricoid cartilage, interarticular fold, left and right epiglottic fold, left and right sphenoid cartilage, left and right maxillary fold, left and right outlet of laryngeal pouch, vocal cords, left and right vocal cords, first tracheal cartilage, kidney, renal papilla, lesser curvature, renal pyramid, renal pelvis, inlet ureter, abdominal aortic plexus, abducens nerve, accessory nerve, accessory sinus nerve, Alderman's nerve, inosseous sinus nerve, tibial nerve, anterior interosseous nerve, anterior superior alveolar nerve, Auerbach's plexus, auriculotemporal nerve, axillary nerve, brachial plexus, buccal branch of facial nerve, buccal nerve, cardiac plexus, cavernous nerve, Cavernous plexus, celiac ganglion, cervical branch of the facial nerve, cervical plexus, chorda tympani nerve, ciliary ganglion, coccygeal nerve, cochlear nerve, common peroneal nerve, common palmodigital median nerve, deep branch of the radial nerve, deep peroneal nerve, deep palpebral nerve, deep temporal nerve, Broca's diagonal band, digital branch of the facial nerve, dorsal branch of the ulnar nerve, dorsal nerve of the clitoris, dorsal nerve of the penis, dorsal nerve of the scapula, esophageal plexus, ethmoid nerve, external laryngeal nerve, external nasal nerve, facial nerve, femoral nerve, frontal nerve, gastric plexus, cerebellate ganglion, femoral genital nerve branch, femoral genital nerve, glossopharyngeal nerve, great auricular nerve, great occipital nerve, great tarsal nerve, hepatic plexus, hypoglossal nerve, iliolumbar nerve, inferior alveolar nerve, inferior anal nerve Hilar nerve, inferior cardiac nerve, inferior cervical ganglion, inferior gluteal nerve, inferior gastric plexus, inferior mesenteric plexus, inferior turbinate nerve, inferior orbital nerve, inferior orbital plexus, inferior turbinate nerve, intercostal nerve, intercostal brachial nerve, intermediate cutaneous nerve, internal carotid plexus, internal laryngeal nerve, internal ganglion, jugular ganglion, lacrimal nerve, lateral cord, lateral cutaneous nerve of forearm, lateral cutaneous nerve of thigh, lateral cutaneous nerve of thigh, lateral plantar nerve of thigh, lateral pterygoid nerve of thigh, lesser occipital nerve, lingual nerve, long ciliary nerve, long root of ciliary ganglion, long thoracic nerve, subscapular nerve, lumbar nerve, lumbar plexus, lumbar splenic nerve, lumbosacral nerve, lumbosacral plexus, lumbosacral trunk, mandibular nerve, mandibular branch of facial nerve, masseteric nerve, maxillary nerve, medial cord, Medial cutaneous nerve of the arm, medial cutaneous nerve of the forearm, medial cutaneous nerve, medial thoracic nerve, medial plantar nerve, medial pterygoid nerve, median nerve, Meissner's plexus, mental nerve, central nerve, middle cervical ganglion, median nerve, motor nerve, muscular branch of the radial nerve, musculocutaneous nerve, myoid nerve, nasociliary nerve, nasopalatine nerve, pterygoid nerve, varanus nerve, quadriceps nerve, piriformis nerve, xiphoid nerve, subclavian nerve, nerve intermedius, spinous nerve, nodose ganglion, greater trochanteric nerve, oculomotor nerve, olfactory nerve, ophthalmic nerve, optic nerve, auricular ganglion, ovarian plexus, palatine nerve, palmar branch of the median nerve, palmar branch of the ulnar nerve, pancreatic plexus, patellar plexus, pelvic splenic nerve, perforator cutaneous nerve,Perineal branch of posterior femoral cutaneous nerve, pharyngeal branch of the vagus nerve, pharyngeal branch of the glossopharyngeal nerve, pharyngeal nerve, pharyngeal plexus, respiratory nerve, respiratory plexus, posterior auricular nerve, posterior branch of spinal nerve, posterior cord, posterior cutaneous nerve of the arm ... forearm, posterior cutaneous nerve of the thigh, posterior scrotal nerve, posterior superior alveolar nerve, proper palmar digital nerve of the median nerve, anterior tentorium, Glandular plexus (neurogenic), pterygopalatine ganglion, saphenous nerve, saphenous plexus, pulmonary branch of the vagus nerve, radial nerve, recurrent laryngeal nerve, renal plexus, sacral plexus, sacrosplenic nerve, saphenous nerve, sciatic nerve, semilunar ganglion, sensory nerve, short ciliary nerve, sphenopalatine nerve, splenic plexus, hyoid branch of the facial nerve, subcostal nerve, submandibular ganglion, suboccipital nerve, superior cardiac nerve, superior cervical ganglion, superior glossopharyngeal ganglion, superior vagal ganglion, superior gluteal nerve, superior subgastric plexus, superior labial nerve, superior laryngeal nerve, superior lateral cutaneous nerve of the upper arm, superior mesenteric plexus, superior rectal plexus, supraclavicular nerve, superior orbital nerve, Superior renal plexus, suprascapular nerve, middle supratympanic nerve, auricular nerve, sympathetic trunk, temporal branch of the facial nerve, third occipital nerve, thoracic aortic plexus, thoracic splenic nerve, thoracic abdominal nerve, thoracic dorsal nerve, tibial nerve, transverse cervical nerve, trigeminal nerve, auricular nerve, tympanic nerve, ulnar nerve, suprascapular nerve, utero-vaginal plexus, vagus nerve, abdominal oblique nerve, cerebellar plexus, vestibular nerve, vestibulocochlear nerve, zygomatic branch of the facial nerve, zygomatic nerve, zygomaticofacial nerve, zygomaticotemporal nerve, cecal foramen, optic canal, superior orbital fissure, foramen rotundum, foramen ovale, foramen spinosum, hiatus, carotid artery, internal auditory foramen, jugular foramen, hypoglossal foramen, foramen major, The method is selected from the list having the following structure:

28. A computer readable storage medium having instructions which, when executed by a computer, cause said computer to carry out the steps of a method characterized by any of claims 1 to 27.

29. A system for detecting a landmark (104, 206) in a volume (101, 201) of a medical image, comprising: - a measuring means for acquiring a computed tomography volume (101, 201) or a magnetic resonance volume of a human patient; a computer system adapted to carry out the steps of the method characterized in any one of claims 1 to 27; The system has:

Images