Method for detecting images in a medical image volume

Abstract

Detection of markers in a medical image volume is an important step of developing various diagnostic procedures. The invention relates to a method for detecting markers in a medical image volume. The medical image volume may be selected from a computed tomography volume or a magnetic resonance volume. The invention includes a residual U-Net architecture to generate a heat map describing the probability of finding a flag, which heat map can be combined with a 3D microspace-to-numerical transformation (DSNT) to obtain coordinates of the flag. According to the invention, evaluation is carried out on two data sets: a data set comprising 157 coronary computed tomography angiography (CCTA) scans and a data set comprising 77 computed tomography angiography (CTA) scans. Compared to known methods, the proposed method provides significantly improved accuracy of flag detection, in particular for open flag detection.

Description

Field of the Invention

[0001] The present invention relates to landmark detection in medical images such as computed tomography (CT) volumes and magnetic resonance (MR) image volumes. Landmarks include, among other things, coronary artery ostia. Detection is accomplished using a deep learning model including a neural network. Specifically, a residual U-Net architecture. The transformation of probabilities to coordinates is accomplished using a non-trainable 3D differentiable spatial to numerical transformation (DSNT).

[0002] background

[0003] Automatic ostia detection is used in various medical image analysis techniques. It is often paired with other tasks, such as image segmentation or registration. The main challenges of automatic ostia detection are anatomical differences between patients and differences in image acquisition. The regression of landmark coordinates is done globally and does not require local refinement, as is required by other most common techniques [4, 17, 18, 26, 31], in particular to achieve state-of-the-art (SOTA) performance of median Euclidean distance errors equal to 1.12 mm and 1.05 mm for left and right ostia, respectively.

[0004] The first approach to addressing the problem of detecting the coronary ostium by coronary computed tomography angiography (CCTA) was based on finding intensity patterns. Waechter et al.

[26] and Elattar et al. [4] used previous aortic valve segmentation to perform intensity-based landmark detection on the aortic surface. Elattar et al. achieved a Euclidean error of 2.81 ± 2.07 mm. These methods are strictly designed for detecting the coronary ostium and require a two-step setup to perform the detection. The required two-step setup is time consuming and these methods are not time efficient. In addition, the accuracy of these methods is limited by anatomical abnormalities and the anatomical differences between patients.

[0005] The introduction of machine learning is beneficial to robustness and time consumption. Classical machine learning methods can be divided into two types: classification-based [2, 3, 7, 8, 12, 13, 19, 25, 31] and regression-based [1, 3, 5, 6, 19, 22, 25] methods. Classification-based methods produce the probability that a given pixel / voxel is a landmark of interest. For application in coronary ostium detection, Ionasec et al. [8] proposed the use of probability boosting trees to achieve a Euclidean error of 2.28 mm. Zheng et al.

[31] used global-to-local marginal space learning and obtained a Euclidean error of 2.11 ± 1.34 mm.

[0006] Regression-based methods are designed to output continuous values ​​defining the sub-resolution coordinates of the landmarks of interest. Al et al. [1] used a group walking-based approach to solve the coronary ostium detection problem and achieved a Euclidean error of 2.01 ± 1.04 mm. Due to recent developments in the field of artificial neural networks, many deep neural network (DNN)-based methods have been introduced to solve the landmark detection problem. The first network to solve the landmark detection problem used a convolutional neural network (CNN) model with a fully connected head to regress landmark coordinates [9, 24]. Mostafa et al.

[14] proposed using a U-Net architecture with a fully connected head

[21] to solve the coronary ostium detection problem. For a threshold of 2 mm, they reported SDRs (successful detection rates) of 68.3% and 74.1%, respectively.

[0007] Another approach using heatmap matching has been introduced in subsequent work

[23] . In this approach, landmarks are represented as Gaussian functions centered at their coordinates. The model included in this approach produces a heatmap that should match the Gaussian function landmark representation and is penalized by a mean squared error (MSE) loss. This landmark representation is similar to classification-based ML (ML-Machine Learning) methods, where the probability of the presence of a landmark at its location is predicted for each pixel / voxel. The current SOTA performance level on landmark detection is achieved by models based on pyramid / hourglass architectures [15,28]. Payer et al.

[20] also used heatmap matching for medical images to perform landmark detection on X-ray images. Wolterink et al.

[27] used it for coronary ostium detection and obtained a Euclidean error of 1.8±1.0 mm on a test set of 36 images.

[0008] The current best performing methods for coronary ostium detection use a two-step hybrid approach. Noothout et al. developed a method in 2018 to perform slice-wise image analysis

[18] . The method consists of a vector pointing to the landmark of interest from the center of each slice. This vector is regressed for each slice and the probability of the presence of a given landmark on a given slice is predicted. A CNN was implemented for this task with two heads and the final landmark locations were found by weighting the predicted vectors by the predicted probability of presence. The backbone of the CNN was responsible for finding rough landmark locations, which were then refined separately by subsequent networks with the same design as the main network. They reported Euclidean errors of 2.19 and 2.88 mm for the right and left coronary ostium, respectively, for a test set of 40 CCTA volumes. Noothout's method was further refined by its creators and extended by Noothout et al. in 2020

[17] . The slice-wise approach was changed to a patch-wise approach and the CCTA dataset was further expanded to 672 volumes, with 200 volumes included in one test set. Using a larger dataset and greater anatomical variability, they achieved a median (IQR-interquartile range) Euclidean error of 1.45 (1.20) mm for the right coronary ostium and 1.55 (0.97) mm for the left coronary ostium. However, such results could only be achieved using a subsequent local refinement network. The lowest median (IQR) Euclidean error achieved using only the global network was reported to be 2.90 (1.93) mm for the right coronary ostium and 3.31 (2.23) mm for the left coronary ostium.

[0009] This review of prior art developments is based on the need for a reliable and accurate method that allows to quickly find landmark coordinates. The present invention addresses this need.

[0010] Brief description of the invention

[0011] The following description is for a better understanding of the principles and benefits of the present invention as claimed in the appended claims. Therefore, it is not meant to be limiting in any sense. In particular, certain explanations of the meaning or function of equations used in the present invention, for example, should not be construed as limiting or exhaustive. They are provided for ease of understanding and cannot replace the complete understanding provided by the general common knowledge of those skilled in the art.

[0012] The present invention relates to the problem of landmark detection, in particular automatic opening detection. The present invention implements a supervised regression model that learns landmark coordinates from, for example, computed tomography (CT) volumes and from human annotations. The term supervised has the typical definition used in the field and refers to the use of examples to teach an algorithm.

[0013] We refer to the input volume as x, the human annotations as y, and the model predictions as Within the scope of the present disclosure, bold symbols used in equations refer to vector quantities, while non-bold symbols are directional quantities.

[0014] The present invention includes heatmap matching regression, which can be combined with extracting heatmap center coordinates through a differentiable spatial to numerical transform (DSNT) layer. The present invention can involve a combination of residual U-Net and 3D DSNT for landmark detection. In particular, for coronary ostium detection.

[0015] To simplify the description, residual blocks can be used interchangeably with blocks. For example, a coded residual block can also be called a coded block. A volume is the result of a medical scan. For example, a computed tomography volume is the result of a CT scan.

[0016] In one aspect, the present invention relates to a method for locating a marker in a medical image volume. The method comprises generating at least one heat map using a trained residual U-Net applied on the medical image volume. For the method, the medical image volume consists of a computed tomography volume or a magnetic resonance volume. The medical image volume comprises voxels. The method may be implemented as a computer-implemented invention. Localization may include, but is not limited to, indicating a possible location on a heat map, providing coordinates of a marker, providing estimated coordinates of a marker, or providing a range of coordinates in which the marker is located. The range of coordinates may be a range for only one coordinate, two, or three coordinates.

[0017] The at least one heat map according to the present invention is generated using a trained residual U-Net applied on a medical image volume. Each of the heat maps represents the probability of finding the localization of the at least one landmark in a voxel.

[0018] The residual U-Net of the present invention comprises a contraction path and an expansion path. The contraction path comprises a residual encoding block for encoding a medical image volume. The expansion path comprises a residual decoding block for decoding a medical image volume encoded by the contraction path.

[0019] The method of the present invention provides fast and significantly improved accuracy of landmark localization when compared to known methods. Furthermore, when trained, the method becomes a one-step method, which is relevant from the perspective of its application in medical practice.

[0020] In one embodiment, the method further involves transforming the at least one heat map into coordinates of at least one marker using a 3D differentiable spatial to numerical transform. This embodiment performs even better when compared to methods using only residual U-Net.

[0021] In one embodiment, each of the heatmaps is normalized. Normalization makes the method less computationally demanding.

[0022] In one embodiment, the medical image volume is a preprocessed medical image volume. The preprocessing may be different for training and for using the method of the present invention. For example, the reprocessed medical image volume is obtained using at least one of center cropping, shearing and setting selected voxel values ​​to defined values, normalizing voxel values, reducing the size of the medical image volume, and increasing it. Preprocessing is performed to remove unnecessary information to reduce the computational requirements of the method.

[0023] The clipping and setting the selected voxel values ​​to the defined values ​​may include setting voxel values ​​less than -300HU to -300HU and setting voxel values ​​greater than 1100HU to 1100HU.

[0024] The normalizing voxel values ​​may include setting voxel values ​​equal to -300HU to 0, setting voxel values ​​equal to 1100HU to 1, and linearly scaling values ​​between -300HU and 1100HU to corresponding values ​​in a scale from 0 to 1.

[0025] The downsizing may include reducing the volume of the medical image by a factor of 3 to 4. In particular, the downsizing includes reducing the volume of the medical image by a factor of 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8 or 3.9.

[0026] Enlargement may include rotations of up to π / 6 radians and translations of up to 10 voxels in each dimension.

[0027] In one embodiment, encoding includes downsampling the medical image volume and increasing the number of channels to produce a downsampled output, and sending the downsampled output to a corresponding encoding residual block and a corresponding decoding residual block.

[0028] In one embodiment, the downsampling is done by a factor of two.

[0029] In another embodiment, the decoding comprises upsampling the input of the residual decoding block of the medical image volume to produce an upsampled output having a reduced number of channels. The upsampling may be done by a factor of two.

[0030] In one embodiment, the training of the trained residual U-Net includes at least one of the following:

[0031] - minimizing a divergence between the at least one heat map and an annotated heat map comprising at least one annotation, wherein the at least one annotation points to a location of a marker on the at least one heat map, and

[0032] - minimizing the Euclidean distance between a location on the at least one heat map and a location on the heat map comprising the at least one annotation.

[0033] In one embodiment, the training of the trained residual U-Net includes minimizing the divergence between the at least one heat map and an annotated heat map including at least one annotation, and minimizing the Euclidean distance between the localization on the at least one heat map and the localization on the heat map including at least one annotation, wherein the at least annotation points to the localization of the marker on the at least one heat map. Training involving both minimizations allows the most accurate results to be obtained.

[0034] In an embodiment, the probability of finding a landmark has a normal distribution.

[0035] In an embodiment, said minimization of the Euclidean distance comprises And the minimization of the divergence includes D JS (·||·) is the Jensen-Shannon divergence.

[0036] In an embodiment, the training of the trained residual U-Net comprises a linear combination of minimization of said divergence and minimization of said Euclidean distance.

[0037] In an embodiment, the at least one heat map and the annotated heat map are generated using the same medical image volume.

[0038] In an embodiment, the 3D differentiable spatial to numerical transformation comprises: in, and is the heatmap H output from the residual U-Net and normalized by the normalization function, where <·,·> F represents the Frobenius inner product of two matrices, where the two matrices are selected from X, Y and Z.

[0039] X, Y, and Z are grids of the form m×n×l matrices, defined as:

[0040]

[0041] This 3D DSNT outperforms known landmark detection methods, especially for the coronary ostium landmark.

[0042] In an embodiment, each residual block comprises:

[0043]

[0044] x l+1 =f(yl )

[0045] Among them, x l and x l+1 is the input and output of the lth residual block, is the residual function, f(y l ) is the activation function, h(x l ) is the identity mapping function.

[0046] In an embodiment, the at least one flag is selected from the list comprising:

[0047] Around the inlet and outlet of the left coronary artery ostium, Around the inlet and outlet of the right coronary artery ostium, Apex of the left ventricle, Base / base of the left ventricle, 17 segments of the left ventricle according to the American Heart Association, The junction of the left and right ventricles, LCA ostium of the left coronary artery ostium, RCA ostium of the right coronary artery ostium, LM branch of the left main coronary artery, DG1 origin of the first diagonal artery, DG2 origin of the second diagonal artery, Origin of the intermedius artery, OM1 origin of the first obtuse marginal artery, OM2 origin of the second obtuse marginal artery, RCA origin of the right coronary artery, AM origin of the sharp marginal branches, PDA origin of the posterior descending artery, PL origin of the posterior left ventricular artery, SVC to Entrance of the right atrium, entrance of the inferior vena cava to the right atrium, entrance of the coronary sinus to the right atrium, entrance of the right atrial appendage / auricle to the right atrium, apex of the right atrial appendage / auricle, entrance of the left superior / inferior and right superior / inferior pulmonary veins to the left atrium, entrance of the right atrial appendage / auricle to the right atrium, apex of the left atrial appendage / auricle, origin of the pulmonary trunk, tricuspid valve including leaflets (anterior leaflet, posterior leaflet and septum), apex of the right ventricle, origin of the ascending aorta, aortic valve including leaflets (right, left, posteroanterior, right anterior, right posterior, anterior coronary, left coronary, right coronary, non-coronary), bicuspid valve including leaflets (anterior and posterior), mitral valve including leaflets (anterior and posterior) valve), anterior cerebral artery, landmarks at the beginning of the horizontal or anterior communicating segment, vertical, posterior communicating segment or the inferior, anterior, superior, and posterior corpus callosum segments, anterior communicating artery, superior cerebellar artery, anterior inferior cerebellar artery, posterior inferior cerebellar artery, middle cerebral artery, landmarks at the sphenoid / horizontal, insular, opercular, and cortical segments, posterior cerebral artery, landmarks at the anterior, posterior, quadrigeminal, and cortical segments, posterior communicating artery, ophthalmic artery, left and right choroidal arteries, basilar artery, anterior spinal artery, brachiocephalic artery / trunk, right / left subclavian artery, right / left common carotid artery, right / left internal carotid artery, right / left external carotid artery, right / left vertebral artery, right / left thyrocarotid trunk, right / left internal thoracic artery, right / left costocervical trunk, right / left suprascapular artery right / left transverse carotid artery, right / left axillary artery, right / left brachial artery, right / left ulnar artery, right / left radial artery, right / left superior thyroid artery, right / left lingual artery, right / left facial artery, right / left ascending pharyngeal artery, right / left occipital artery, right / left posterior auricular artery, right / left superficial temporal artery, right / left maxillary artery, posterior intercostal artery, subcostal artery, superior phrenic artery, pericardial artery, esophageal artery, esophageal branch of inferior thyroid artery (upper third), esophageal branch of thoracic aorta (middle third), esophageal branch of left gastric artery (lower third), inferior phrenic artery, lumbar artery, celiac trunk, superior mesenteric artery, inferior mesenteric artery, middle adrenal artery, renal artery, testicular artery / internal spermatic artery / ovarian artery, common iliac artery, internal iliac artery, external iliac artery, femoral artery, popliteal artery, anterior tibial artery,dorsalis pedis artery, posterior tibial artery, medial plantar artery, lateral plantar artery, superior vena cava, brachiocephalic vein, subclavian vein, jugular vein, inferior vena cava, common iliac vein, external iliac vein, internal iliac vein, cardiac vein, great cardiac vein, small cardiac vein, middle cardiac vein, posterior left ventricular vein, oblique left atrial vein, anterior cardiac vein, right marginal cardiac vein, left marginal cardiac vein, smallest cardiac vein, cerebral vein, internal cerebral vein, anterior septal vein, posterior septal vein, superior thalamostriatal vein, inferior thalamostriatal vein, superior choroid plexus vein, inferior choroid plexus vein, great cerebral vein, superior cerebral vein, inferior cerebral vein, anterior cerebral vein, deep middle cerebral vein, superficial middle cerebral vein, anastomotic vein, upper anastomotic vein, lower anastomotic vein, basal vein, anterior communicating vein, posterior communicating vein, cerebral venous circle, posterior communicating vein, superior lumbar vein, superior cerebellar vein, inferior lumbar vein, inferior cerebellar vein, occipital vein, frontal lumbar vein, anterior temporal lumbar vein, posterior temporal lumbar vein, occipital lumbar vein, emissary vein, parietal emissary vein, mastoid emissary vein, ankle emissary vein, occipital emissary vein, foramen ovale venous plexus, internal jugular venous plexus, hypoglossal canal venous plexus, meningeal vein, middle meningeal vein, dural sinus, superior sagittal sinus, inferior sagittal sinus, straight sinus, transverse sinus, sigmoid sinus, occipital sinus, sinus confluence, cavernous sinus, intercavernous sinus, basal plexus, superior ophthalmic vein, lacrimal vein, anterior and posterior ethmoid veins, nasofrontal vein, inferior ophthalmic vein, facial vein, angular vein, supratrochlear vein, supraorbital vein, palpebral vein, external nasal vein, labial vein, deep facial vein, parotid branch, submental vein, external palatine vein, posterior mandibular vein, superficial temporal vein Veins, middle temporal vein, transverse facial vein, mandibular joint vein, stylomastoid vein, anterior auricular vein, parotid vein, maxillary vein, external jugular vein, posterior auricular vein, occipital vein, suprascapular vein, transverse jugular vein, anterior jugular vein, pterygoid plexus, sphenopalatine vein, middle meningeal vein, deep temporal vein, superior alveolar vein, masseteric vein, inferior alveolar vein, vertebral vein, anterior vertebral vein, deep jugular vein, internal jugular vein, sigmoid sinus, occipital sinus, hypoglossal canal venous plexus, cochlear canal vein, inferior petrosal sinus, occipital vein, facial vein, pharyngeal vein, lingual vein, deep lingual vein, dorsal lingual vein, hypoglossal nerve accompanying vein, superior thyroid vein, subfascial vein, deep palmar venous arch, radial vein, ulnar vein, brachial vein, common brachial vein, basilic vein, musculus vein, ulnar collateral vein, deep brachial vein , axillary vein, subclavian vein, cutaneous vein, digital vein, intercarpal vein, palmar venous arch, interdigital vein, dorsal digital venous arch, dorsal venous network of hand, dorsal palm vein, dorsal palmar venous arch, marginal vein, dorsal vein of little finger, thumb vein, cephalic vein, cephalic vein of forearm, accessory cephalic vein, median cubital vein, basilic vein, median forearm vein, median basilic vein, median head vein, superior vena cava, brachiocephalic vein, venous angle, inferior thyroid vein, inferior thyroid vein, vertebral vein, deep cervical vein, external jugular vein, pericardial phrenic vein, thymic vein, pericardial vein, mediastinal vein, bronchial vein, tracheal vein, esophageal vein, internal thoracic vein, muscular diaphragm vein, superior abdominal wall vein, anterior intercostal vein, perforator, sternal branch, superior intercostal vein, thoracic and abdominal cutaneous vein, areolar venous plexus,Thoracic and abdominal wall veins, costal axillary veins, azygos veins, hemiazygos veins, extravertebral venous plexus, internal vertebral veins, vertebral veins, intervertebral veins, spinal veins, internal spinal veins, external spinal veins, root veins, dorsal pedis veins, lateral plantar veins, plantar venous arches, medial plantar veins, cardiac veins, perforating veins, anterior tibial veins, posterior tibial veins, peroneal veins, popliteal veins, genicular veins, sural veins, small saphenous veins, femoral veins, superficial abdominal wall veins, superficial circumflex iliac veins, thoracic and abdominal wall veins, external pudendal veins, anterior scrotal veins, anterior labial veins, superficial dorsal veins of the penis, superficial dorsal veins of the clitoris, great saphenous veins, deep femoral veins, perforating veins, medial circumflex femoral veins, lateral circumflex femoral veins, superior gluteal veins, inferior gluteal veins, dorsal digital veins, intermetacarpal veins, dorsal plantar veins, dorsal pedis venous arches, soles of toes Veins, plantar venous arch, plantar venous network, dorsal venous network, saphenous vein, small saphenous vein, inferior vena cava, inferior phrenic vein, lumbar vein, hepatic vein, renal vein, adrenal vein, testicular vein, ovarian vein, common iliac vein, median sacral vein, external iliac vein, deep circumflex iliac vein, inferior epigastric vein, internal iliac vein, iliolumbar vein, lateral sacral vein, superior gluteal vein, inferior gluteal vein, obturator vein, internal pudendal vein, deep veins of penis and clitoris, bulbar vein of penis, bulbar vein of vagina, posterior scrotal vein, posterior labial vein, inferior rectal vein, dorsal vein of penis, circumflex vein of penis, dorsal vein of clitoris, superficial dorsal vein of penis, superficial dorsal vein of clitoris, rectal venous plexus, vesical venous plexus, prostatic venous plexus, uterine venous plexus, vaginal venous plexus, portal vein, superior mesenteric vein , intestinal and ileal veins, ileocolic vein, appendiceal vein, right colic vein, pancreatic vein, right gastroepiploic vein, inferior pancreaticoduodenal vein, splenic vein, pancreatic vein, short gastric vein, left gastroepiploic vein, inferior mesenteric vein, sigmoid vein, left colic vein, superior rectal vein, left gastric vein, right gastric vein, prepyloric vein, posterior superior pancreaticoduodenal vein, cystic vein, umbilical vein, major duodenal papilla / Vat's papilla, entrance of cystic duct to common bile duct, exit of cystic duct from gallbladder, junction of right and left hepatic ducts, junction of right anterior and right posterior hepatic ducts, carina, branches of right upper lobe bronchus, branches of right middle lobe bronchus, branches of right lower lobe bronchus, branches of left upper lobe bronchus, branches of left lower lobe bronchus, branches of right first segment bronchus, branches of right second segment bronchus Tracheal branches, right third segment bronchial branches, right fourth segment bronchial branches, right fifth segment bronchial branches, right sixth segment bronchial branches, right seventh segment bronchial branches, right eighth segment bronchial branches, right ninth segment bronchial branches, right tenth segment bronchial branches, right first segment bronchial branches, left first segment bronchial branches, left second segment bronchial branches, left third segment bronchial branches, left fourth segment bronchial branches, left fifth segment bronchial branches, left sixth segment bronchial branches, left seventh segment bronchial branches, left eighth segment bronchial branches, anterior nares, right posterior ethmoid atrium to outlet / drainage of right sphenoethmoid recess, left posterior ethmoid atrium to outlet / drainage of left sphenoethmoid recess, right sphenoid sinus to outlet / drainage of right sphenoethmoid recess, right sphenoethmoid recess to outlet / drainage of right superior nasal meatus,Left sphenoethmoid recess to outlet / drainage of left superior nasal meatus, right frontal sinus to outlet / drainage of right frontal vestibule, left frontal sinus to outlet / drainage of left frontal vestibule, right frontal vestibule to outlet / drainage of right middle nasal meatus, left frontal vestibule to outlet / drainage of left middle nasal meatus, right frontoethmoid cell to outlet / drainage of right oro-meatal complex, left frontoethmoid cell to outlet / drainage of left oro-meatal complex, right maxillary sinus to outlet / drainage of right oro-meatal complex, left maxillary sinus to outlet / drainage of left oro-meatal meatus, right nasolacrimal duct to outlet / drainage of right inferior nasal meatus, left nasolacrimal duct to outlet / drainage of left inferior nasal meatus, choanae / choanae / ostium, uvula, pharyngeal tonsils, left and right ear canal openings, tip of epiglottis, left and right piriformis recess / fossa / antrum, posterior cricoid area, cricoid cartilage, interarytenoid folds, left and right Aryepiglottic folds, left and right arytenoid cartilages, left and right vestibular folds, left and right laryngeal sac outlets, vocal cords, left and right vocal cords, first tracheal cartilage, kidney, renal papilla, renal calyx, renal pyramid, renal pelvis, ureteral inlet, abdominal aortic plexus, abducens nerve, accessory nerve, accessory obturator nerve, Alderman nerve, anocaudal nerve, cervical loop, anterior interosseous nerve, anterior superior alveolar nerve, Auerbach plexus, auriculotemporal nerve, axillary nerve, brachial plexus, buccal branch of facial nerve, buccal nerve, cardiac plexus, cavernous nerve, cavernous plexus, celiac ganglion, cervical branch of facial nerve, cervical plexus, chorda tympani, ciliary ganglion, cochlear nerve, common peroneal nerve, common palmar digital nerve of median nerve, deep radial nerve branch, deep peroneal nerve, deep temporal nerve, digastric band of Broca, dorsal branch of facial nerve, dorsal branch of ulnar nerve, dorsal branch of clitoris Nerve, dorsal nerve of penis, dorsal nerve of scapula, esophageal plexus, ethmoid nerve, external laryngeal nerve, external nasal nerve, facial nerve, femoral nerve, frontal nerve, gastric plexus, geniculate ganglion, genital branch of genitofemoral nerve, genitofemoral nerve, glossopharyngeal nerve, greater auricular nerve, greater occipital nerve, greater petrosal nerve, hepatic plexus, hypoglossal nerve, iliohypogastric nerve, ilioinguinal nerve, inferior alveolar nerve, inferior anal nerve, inferior cardiac nerve, inferior cervical ganglion, inferior gluteal nerve, inferior hypogastric plexus, inferior mesenteric plexus, infrapalpebral nerve, infraorbital nerve, infraorbital plexus, infratrochlear nerve, intercostal nerve, intercostobrachial nerve, intermediate cutaneous nerve, internal cervical plexus, internal laryngeal nerve, intermediate neuron, cervical ganglion, lacrimal nerve, lateral cord, lateral cutaneous nerve of forearm, lateral cutaneous nerve of thigh, lateral thoracic nerve, lateral plantar nerve, lateral pterygoid nerve, occipital Small nerve, lingual nerve, long ciliary nerve, long root of ciliary ganglion, long thoracic nerve, subscapular nerve, lumbar nerve, lumbar plexus, lumbar visceral nerve, lumboinguinal nerve, lumbosacral plexus, lumbosacral trunk, mandibular nerve, marginal mandibular branch of facial nerve, masseteric nerve, maxillary nerve, medial cord, medial brachial cutaneous nerve, medial forearm cutaneous nerve, medial cutaneous nerve, medial thoracic nerve, medial plantar nerve, median nerve, Meissner's plexus, mental nerve, cardiac nerve, middle cervical ganglion, middle meningeal nerve, motor nerve, radial nerve muscular branch, musculocutaneous nerve, mylohyoid nerve, nasociliary nerve, nasopalatine nerve, pterygoid nerve, internal obturator nerve, quadratus femoris nerve, piriformis nerve, stapedial nerve, subclavian nerve, intermediate nerve, spinous nerve, nodose ganglion, obturator nerve, oculomotor nerve,Olfactory nerve, ophthalmic nerve, optic nerve, otic ganglion, ovarian plexus, palatine nerve, palmar branch of median nerve, palmar branch of ulnar nerve, pancreatic plexus, patellar plexus, pelvic visceral nerve, perforating nerve, perineal branch of posterior femoral cutaneous nerve, perineal nerve, petrosal ganglion, pharyngeal branch of vagus nerve, pharyngeal branch of glossopharyngeal nerve, pharyngeal nerve, pharyngeal plexus, phrenic nerve, phrenic plexus, posterior auricular nerve, posterior branch of spinal nerve, posterior bundle, posterior brachial cutaneous nerve, posterior forearm cutaneous nerve, posterior thigh cutaneous nerve, posterior scrotal nerve, posterior superior alveolar nerve, palmar proper nerve of median nerve, prostatic plexus (nerve), pterygopalatine ganglion, pudendal nerve, pudendal plexus, pulmonary branch of vagus nerve, radial nerve, recurrent laryngeal nerve, renal plexus, sacral plexus, sacral visceral nerve, saphenous nerve, sciatic nerve, semilunar ganglion, sensory nerve, short ciliary nerve, sphenopalatine nerve, splenic plexus, stylohyoid branch of facial nerve, subcostal nerve, submandibular ganglion, suboccipital nerve, superficial radial nerve, superficial peroneal nerve Nerve, superior cardiac nerve, superior cervical ganglion, superior glossopharyngeal ganglion, superior vagus ganglion, superior gluteal nerve, superior hypogastric plexus, superior labial nerve, superior laryngeal nerve, lateral cutaneous nerve of the upper arm, superior mesenteric nerve, superior rectal plexus, supraclavicular nerve, supraorbital nerve, adrenal plexus, suprascapular nerve, supratrochlear nerve, sural nerve, sympathetic trunk, temporal branch of facial nerve, third occipital nerve, thoracic aortic plexus, thoracic visceral nerve, thoracoabdominal nerve, thoracodorsal nerve, tibial nerve, transverse cervical nerve, trigeminal nerve, trochlear nerve, tympanic nerve, ulnar nerve, superior subscapular nerve, uterovaginal plexus, vagus nerve, abdominal branch, bladder plexus, vestibular nerve, vestibulocochlear nerve, zygomatic branch of facial nerve, zygomatic nerve, zygomaticofacial nerve, zygomaticotemporal nerve, blind hole, optic canal, superior orbital fissure, foramen rotundum, foramen ovale, foramen spinosum, foramen hiatus, carotid canal, internal auditory foramen, jugular foramen, hypoglossal canal, foramen magnum. ,

[0048] Another aspect of the present invention relates to a computer-readable [storage] medium comprising instructions which, when executed by a computer, cause the computer to perform the steps according to any embodiment of the present invention.

[0049] Another aspect of the invention relates to a system for defining landmark coordinates in a medical image volume, wherein the system comprises a measuring device for collecting a computed tomography volume or a magnetic resonance volume of a human patient, and a computer system adapted to perform the steps of any embodiment of the method of the invention.

[0050] In one embodiment, the system according to the invention comprises a computer system adapted to perform the steps of the method including an embodiment with training. This embodiment allows obtaining coordinates even faster. In particular, for this embodiment, the operator acquiring the CT or MR image can prepare annotations indicating landmarks for at least a portion of the image, thereby allowing the method of the invention to learn using said portion of the image and define coordinates for the rest of the image. This also allows the method to be adjusted towards a specific landmark, since the annotations include the specific landmark.

[0051] Other features and advantages of the present invention will become more apparent from the detailed description and drawings of non-limiting embodiments. It should be understood that, unless otherwise stated, the embodiments presented in the specification and claims can be combined in any order and quantity to produce new embodiments that form part of the present disclosure. This description includes a large number of references to prior art documents, in particular scientific journals. The entire disclosure of the references is part of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present invention will now be described in more detail with reference to the accompanying drawings.

[0053] Figure 1 An application of an embodiment of the method of the present invention comprising a trained residual U-Net is depicted.

[0054] Figure 2 A detailed diagram of an embodiment of the method of the invention including a training step is shown.

[0055] Figure 3 The architecture of a residual U-Net according to an embodiment of the method of the present invention is presented.

[0056] Figure 4 The architecture of a coding block according to an embodiment of the present invention is presented.

[0057] Figure 5 The architecture of a decoding block according to an embodiment of the present invention is presented.

[0058] Figure 6 More detailed steps of training the residual U-Net according to an embodiment of the method of the present invention are shown.

[0059] Figure 7 Exemplary slices from two datasets with coronary ostia annotations are presented. The first row shows examples from the CCTA dataset and the second row shows examples from the ImageTBAD CTA dataset.

[0060] Figure 8 An example of a 3D rendering from a CCTA dataset is presented, showing the location of the left and right coronary artery ostia.

[0061] Fig. 9 The successful detection rate (SDR) for landmark localization in experiments on CCTA dataset and ImageTBAD CTA dataset is presented.

[0062] Detailed Description

[0063] The foregoing description has focused on embodiments comprising computed tomography volumes. Following adaptation of the described embodiments, magnetic resonance (MR) image volumes may also be used in the described embodiments.

[0064] At the heart of one embodiment of the present invention is a deep learning model 103 of a 3D differentiable spatial to numerical transform (DSNT) enhanced with a residual U-Net architecture. In other embodiments, the residual U-Net architecture may be used without DSNT. In order to use the model, the model must first be trained. Various embodiments of the method including training and architecture details are described in Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6 104. In one embodiment, the output is the coronary ostium landmark coordinates 104. Once the model 103 is trained, then in one embodiment, it infers two coordinate predictions with subvoxel accuracy for a given pre-processed input CT volume. One coordinate prediction is for the left coronary ostium, and one coordinate prediction is for the right coronary ostium. Each coordinate includes three numbers for sagittal, coronary, and axial positions, respectively. The use of the present invention including the trained model is referred to as the inference phase. The training of the model is referred to as the training phase.

[0065] The invention will be described in detail with respect to the coronary ostium landmark. However, the invention is applicable to other modalities and other landmarks. This requires the use of a different data set and, if necessary, one or more steps of preprocessing. The skilled person will appreciate that different data preprocessing may be required for different data sets. The principle of preprocessing remains the same and can be defined as obtaining a data set that is more easily usable in a particular detection. In order to perform localization of a different number of points, the C in the decoding block (310) is out must be set to a different number. Therefore, the output (311) will include the different number of landmark predictions. A list of landmarks for which coordinate detection can be performed is included in the description and the appended claims.

[0066] Figure 1 An application of a method suitable for coronary ostium detection in a CT volume 101 according to the present invention including a trained residual U-Net is shown. A CT volume is a 3-dimensional grid composed of 3-dimensional pixels called voxels. Each voxel contains information about relative radiodensity. Voxels are usually displayed in Hounsfield units (HU) according to the average attenuation of the tissue to which they correspond. In the Hounsfield scale, the range of values ​​is +3 071HU to -1 024HU. The volume is expected to be calibrated according to water with an attenuation of 0HU. A typical volume has a width of 512 voxels, a height of 512 voxels, and a length of z number of voxels. The number of voxels of length z depends on the number of slices. For example, Figure 7 The CTA volume shown in images 730, 740, and 750 in FIG. 5 has 330 slices and therefore contains a total of 512×512×330 voxels and is 330 voxels long. For example, Figure 7 The CCTA volumes shown in slices 700, 710, and 720 in FIG. 4 have voxel dimensions of 0.3281 x 0.3281 x 0.4 mm. 3 .

[0067] The input CT volume is preprocessed 102 to remove unnecessary information and normalize it to reduce the computational requirements of the method. In the preprocessing step, the voxels in the volume are clipped to a range from -300HU to 1100HU. That is, each value less than -300HU is set to -300HU, and each value above 1100HU is set to 1100HU. Next, the voxel values ​​are normalized by linearly scaling the voxel values ​​to values ​​from 0 to 1. That is, each voxel of -300HU is set to 0, each voxel of 1100HU is set to 1, and voxels with values ​​between -300HU and 1100HU are set to corresponding values ​​in the scale from 0 to 1. Other normalization techniques may be used. In order to retain the most information and address memory issues, the volume is downsized in the preprocessing step. In one embodiment, the volume is downsized to a spatial size of 144×144×144 voxels using trilinear interpolation. Other interpolation techniques may be used. In other embodiments, the input CT volume is downsized by a factor of 3-4, in particular by a factor of 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8 or 3.9. Even though the downsizing makes the data less detailed and spatially accurate, it has been shown that the accuracy of the results obtained using the method according to the invention is not affected and even exceeds known methods using local refinement.

[0068] Figure 2 Details of the training phase of the model on a single input volume are shown. The training model used in the present invention does not require Figure 2 207, 208, 209 and 210 and the increase in CT volume.

[0069] The training phase includes inputting volume x 201. The same input can be used for the inference phase. In one embodiment, the training phase also includes inputting coordinates y of two coronary ostia, where the coordinates are provided by a trained radiologist (CT landmark annotation 209). In the same embodiment, the training phase includes providing two coronary ostia locations. (Landmark coordinate prediction 206).

[0070] Generally, the method combines heatmap matching regression with extraction of heatmap center coordinates (as shown in step 210 ) via a differentiable spatial to numerical transform (DSNT) layer shown in step 205 .

[0071] Preprocessing and augmentation for training

[0072] The volume may be preprocessed in the same manner in the training phase and in the inference phase. In one embodiment, the preprocessing in the training phase additionally includes increasing the volume, as disclosed in the present application. The input volume may be preprocessed and then increased 202. The voxels may be clipped to a range from -300HU to 1100HU. In this embodiment, each value less than -300HU is set to -300HU, and each value above 1100HU is set to 1100HU. Next, the voxel values ​​are normalized by linearly scaling the voxel values ​​to values ​​from 0 to 1. That is, each voxel of -300HU is set to 0, each voxel of 1100HU is set to 1, and HU values ​​between -300HU and 1100HU are set to corresponding values ​​in a scale from 0 to 1. Finally, the volume may be resized to a spatial size of 144×144×144 voxels using trilinear interpolation. Other spatial sizes and other interpolation techniques may be used. In addition to this, the input volume can be enlarged by rotating up to π / 6 radians and translating up to 10 voxels in each dimension.

[0073] During the training and inference phases of the ImageTBAD CTA dataset, an additional preprocessing step in the form of center cropping can be applied to each volume. In one example, center cropping can include cutting 112 voxels along the Z axis from a volume of size 512x512xZ voxels on a slice to reach a volume of 288x288xZ. Center cropping removes unnecessary areas from the volume, leaving only the hearth area.

[0074] Heatmap generation

[0075] Usually, the unnormalized heat map H = f Θ (x) is the model f Θ The output of is parameterized by a learnable parameter (also called weight) θ with the input volume x. At step 203, f, which serves as the backbone feature extraction network in the present invention, Θ The model is a residual U-Net [10, 30], which takes the classic U-Net architecture

[21] and combines it with a residual neural network. The residual U-Net of the present invention discovers a heat map, which can be referred to as a feature discovered by the residual U-Net. For the avoidance of doubt, f ΘDenotes the residual U-Net. Θ is discovered during training and remains unchanged in application (inference phase). Those skilled in the art know that there are many residual U-Net architectures, and the difference between them lies in the structure of the encoding residual block and the decoding residual block included in the architecture. The structure of the residual block affects the application of a specific residual U-Net architecture. Therefore, the specific structure of the residual U-Net architecture can be defined by its purpose. Nevertheless, and in order to provide a complete disclosure of the present invention, details of embodiments of the residual U-Net architecture of the present invention are provided in the specification and claims.

[0076] Figure 3 The residual U-Net architecture according to an embodiment of the present invention is shown. It receives an input volume 301. The input 301 is passed to a series of encoding residual blocks (302, 303, 304, 305), which sequentially reduce the dimensionality and resolution size of the volume until the final encoding residual block 306 is achieved. Figure 3 In the figure, for the sake of brevity, encoder block and decoder block are used instead of encoding residual block and decoding residual block, respectively.

[0077] In an embodiment, the coded residual block receives a specific input (C in ) channels and outputs an input volume with output (C out ) channels. It also reduces the size of the volume according to the value of stride (s). For example, when stride is equal to 2, the output volume reduces its size by 2. When stride is equal to 1, the volume does not reduce its size. Typically, all convolution kernels in the coded convolution layer of the coded residual block have a dimension of 3x3x3 and are denoted as k s =3. As in the coded residual block 306, k s =1 means that the kernel dimension of the coded convolution layer in the mapping function in the coded residual block has a size of 1x1x1. The coded residual block is also connected to the corresponding decoding residual block (307, 308, 309, 310) to share information that would otherwise be lost during the spatial size reduction. Sharing should be understood as copying information. In other words, each coded residual block connected to another coded residual block outputs the same information twice, and one copy is transmitted to another coded residual block, and one copy is transmitted to the corresponding decoding residual block.

[0078] Each volume contains X, Y, Z and C dimensions. X, Y, Z are called spatial or shape dimensions. C is the number of channels. According to an embodiment, the volume in the architecture of the residual U-Net is transmitted as follows: an input volume 301 with 1 channel (grayscale Heinz scale) enters the encoding block 302 and generates a volume with 16 channels and half the size shape (stride = 2). A copy of the signal follows the encoding block 303 and the decoding block 310. The encoding block 303 generates a volume with 32 channels by downsampling by a factor of 2 (stride = 2). A copy of the signal follows the encoding block 304 and the decoding block 309. The encoding block 304 generates a volume with 64 channels by downsampling by a factor of 2 (stride = 2). A copy of the signal follows the encoding block 305 and the decoding block 308. The encoding block 305 generates a volume with 128 channels by downsampling by a factor of 2 (stride = 2). A copy of the signal follows the encoding block 306 and the decoding block 307. In contrast to the other encoding blocks (stride=1), encoding block 305 generates a volume with 128 channels and does not change the volume size. The output from 306 proceeds to decoding block 307. Decoding block 307 joins the signals from encoding blocks 305 and 306, resulting in a volume with 384 channels on its input. Decoding block 307 generates a volume with 64 channels by upsampling by a factor of 2. The output from 307 proceeds to decoding block 308. Decoding block 308 joins the signals from encoding blocks 304 and 307, resulting in a volume with 128 channels on its input. Decoding block 308 generates a volume with 32 channels by upsampling by a factor of 2. The output from decoding block 308 proceeds to decoding block 309. Decoding block 309 joins the signals from encoding blocks 303 and 308, resulting in a volume with 64 channels on its input. Decoding block 309 generates a volume with 16 channels by upsampling by a factor of 2. The output from the decoding block 309 proceeds to the decoding block 310. The decoding block 310 joins the signals from the encoding blocks 302 and 309, resulting in a volume with 32 channels at its input. The decoding block 310 produces a volume with 2 channels by upsampling by a factor of 2. The last step of the architecture is that the output volume 311 has the same spatial size as the input volume 301 but has two channels. In an embodiment, each channel corresponds to an estimated probability of landmark location. One probability for locating the left coronary ostium and one probability for locating the right coronary ostium. In other embodiments, the number of channels, downsampling factors, upsampling factors and / or stride values ​​may be different.

[0079] Each residual block for both encoding and decoding can be defined as follows:

[0080]

[0081] x l+1 =d(y l ), (1)

[0082] Among them, x l and x l+1 is the input and output of the lth residual unit, is the residual function, f(y l ) is the activation function, and h(x l ) is usually the identity mapping function. The residual function Depends on the residual block structure. For each coded block, it is defined as the product of the following terms: 402, 403, 404, 405, 406, 407. For each decoded block, it is defined as the product of the following terms: 506, 507, 508.

[0083] Figure 4 1 shows the architecture of a coding residual block according to an embodiment of the present invention. Each coding residual block includes a coding layer. For the sake of brevity, the word "coding" is omitted in the drawings. The volume entering the coding residual block can be referred to as the coding residual block input. The coding residual block receives a specific input (C in ) channels of the input volume 401 and output with output (C out ) channels. 402 is the first encoding convolutional layer, and 405 is the residual function of the encoding residual block At 408, the mapping function h(x l ) = conv(x l ,K,C in ,C out ,s), which consists of a single encoding convolutional layer.

[0084] For encoding the residual block according to an embodiment of the present invention, h(x l ) = conv(x l ,K,C in ,C out ,s), and is defined as follows:

[0085]

[0086] Where ★ is the 3D cross-correlation operator or its strided version. C in is the number of input channels, C out is the number of output channels. K is a set of learnable weights in the form of convolution kernels, and s is the stride value. The 3D cross-correlation is defined as follows:

[0087]

[0088] where K is the kernel positioned as a 3D matrix of size N×N×N, and I(x,y,z) is a voxel of an example input volume I with coordinates x,y,z. The strided version of the 3D cross-correlation is defined as:

[0089]

[0090] where K is the kernel defined as a 3D matrix, I is the example input volume, and s is the stride.

[0091] The first encoding convolutional layer 402 and the subsequent encoding convolutional layer 408 increase the number of channels from C in Increase to C out , while the second encoding convolutional layer 405 does not change the number of channels, for which C in =C out The first coded convolutional layer 402 and the subsequent coded convolutional layer 408 use a strided 3D cross-correlation operator, while the second coded convolutional layer 405 does not use a stride (i.e., the stride is equal to 1). The first coded convolutional layer 402 and the second coded convolutional layer 405 have a kernel size of k. s =3, which means that the kernel size has the dimension 3x3x3. The subsequent encoding convolutional layer 408 has a kernel size of k s , which is equal to 3 for the coded residual blocks 302 , 303 , 304 , 305 , and is equal to 1 for the coded residual block 306 .

[0092] The encoded convolutional layers 402 and 405 are followed by encoding the normalization layers 403 and 406, respectively. Examples of encoding the normalization layers include batch normalization

[33] , instance normalization

[36] , layer normalization

[37] , group normalization

[35] , etc. In one embodiment, 3D batch normalization

[33] is used. It is defined as follows:

[0093] where x is the input and y is the output of the encoded norm layer. γ and β are learnable parameters, and ∈ is a small constant value added to the denominator for numerical stability.

[0094] The encoding specification layers 403 and 406 are followed by activation layers 404 and 407, respectively, which encode activation functions. Examples of activation functions include tanh, rectified linear unit (ReLU), parameter rectified linear unit (PReLU), etc. In one embodiment, PReLU is used. It is defined as follows:

[0095] f(y i )=y i If y i ≥0,

[0096] f(yi )=α i y i If y i <0

[0097] Among them, y i is the i-th element of the input volume y, and a is a learnable parameter.

[0098] The outputs from 407 and 408 are combined with an element-wise addition function, which is defined as a regular matrix addition 409 and produces a matrix C of shape out The output volume is 410.

[0099] Figure 5 1 shows the architecture of the decoding residual block of an embodiment of the present invention. Each decoding residual block includes a decoding layer. For the sake of brevity, the word "decoding" is omitted in the drawings. The volume entering the decoding residual block can be referred to as the decoding residual block input. The decoding residual block receives a specific input (C in ) channels of the input volume 501 and output with output (C out ) channels. First, the input volume advances to 502, which is the first decoding convolutional layer defined for 402, 405 or 408. It converts the number of channels from C in Reduce to C out It uses a non-strided 3D cross-correlation operator. The decoding convolution layer 502 uses a constant kernel size k s = 1, size 1x 1x 1. 503 spatially upsamples the volume by a factor of 2 using nearest neighbor interpolation. 504 applies a decoding norm layer, here 3D batch normalization as defined for 403 or 406. 505 is the first decoding activation layer, here PReLU as defined for 404 or 407. 506 is the second decoding convolutional layer in the residual function of the residual decoding block. It is defined as 402, 405 and 408. It maintains the same number of channels C out The second decoding convolutional layer 506 uses a constant kernel size k s =3, of size 3x 3x 3. This is followed by a second decoding norm layer 507, which is here a 3D batch normalization as defined for 403 or 406. The second decoding norm layer 507 is followed by a second decoding activation layer 508, which is here a PReLU as defined for 404 or 407. The outputs from 505 and 508 are combined with an element-wise addition function, which is defined as a regular matrix addition 509 and produces a shape C out The output volume 510 of the decoded residual block has a mapping function h(x l )=x l of the form, and is Figure 5 The connection between 505 and 509.

[0100] The learnable parameters are the parameters that change during the training phase, which are: the kernel weights K for the convolutional layers, the parameters γ and β in the batch normalization layers, and the parameter a in the activation layers.

[0101] Heatmap normalization function

[0102] Figure 2 204 of : Normalization function Yes followed by L 1 Normalization correction operation. The single-channel heat map H output from the residual U-Net is normalized by the normalization function to produce a normalized single-channel heat map are output to 208 and 205 .

[0103] 3D DSNT

[0104] like Figure 2 , 3D Differentiable Spatial to Numerical Transform (3D DSNT)

[16] 205 is a fully differentiable and non-trainable layer. Due to this fact, the numerical coordinates are available during training and can be used as a learning criterion since backpropagation is indeed passed through 3D DSNT.

[0105] We define the 3D extension of DSNT as follows:

[0106]

[0107] in, and is a single channel heat map H output from the residual U-Net normalized by a normalization function (step 204). <·,·> F represents the Frobenius inner product of two matrices. The two matrices are selected from X, Y and Z.

[0108] X, Y, and Z are grids in the form of m×n×l matrices with expected values ​​for calculating the coordinates of the markers and are defined as:

[0109]

[0110] Each entry in X, Y, and Z contains its own coordinates, scaled so that the upper left corner of the volume is at (-1, -1, -1) and the lower right corner is at (1, 1, 1).

[0111] Loss Function

[0112] The coronary ostium locations manually annotated by trained radiologists are provided in step 209. The conversion from the manually annotated coronary ostium locations to an annotated heat map is done in step 207. This can be done by assuming a normal distribution of the coronary ostium locations on the heat map.

[0113] In 208, completed Minimize the divergence between and the annotated heatmap. This can be done using the Jensen-Shannon divergence regularization as the term to be calculated and by executing and an appropriate target normal distribution This is done by minimizing the divergence between , where σ is a hyperparameter that controls the spread of the desired distribution.

[0114] In 210, the minimization of the Euclidean distance between the annotated coronary ostium localization and the coronary ostium localization provided by 3D DSNT is performed. We can use the network output and the common Euclidean metric between the ground truth y It has been shown

[16] that forcing the network to produce heatmaps with normally distributed probabilities (e.g., Gaussian functions) improves the quality of the overall prediction.

[0115] The full loss is defined as a linear combination of the loss function for divergence (208), and the loss function for regression (210) can be defined as:

[0116]

[0117] Among them, λ is the regularization coefficient hyperparameter that controls the regularization strength.

[0118] The Euclidean loss LE (available for 210) can be defined as:

[0119]

[0120] And the Jensen-Shannon divergence regularized LD (available for 208) can be defined as:

[0121]

[0122] Among them, D JS (·||·) is the Jensen–Shannon divergence

[11] .

[0123] Training process

[0124] Figure 6An exemplary training process of the method is shown in more detail. Step 602 requires a training dataset, which may be a dataset for which landmark prediction is desired. The training dataset may be the same dataset for which landmark detection is desired. In an embodiment, a CCTA or CTA dataset is used. The training dataset is divided into training and validation data splits. The training dataset split is used to train the model weights, while the validation data split is used to validate the model performance after each training epoch. An epoch includes one iteration over all volumes in the training data split.

[0125] For the CCTA dataset, 3-fold cross validation can be used. This gives 104 volumes in the training data split and 53 volumes in the validation data split per cross. For the image TBAD dataset, 4-fold cross validation can be used to produce 58 volumes in the training data split and 19 volumes in the validation data split per cross. Each volume x in the dataset has a corresponding landmark annotation y.

[0126] At step 601, a set of training parameters (weights) are initialized and the training parameters are selected. There are two sets of parameters: trainable parameters and training parameters (hyperparameters). The deep learning model weights W include all trainable parameters. That is, W includes the kernel weights K in the convolutional layer, the parameters γ and β in the batch normalization layer, and the parameter a in the PReLU activation layer. In one embodiment, the model weights are initialized as follows: For the convolutional layer kernel weights K are initialized by sampling from the Kaiming uniform distribution as described in

[32] . The parameters γ and β in the batch normalization layer are set to 1 and 0, respectively. The parameter a in the PReLU activation layer is initialized to 0.25. These weights are used as the initialization weights in step 601 and are updated after each iteration during the training phase.

[0127] We can use the following training parameters: 1) The number of epochs, which is the maximum number of epochs used in the training phase. For example, it can be set to 1000 epochs. 2) The training batch size value, which is the number of training volumes in one iteration of training, which is usually limited by the GPU memory on which the training is performed. It can be set to four. That is, four volumes are cascaded and passed through the network in a single iteration of training. Other values ​​can be used. 3) The learning rate, which controls the strength of the weight update, is a scalar that is multiplied by the weight derivatives during training. It can be set to 0.00025. 4) The hyperparameter selected can be the optimization algorithm. The Adam optimization algorithm as defined in

[34] can be used as a hyperparameter. 5) Next, the hyperparameter is used as an evaluation metric. This hyperparameter is defined to monitor the model quality and is used for evaluation after each epoch. In one embodiment, we define the hyperparameter as the localization accuracy on the validation data split. 6) The stopping criterion is a hyperparameter used for early stopping of the training phase. In one embodiment, it is defined as reaching the maximum number of epochs, which can be 1000. Other quantities can be used. 7) Finally, the hyperparameters specific to the inventive architecture are the parameter σ, which controls the spread of the distribution of localization annotations in the divergence loss, and the regularization rate parameter λ, which controls the strength of the divergence loss effect. In one embodiment, the parameters are set to σ=3 and λ=1.

[0128] 3D DSNT 604 (deep learning model) enhanced with residual U-Net and 103 includes residual U-Net 203, heatmap normalization function 204 and 3D DSNT 205. In each iteration, it receives batch data 603 from training data separation 602, which includes training CT input volume 201 and corresponding CT landmark annotations 209. The deep learning model 604 uses the current weight values ​​in the forward propagation step 60 to infer predictions from the batch data 603. These predictions are then used in the loss calculation step 606 together with the corresponding data annotations of the batch data 603. The loss calculation step 606 includes the loss function for divergence from step 208 and the loss function for regression from step 210. As shown in step 207, the CT landmark annotations 209 must be converted into heat maps in order to calculate the divergence loss 208.

[0129] Once the loss is calculated, after each epoch, the model of the present invention infers predictions for the entire data validation split. The decision to continue training is made in step 607 after each epoch. If the requirements are not met, a backpropagation step 608 is performed to calculate the weight gradients with respect to the calculated loss 606 using, for example, the chain rule. Specifically, given the full loss function for the current training iteration For each deep learning trainable parameter weight W iCalculate the gradient, where Wi is the i-th trainable parameter. If the model has m trainable parameters, the gradient of the loss function is defined as follows:

[0130]

[0131] In one embodiment, a stochastic gradient descent version of backpropagation is used on the training batches. The gradient shows how much the parameters Wi need to change in the positive or negative direction to minimize the loss L 606.

[0132] At 609, the weights are updated according to the loss gradients from 608 according to the optimizer and its learning rate. An optimizer is an algorithm that modifies the properties of a neural network, such as weights and learning rate. Thus, it helps to reduce the overall loss and improve accuracy. In one embodiment, the Adam optimization algorithm as defined in

[34] is used.

[0133] The batch data 603 is updated for the next iteration using the annotations and new volumes from the training data 603 in each iteration.

[0134] Training continues batch by batch in each iteration of all epochs until requirement 607 is met.

[0135] If the requirements in step 607 are met, the training process stops in step 610 and the model weights are trained and ready for use in the 3D DSNT enhanced residual U-Net in the inference phase, as Figure 1 shown.

[0136] Dataset

[0137] CCTA Dataset

[0138] The CCTA dataset includes 157 CCTA volumes collected from eight different medical centers. It includes 67 female patient volumes and 90 male patient volumes with a mean age of 69.7±11.1 years. Images were acquired from six scanners: Siemens SOMATOM Force, Siemens SOMATOM Definition Flash, Siemens Sensation Cardiac 64, Canon Medical Systems Aquilion ONE, GE Medical Systems Discovery CT750HD, and GE Medical Systems Optima CT660. The volumes have a size of 512×512×(130-420), a pitch of 0.4-0.6 mm, and a typical voxel size of 0.25 mm. Landmarks include the location of the left and right coronary artery ostia and are manually annotated by trained radiologists.

[0139] ImageTBAD CTA dataset

[0140] The ImageTBAD CTA dataset

[29] is a dataset of computed tomography angiography (CTA) images of type B aortic dissection. The dataset consists of 100 CTA images collected from Guangdong Provincial People's Hospital, China. It includes 31 female patient volumes and 69 male patient volumes with a mean patient age of 52.5 ± 11.3 years. Images were acquired from two scanners: Siemens SOMATOM Force and Philips 256-slice Brilliance iCT system. The volumes have dimensions of 512 × 512 × (135–416), a pitch of 0.75 mm, and a typical voxel size of 0.25 mm. The dataset includes type B aortic dissection segmentation annotations and has been extended by trained radiologists for the purpose of this invention. As with the CCTA dataset, landmarks include the location of the left and right coronary artery ostia and are manually annotated by trained radiologists. During annotation, 23 volumes were discarded due to insufficient data quality and reconstruction artifacts near the ostia, resulting in 77 volumes with coronary artery ostia annotations. The inventors share these annotations with the public for further use.

[0141] Comparison with known methods

[0142] The method according to the present invention is compared with three commonly used regression methods using the above two datasets. For reliable comparison of all methods, the commonly used regression methods used in the comparison include the same residual U-Net backbone used in the present invention. The three commonly used regression methods are CNN (consisting of the U-Net encoding part) + FC for coordinate regression, residual U-Net + FC for coordinate regression, and residual U-Net heat map regression with argmax. We used standard evaluation criteria [as disclosed in 17] to compare methods in the landmark detection task. Standard evaluation criteria include Euclidean distance (EUC) and its median, mean, standard deviation (STD), interquartile range (IQR) and successful detection rate (SDR). SDR is defined as the percentage of predicted landmarks with a Euclidean distance to the reference landmark below a given threshold.

[0143] Table 1 shows the comparison between the method according to the present invention and three commonly used methods. Fig. 9SDR histograms are shown in . The trends are similar for both datasets. In contrast, the architecture using a fully connected (FC) layer at the end of the backbone performs worse. This demonstrates that for the task of opening regression or in particular landmark regression, utilizing heatmap generation is more advantageous than exact coordinate regression. In general, using argmax on heatmaps and using DSNT according to the present invention perform better. The method using DSNT achieved a median Euclidean distance error of 1.11 mm and 1.05 mm on the left and right openings, respectively, with an average Euclidean distance error of 1.32 ± 0.94 mm and 1.48 ± 1.66 mm, and an IQR of 0.76 mm and 0.79 mm over all 157 scans by applying 3-fold cross validation. Of all 314 landmarks (157 scans with two annotated landmarks), only 14 landmarks had an EUC error higher than 3.5 mm, which is equivalent to 95.5% SDR. For a clinically acceptable SDR range of 3.5 mm, the present invention outperforms other methods on both CCTA and ImageTBAD CTA datasets by achieving 95.5% and 65% accuracy on both datasets.

[0144] Table 1: Euclidean distance error (mm) Comparison of the method of the present invention with three commonly used methods.

[0145]

[0146]

[0147] Ablation studies

[0148] σ in Jensen-Shannon divergence regularization The effect of σ in is evaluated, which controls the expected heat map distributed over the volume on the CCTA dataset. The evaluation results are shown in Table 2. If σ is too large or too small, the heat map will be forced to be shaped to an unnatural size. These result in the heat map having too much or too little background, and this affects the accuracy of the prediction of the mouth position. In the preferred embodiment, the best results are obtained for σ=3.

[0149] Table 2: Euclidean distance error (mm). Ablation study of the standard deviation parameter in heatmap generation from ground truth for CCTA dataset.

[0150] The method according to the present invention is superior to other methods, achieving an average Euclidean distance error of 1.32±0.94mm and 1.48±1.66mm for the left and right mouths on the CCTA dataset, and an average Euclidean distance error of 3.19±2.06mm and 3.09±2.12mm for the left and right mouths on the ImageTBAD CTA dataset. For a clinically acceptable SDR range of 3.5mm, the method of the present invention achieves an accuracy of 95.5% on the CCTA dataset and an accuracy of 65% on the ImageTBADCTA dataset.

[0151] Figure 8 A 3D rendering of a CCTA volume is shown with a left coronary ostium 801 and a right coronary ostium 802. The present invention was tested and evaluated on a CCTA dataset with 157 volumes and a CTA dataset with 77 volumes. Figure 7 Slices of two exemplary volumes from these data sets are shown. Images 700, 710, and 720 are exemplary slices from the CCTA data set in sagittal, coronal, and axial views, respectively. Points 701, 711, and 712 show the location of the left coronary artery ostium. Images 730, 740, and 750 are exemplary slices from the CCTA data set in sagittal, coronal, and axial views, respectively. Points 731, 741, and 752 show the location of the left coronary artery ostium.

[0152] Table 2:

[0153]

[0154]

[0155] References

[0156] [1] Al, W.A., Jung, H.Y., Yun, I.D., Jang, Y., Park, H.B., Chang, H.J.: Automatic aortic valve landmark localization in coronary ct angiography using colonial walk, PloS one 13(7), e0200317 (2018). [2] Dabbah, M.A., Murphy, S., Pello, H., Courbon, R., Beveridge, E., Wiseman, S., Wyeth, D., Poole, I.: Detection and location of 127 anatomical landmarks in diverse ct datasets, in Medical Imaging 2014: Image Processing, Volume 9034, page 903415, International Society for Optics and Photonics (2014). [3] Donner, R., Menze, B.H., Bischof, H., Langs, G.: Global localization of 3d anatomical structures by pre-filtered hough forests and discrete optimization, Medical image analysis 17(8), pages 1304–1314 (2013). [4] Elattar, M., Wiegerinck, E., van Kesteren, F., Dubois, L., Planken, N., Vanbavel, E., Baan, J., Marquering, H.: Automatic aortic root landmark detection in cta images for preprocedural planning of transcatheter aortic valve implantation, The international journal of cardiovascular imaging 32(3), pages 501–511 (2016).[5] Gao, Y., Shen, D.: Collaborative regression-based anatomical landmark detection, Physics in Medicine & Biology 60(24), 9377 (2015). [6] Han, D., Gao, Y., Wu, G., Yap, P.T., Shen, D.: Robust anatomical landmark detection for mr brain image registration, in International Conference on Medical Image Computing and Computer-Assisted Intervention pp. 186 - 193, Springer (2014). [7] Ibragimov, B., Likar, B., Pernus, F., Vrtovec, T.: Computerized cephalometry by game theory with shape-and appearance-based landmark refinement. Proceedings of International Symposium on Biomedical imaging (ISBI) (2015). [8] Ionasec, R.I., Georgescu, B., Gassner, E., Vogt, S., Kutter, O., Scheuering, M., Navab, N., Comaniciu, D.: Dynamic model-driven quantitative and visual evaluation of the aortic valve from 4dct, in International Conference on Medical Image Computing and Computer-Assisted Intervention pp. 686 - 694, Springer (2008). [9] Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. Advances in neural information processing systems 28, 2017–2025 (2015).

[10] Khanna, A., Londhe, N.D., Gupta, S., Semwal, A.: A deep residual u-net convolutional neural network for automated lung segmentation in computed tomography images, Biocybernetics and Biomedical Engineering 40(3), pp. 1314–1327 (2020).

[11] Lin, J.: Divergence measures based on the shannon entropy, IEEE Transactions on Information theory 37(1), pp. 145–151 (1991).

[12] Lu, X., Jolly, M.P.: Discriminative context modeling using auxiliary markers for lv landmark detection from a single mr image, International Workshop on Statistical Atlases and Computational Models of the Heart, pp. 105–114, Springer (2012).

[13] Mahapatra, D.: Landmark detection in cardiac mri using learned local image statistics, International Workshop on Statistical Atlases and Computational Models of the Heart, pp. 115–124, Springer (2012).

[14] Mostafa, A., Ghanem, A.M., El-Shatoury, M., Basha, T.: Improved centerline extraction in fully automated coronary ostium localization and centerline extraction framework using deep learning, 2021 43rd Annual International Conference of the IEEE Engineering in Medicine Biology Society (EMBC), pp. 3846–3849 (2021).

[15] Newell, A., Yang, K., Deng, J.: Stacked hourglass networks for human pose estimation, European conference on computer vision, pp. 483–499, Springer (2016).

[16] Nibali, A., He, Z., Morgan, S., Prendergast, L.: Numerical coordinate regression with convolutional neural networks. arXiv preprint arXiv:1801.07372 (2018).

[17] Noothout, J.M., De Vos, B.D., Wolterink, J.M., Postma, E.M., Smeets, P.A., Takx, R.A., Leiner, T., Viergever, M.A., I.: Deep learning-based regression and classification for automatic landmark localization in medical images, IEEE transactions on medical imaging 39(12), pp. 4011–4022 (2020).

[18] Noothout, J.M., de Vos, B.D., Wolterink, J.M., Leiner, T., I.: CNN-based landmark detection in cardiac CTA scans. arXiv preprint arXiv:1804.04963 (2018).

[19] Oktay, O., Bai, W., Guerrero, R., Rajchl, M., de Marvao, A., O’Regan, D.P., Cook, S.A., Heinrich, M.P., Glocker, B., Rueckert, D.: Stratified decision forests for accurate anatomical landmark localization in cardiac images, IEEE transactions on medical imaging 36(1), pp. 332–342 (2016).

[20] Payer, C., D., Bischof, H., Urschler, M.: Integrating spatial configuration into heatmap regression based cnns for landmark localization, Medical image analysis 54 pp. 207–219 (2019).

[21] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation, International Conference on Medical image computing and computer-assisted intervention pp. 234–241, Springer (2015).

[22] D., Ebner, T., Urschler, M.: From local to global random regression forests: exploring anatomical landmark localization, in International Conference on Medical Image Computing and Computer - Assisted Intervention, pages 221 - 229, Springer (2016).

[23] Tompson, J.J., Jain, A., LeCun, Y., Bregler, C.: Joint training of a convolutional network and a graphical model for human pose estimation, Advances in neural information processing systems 27, pages 1799–1807 (2014).

[24] Toshev, A., Szegedy, C.: Deep pose: Human pose estimation via deep neural networks, Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1653–1660 (2014).

[25] Urschler, M., Ebner, T., D.: Integrating geometric configuration and appearance information into a unified framework for anatomical landmark localization, Medical image analysis 43, pages 23–36 (2018).

[26] I., Kneser, R., Korosoglou, G., Peters, J., Bakker, N., Weese, J. et al.: Patient specific models for planning and guidance of minimally invasive aortic valve implantation, in International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 526 - 533, Springer (2010).

[27] Wolterink, J.M., van Hamersvelt, R.W., Viergever, M.A., Leiner, T., I.: Coronary artery centerline extraction in cardiac ct angiography using a cnn-based orientation classifier, Medical image analysis 51, pp. 46–60 (2019).

[28] Yang, W., Li, S., Ouyang, W., Li, H., Wang, X.: Learning feature pyramids for human pose estimation, proceedings of the IEEE international conference on computer vision, pp. 1281–1290 (2017).

[29] Yao, Z., Xie, W., Zhang, J., Dong, Y., Qiu, H., Yuan, H., Jia, Q., Wang, T., Shi, Y., Zhuang, J., et al.: Imagetbad: A 3d computed tomography angiography image dataset for automatic segmentation of type-b aortic dissection, Frontiers in Physiology 12 (2021).

[30] Zhang, Z., Liu, Q., Wang, Y.: Road extraction by deep residual u-net, IEEE Geoscience and Remote Sensing Letters 15(5), pp. 749–753 (2018).

[31] Zheng, Y., John, M., Liao, R., Nottling, A., Boese, J., Kempfert, J., Walther, T., Brockmann, G., Comaniciu, D.: Automatic aorta segmentation and valve landmark detection in c-arm ct for transcatheter aortic valve implantation, IEEE transactions on medical imaging 31(12), pp. 2307–2321 (2012).

[32] He, Kaiming et al., "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification", Proceedings of the IEEE international conference on computer vision, 2015.

[33] Ioffe, Sergey and Christian Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift", International conference on machine learning, PMLR, 2015.

[34] Kingma, Diederik P. and Jimmy Ba, "Adam: A method for stochastic optimization", arXiv preprint arXiv:1412.6980 (2014).

[35] Wu, Yuxin and Kaiming He, "Group normalization", Proceedings of the European conference on computer vision (ECCV), 2018.

[36] Ulyanov, Dmitry, Andrea Vedaldi and Victor Lempitsky, "Instance normalization: The missing ingredient for fast stylization", arXiv preprint arXiv:1607.08022 (2016).

[37] Ba, Jimmy Lei, Jamie Ryan Kiros and Geoffrey E. Hinton, "Layer normalization", arXiv preprint arXiv:1607.06450 (2016).

[0157] Any computational step or sub-step of the method according to the present invention can be implemented using a computer or computer program. In some specific embodiments, some or all of the computations are performed using a computer program stored on a computer or on any type of memory device or both. In another embodiment, some or all of the computations for the purpose of the method can be performed remotely, for example using a cloud-based infrastructure, which can include the use of the Internet or a local network.

[0158] Although all the measurement methods and algorithms described herein are intended to define the parameters of the present invention and provide tangible results compared with clinical trials, they are by no means restrictive, but exemplary. Words such as "including" or "comprising" are not restrictive, and for example, when element A includes another element B, element A may include one or more other elements in addition to element B. The use of singular or plural forms does not limit the scope of the present disclosure, and for example, a part of the description indicating that element A includes element B also discloses that multiple element Bs are included in one element A and multiple element A is included in one element B and wherein multiple element A includes multiple element B embodiments. After reviewing the contents of the present disclosure, many other embodiments will be obvious and clear to those skilled in the art. Therefore, the scope of the present invention should be determined with reference to the attached claims together with the entire scope of equivalents to which the claims are entitled.

Claims

1. A method for detecting landmarks in a medical image volume, wherein: The method comprises generating at least one heat map using a trained residual U-Net applied to the medical image volume, wherein the medical image volume comprises a computed tomography volume or a magnetic resonance volume, wherein the medical image volume comprises voxels, wherein the at least one heat map comprises a probability of finding the location of the marker in the voxel, wherein the residual U-Net comprises a contraction path and an expansion path, and wherein the contraction path comprises an encoding residual block for encoding the medical image volume, and the expansion path comprises a decoding residual block for decoding the medical image volume encoded by the contraction path.

2. The method according to claim 1, wherein: The method further includes transforming the at least one heat map into coordinates of at least one marker using a 3D differentiable spatial to numerical transform.

3. The method according to any one of claims 1 to 2, wherein: The medical image volume is a preprocessed medical image volume.

4. The method according to claim 3, wherein: The reprocessed medical image volume is obtained using at least one of center cropping, shearing and setting selected voxel values ​​to defined values, normalizing voxel values, downsizing and upsizing the medical image volume.

5. The method according to claim 4, wherein: The clipping and setting the selected voxel values ​​to the defined values ​​includes setting voxel values ​​less than -300HU to -300HU, and setting voxel values ​​greater than 1100HU to 1100HU.

6. The method according to claim 5, wherein: Normalizing the voxel values ​​includes setting voxel values ​​equal to -300HU to 0, setting voxel values ​​equal to 1100HU to 1, and linearly scaling values ​​between -300HU and 1100HU to corresponding values ​​in a scale from 0 to 1.

7. The method according to any one of claims 4 to 6, wherein: The downsizing includes reducing the volume of the medical image by 3 to 4 times.

8. The method according to claim 7, wherein: The downsizing comprises reducing the volume of the medical image by a factor of 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8 or 3.

9.

9. The method according to any one of claims 4 to 8, wherein: The enlargement includes rotations of up to π / 6 radians and translations of up to 10 voxels in each dimension.

10. The method according to any one of claims 1 to 9, wherein: Encoding includes downsampling the medical image volume and increasing the number of channels to produce a downsampled output and sending the downsampled output to a corresponding encoding residual block and a corresponding decoding residual block.

11. The method according to claim 10, wherein: Downsampling is done by a factor of two.

12. The method according to any one of claims 1 to 11, wherein: Decoding includes upsampling an input to a residual decoding block of the medical image volume to produce an upsampled output having a reduced number of channels.

13. The method according to claim 12, wherein: Upsampling is done by a factor of two.

14. The method according to any one of claims 1 to 13, wherein: The training of the trained residual U-Net includes at least one of the following: - minimization of a divergence between the at least one heat map and an annotated heat map comprising at least one annotation, wherein the at least annotation refers to the location of the landmark on the at least one heat map, and - Minimization of the Euclidean distance between a location on the at least one heat map and a location on a heat map comprising at least one annotation.

15. The method according to any one of claims 1 to 14, wherein: The probability of finding the landmark has a normal distribution.

16. The method according to any one of claims 14-15, wherein: The minimization of the Euclidean distance includes And the minimization of the divergence includes Where D JS (·||·) is the Jensen-Shannon divergence, is the heatmap generated by the residual U-Net and normalized by the normalization function, y is the expected positioning, is predictive positioning, and From the normal distribution Sampling generates an expected heatmap with standard deviation σ about the expected location y.

17. The method according to claim 16, wherein: σ is equal to 3.

18. The method according to any one of claims 14 to 17, wherein: The training of the trained residual U-Net consists of a linear combination of minimization of the divergence and minimization of the Euclidean distance: Among them, λ is the regularization coefficient hyperparameter.

19. The method according to claim 18, wherein: λ is equal to 1.

20. The method according to any one of claims 14 to 19, wherein: The at least one heat map and the annotated heat map are generated using the same medical image volume or using different medical image volumes.

21. The method according to any one of claims 1 to 20, wherein: The 3D differentiable space to numerical transformation includes: in, and is the heat map H generated by the residual U-Net and normalized by the normalization function, where <·,·> F represents the Frobenius inner product of two matrices, where the two matrices are selected from X, Y and Z, and where X, Y, and Z are grids of the form m×n×l matrices and are defined as:

22. The method according to any one of claims 1 to 21, wherein: Each coded residual block includes: a first coded convolution layer for receiving a coded residual block input, a first coded norm layer for receiving an output from the first coded convolution layer, a first coded activation layer for receiving an output from the first coded norm layer, a second coded convolution layer for receiving an output from the first coded activation layer, a second norm layer for receiving an output from the second coded convolution layer, a second coded activation layer for receiving an output from the second coded norm layer, and a subsequent coded convolution layer for receiving the coded residual block input, and wherein the subsequent coded convolutional layer comprises a 3D matrix of size 3x3x3 or 1x1x1, wherein the second coded convolutional layer comprises a 3D matrix of size 3x3x3, and the first coded convolutional layer comprises a 3D matrix of size 3x3x3, wherein the output from the subsequent coded convolutional layer has a greater number of channels than the coded residual block input, wherein the output from the first coded convolutional layer has a greater number of channels than the coded residual block input, wherein the outputs from the subsequent convolutional coding layer and from the second coding activation layer are combined by regular matrix addition, Wherein, the first coded convolutional layer and the subsequent coded convolutional layer include a strided 3D convolution operator, wherein the strided 3D convolution operator includes: where K is a kernel comprising a 3D matrix, I is the sample input volume, and s is the stride, and The first coding specification activation layer and the second coding activation layer include activation functions, wherein the activation function is selected from tanh, a rectified linear unit, or a parameterized rectified linear unit, wherein the parameterized rectified linear unit includes: f(y i )=y i If y i ≥0, f(y i )=α i y i If y i <0 Among them, y i is the i-th element of the input volume y, and a is a learnable parameter, and The first coding specification layer and the second coding specification layer include batch normalization, instance normalization, layer normalization or 3D batch normalization, and the 3D batch normalization includes: where x is the input and y is the output of the batch normalization layer; γ and β are learnable parameters, and ∈ is a small constant value added to the denominator for numerical stability.

23. The method according to any one of claims 1 to 22, wherein: Each decoded residual block consists of: A first decoding convolutional layer, for receiving a decoding residual block input, comprising a 3D cross-correlation operator and a 3D matrix of size 1×1×1; a decoding upsampling layer, for receiving an output of the first decoding convolutional layer, wherein the decoding upsampling layer is configured to spatially upsample the output of the first decoding convolutional layer by a factor of 2 using nearest neighbor interpolation; a first decoding norm layer, for receiving the output of the decoding upsampling layer; a first activation layer, for receiving the output of the first decoding norm layer; a second decoding convolutional layer, for receiving the output of the first activation layer, wherein the second decoding convolutional layer comprises a kernel comprising a 3D matrix of size 3×3×3; a second decoding norm layer, for receiving the output of the second decoding convolutional layer; A second decoding activation layer, configured to receive an output of the second decoding normalization layer; and Wherein, the 3D cross-correlation operator includes: where K is a kernel comprising a 3D matrix of size N×N×N, and I(x,y,z) is a voxel of an example input volume I with coordinates x,y,z, The first decoding specification layer and the second decoding specification layer include batch normalization, instance normalization, layer normalization or 3D batch normalization, and the 3D batch normalization includes: where x is the input and y is the output of the batch normalization layer; γ and β are learnable parameters, and ∈ is a small constant value added to the denominator for numerical stability, and The first decoding activation layer and the second decoding activation layer include an activation function, wherein the activation function is selected from tanh, a rectified linear unit, or a parameter-rectified linear unit, wherein the parameter-rectified linear unit includes: f(y i )=y i If y i ≥0, f(y i )=α i y i If y i <0 Among them, y i is the i-th element of the input volume y, and a is a learnable parameter.

24. The method according to any one of claims 22-23, wherein: Each residual block consists of: x l+1 =f(y l ) Among them, x l and x l+1 is the input and output of the lth residual block, is the residual function, f(y l ) is the activation function, and h(x l ) is the identity mapping function, Wherein, for each coded residual block, the residual function includes the first coded convolution layer, the first coded norm layer, the first coded activation layer, the second coded convolution layer, the second norm layer and the second coded activation layer, and Wherein, for each decoding residual block, the residual function includes the second decoding convolution layer, the second decoding normalization layer and the second decoding activation layer.

25. The method according to claim 24, wherein: For each encoded residual block: Where, * is the 3D cross-correlation operator or the strided 3D cross-correlation operator; C in is the number of input channels; C out is the number of output channels; K is a set of learnable weights in the form of convolution kernels; and s is the stride value, where the 3D cross-correlation operator includes: where K is a kernel comprising a 3D matrix of size N×N×N, and I(x, y, z) is a voxel of an example input volume I with coordinates x, y, z.

26. The method according to any one of claims 22 to 25, wherein: For each decoded residual block, h(x l )=x l .

27. The method according to any one of claims 1 to 26, wherein: The at least one landmark is selected from the list consisting of: surrounding the inlet and outlet of the left coronary artery ostium; The entrance and exit around the right coronary artery ostium; the apex of the left ventricle; the base / base of the left ventricle; the 17 segments of the left ventricle according to the American Heart Association; the junction of the left and right ventricles; LCA ostium of left coronary artery; RCA ostium of right coronary artery; LM branch of left main coronary artery; DG1 origin of first diagonal artery; DG2 origin of second diagonal artery; origin of intermediate artery; OM1 origin of first obtuse marginal artery; OM2 origin of second obtuse marginal artery; RCA origin of right coronary artery; AM origin of sharp marginal branch; PDA origin of posterior descending artery; PL origin of posterior ventricular artery; entrance of superior vena cava to right atrium; entrance of inferior vena cava to right atrium; entrance of coronary sinus to right atrium; entrance of right atrial appendage / atrial appendage to right atrium; apex of right atrial appendage / atrial appendage; left upper / lower and right upper / Entrance of the inferior pulmonary vein to the left atrium; Entrance of the right atrial appendage / atrial appendage to the right atrium; Apex of the left atrial appendage / atrial appendage; Origin of the pulmonary trunk; Tricuspid valve, including leaflets: anterior, posterior, and septum; Apex of the right ventricle; Origin of the ascending aorta; Aortic valve, including leaflets: right, left, posteroanterior, right anterior, right posterior, anterior coronary, left coronary, right coronary, noncoronary; Mitral valve, including leaflets: anterior and posterior; Mitral valve, including leaflets: anterior and posterior; Anterior cerebral artery; Landmarks at the beginning of the horizontal or anterior communicating segment, vertical, posterior communicating segment or inferior, anterior, superior, and posterior corpus callosal segments; Anterior communicating artery; Superior cerebellar artery; anterior inferior cerebellar artery; posterior inferior cerebellar artery; middle cerebral artery; landmarks at the sphenoid / horizontal, insular, opercular, and cortical segments; posterior cerebral artery; landmarks at the anterior communicating, posterior communicating, quadrigeminal, and cortical segments; posterior communicating artery; ophthalmic artery; left and right choroidal arteries; basilar artery; anterior spinal artery; brachiocephalic artery / truncus; right / left subclavian artery; right / left common carotid artery; right / left internal carotid artery; right / left external carotid artery; right / left vertebral artery; right / left thyrocarotid trunk; right / left internal thoracic artery; right / left costocervical trunk; right / left suprascapular artery; right right / left transverse carotid artery; right / left axillary artery; right / left brachial artery; right / left ulnar artery; right / left radial artery; right / left superior thyroid artery; right / left lingual artery; right / left facial artery; right / left ascending pharyngeal artery; right / left occipital artery; right / left posterior auricular artery; right / left superficial temporal artery; right / left maxillary artery; posterior intercostal artery; subcostal artery; superior phrenic artery; pericardial artery; esophageal artery; esophageal branches of the inferior thyroid artery (upper third); esophageal branches of the thoracic aorta (middle third); esophageal branches of the left gastric artery (lower third); inferior phrenic artery; lumbar artery; celiac trunk; Superior mesenteric artery; inferior mesenteric artery; middle adrenal artery; renal artery; testicular artery / spermatic artery / ovarian artery; common iliac artery; internal iliac artery; external iliac artery; femoral artery; popliteal artery; anterior tibial artery; dorsalis pedis artery; posterior tibial artery; medial plantar artery; lateral plantar artery; superior vena cava; brachiocephalic vein; subclavian vein; jugular vein; inferior vena cava; common iliac vein; external iliac vein; internal iliac vein; cardiac vein; great cardiac vein; small cardiac vein; middle cardiac vein; posterior left ventricular vein; oblique left atrial vein; anterior cardiac vein; right marginal cardiac vein; left marginal cardiac vein; smallest cardiac vein; cerebral vein; internal cerebral vein; anterior septal vein; posterior septal vein; superior thalamostriate vein; inferior thalamostriate vein; superior choroid plexus vein; inferior choroid plexus vein; great cerebral vein; cerebral The superior vein; the inferior cerebral vein; the anterior cerebral vein; the deep middle cerebral vein; the superficial middle cerebral vein; the anastomotic vein; the superior anastomotic vein; the inferior anastomotic vein; the basal vein; the anterior communicating vein; the posterior communicating vein; the cerebral venous circle; the posterior communicating vein; the superior lumbrical vein; the superior cerebellar hemispheric vein; the inferior lumbrical vein; the inferior cerebellar hemispheric vein; the lamina vein; the frontal lamina vein; the anterior temporal lamina vein; the posterior temporal lamina vein; the occipital lamina vein; the emissary vein; the parietal emissary vein; the mastoid emissary vein; the ankle emissary vein; the occipital emissary vein; the venous plexus of the foramen ovale; the internal jugular plexus; the venous plexus of the hypoglossal canal; the meningeal vein; the middle meningeal vein; the dural sinus; the superior sagittal sinus; the inferior sagittal sinus; the straight sinus; the transverse sinus; the sigmoid sinus; the occipital sinus; the sinus confluence; the cavernous sinus; the intercavernous sinus; the basal plexus; the superior ophthalmic vein; the lacrimal vein; the anterior and posterior ethmoid veins; frontal nasal vein; subophthalmic vein; facial vein; angular vein; supratrochlear vein; supraorbital vein; palpebral vein; external nasal vein; labial vein; deep facial vein; parotid branch; submental vein; external palatine vein; retromandibular vein; superficial temporal vein; middle temporal vein; transverse facial vein; mandibular joint vein; stylomastoid vein; anterior auricular vein; parotid vein; maxillary vein; external jugular vein; posterior auricular vein; occipital vein; suprascapular vein; transverse jugular vein; anterior jugular vein; pterygoid plexus; sphenopalatine vein; middle meningeal vein; deep temporal vein; superior alveolar vein; masseteric vein; inferior alveolar vein; vertebral vein; anterior vertebral vein; deep cervical vein; internal jugular vein; sigmoid sinus; occipital sinus; hypoglossal canal venous plexus; cochlear canal vein; inferior petrosal sinus; occipital vein; facial vein; pharyngeal vein; lingual vein; deep lingual vein; dorsal lingual vein; hypoglossal nerve Accompanying veins; superior thyroid vein; subfascial vein; deep palmar venous arch; radial vein; ulnar vein; brachial vein; common brachial vein; basilic vein; musculus vein; ulnar collateral vein; deep brachial vein; axillary vein; subclavian vein; cutaneous vein; digital vein; intercarpal vein; digital palmar venous arch; interdigital vein; dorsal digital venous arch; dorsal hand venous network; dorsal palmar vein; dorsal palmar venous arch; marginal vein; dorsal vein of the little finger; vein of the thumb; cephalic vein; cephalic vein of the forearm; accessory cephalic vein; median cubital vein; basilic vein; median forearm vein; median basilic vein; median head vein; superior vena cava; brachiocephalic vein; venous angle; inferior thyroid vein; inferior thyroid vein; vertebral vein; deep cervical vein; external jugular vein; pericardial phrenic vein; thymic vein; pericardial vein;mediastinal veins; bronchial veins; tracheal veins; esophageal veins; internal thoracic veins; muscular septal veins; superior epigastric veins; anterior intercostal veins; perforators; sternal branches; superior intercostal veins; thoracoabdominal cutaneous veins; areolar venous plexus; thoracoabdominal veins; costal axillary veins; azygos veins; hemiazygos veins; extravertebral venous plexus; internal vertebral veins; vertebral veins; intervertebral veins; spinal veins; internal spinal veins; external spinal veins; radicular veins; dorsal veins of the foot; lateral plantar veins; plantar venous arches; medial plantar veins; cardiac veins of the foot; perforating veins; anterior tibial veins; posterior tibial veins; peroneal veins; popliteal veins; genicular veins; Sural vein; small saphenous vein; femoral vein; superficial abdominal wall vein; superficial circumflex iliac vein; thoracic and abdominal wall vein; external pudendal vein; anterior scrotal vein; anterior labial vein; superficial dorsal vein of penis; superficial dorsal vein of clitoris; great saphenous vein; deep femoral vein; perforating vein; medial circumflex femoral vein; lateral circumflex femoral vein; superior gluteal vein; inferior gluteal vein; dorsal digital vein; intermetacarpal vein; dorsal plantar vein; dorsal venous arch of foot; plantar venous arch; plantar venous network; dorsal venous network; saphenous vein; small saphenous vein; inferior vena cava; inferior phrenic vein; lumbar vein; hepatic vein; renal vein; adrenal vein; testicular vein; ovarian vein iliac vein; common iliac vein; median sacral vein; external iliac vein; deep circumflex iliac vein; inferior epigastric vein; internal iliac vein; iliolumbar vein; lateral sacral vein; superior gluteal vein; inferior gluteal vein; obturator vein; internal pudendal vein; deep veins of penis and clitoris; bulbar vein of penis; bulbar vein of vagina; posterior scrotal vein; posterior labial vein; inferior rectal vein; dorsal vein of penis; circumflex vein of penis; dorsal vein of clitoris; superficial dorsal vein of penis; superficial dorsal vein of clitoris; rectal venous plexus; vesical venous plexus; prostatic venous plexus; uterine venous plexus; vaginal venous plexus; portal vein of liver; superior mesenteric vein; intestine and ileum Veins; Ileocolic vein; Appendiceal vein; Right colic vein; Pancreatic vein; Right gastroepiploic vein; Inferior pancreaticoduodenal vein; Splenic vein; Pancreatic vein; Short gastric vein; Left gastroepiploic vein; Inferior mesenteric vein; Sigmoid vein; Left colic vein; Superior rectal vein; Left gastric vein; Right gastric vein; Prepyloric vein; Posterior superior pancreaticoduodenal vein; Gallbladder vein; Umbilical vein; Major duodenal papilla / Vat's papilla; Entrance of cystic duct to common bile duct; Exit of cystic duct from gallbladder; Junction of right and left hepatic ducts; Junction of right anterior and right posterior hepatic ducts; Ridge; Branches of right upper lobe bronchus; branches of the right middle lobe bronchus; Right lower lobe bronchial branches; Left upper lobe bronchial branches; Left lower lobe bronchial branches; The first right bronchus; branches of the second segment of the right bronchus; branches of the third segment of the right bronchus; branches of the fourth segment of the right bronchus; branches of the fifth segment of the right bronchus; Branches of the sixth segment of the right bronchus; branches of the seventh segment of the right bronchus; branches of the eighth segment of the right bronchus; branches of the right ninth segment bronchus; branches of the tenth segment of the right bronchus; Right first segment bronchus; Left first segment bronchus; Left second segment bronchus; Left third segment bronchus; Left fourth segment bronchus; Left fifth segment bronchus; Left sixth segment bronchus; Left seventh segment bronchus; Left eighth segment bronchus; Anterior nares; Right posterior ethmoid cell to outlet / drainage of right sphenoethmoid recess; Left posterior ethmoid cell to outlet / drainage of left sphenoethmoid recess; Right sphenoid sinus to outlet / drainage of right sphenoethmoid recess; Right sphenoid sinus to outlet / drainage of right sphenoethmoid recess; Left sphenoethmoid recess to outlet / drainage of left superior nasal meatus; Right frontal sinus to outlet / drainage of right frontal vestibule; Left frontal sinus to outlet / drainage of left frontal vestibule; Right frontal vestibule to outlet / drainage of right middle nasal meatus; Left frontal vestibule to outlet / drainage of left middle nasal meatus; Right frontal ethmoid Atrium to outlet / drainage of right oro-meatal complex; Left fronto-ethmoid atrium to outlet / drainage of left oro-meatal complex; Right maxillary sinus to outlet / drainage of right oro-meatal complex; Left maxillary sinus to outlet / drainage of left oro-meatal complex; Right nasolacrimal duct to outlet / drainage of right inferior nasal meatus; Left nasolacrimal duct to outlet / drainage of left inferior nasal meatus; Choranium / posterior foramen / ostium; Uvula; Pharyngeal tonsil; Right and left ear canal openings; Tip of epiglottis; Left and right piriform recess / fossa / antrum; Post-cricoid region; Cricoid cartilage; Interarytenoid folds; Right and left aryepiglottic folds; Right and left arytenoid cartilages; Left and right vestibular folds; Left and right laryngeal sac outlets; Vocal cords; Right and left vocal cords; First tracheal cartilage; Kidney; Renal papilla; Renal calyx; Renal pyramid; Renal pelvis; Ureteral inlet; Abducens nerve; Accessory nerve; Accessory obturator nerve; Oldman's nerve; anocaudal nerve; cervical loop; anterior interosseous nerve; anterior superior alveolar nerve; Auerbach's plexus; auriculotemporal nerve; axillary nerve; brachial plexus; buccal branch of facial nerve; buccal nerve; cardiac plexus; cavernous nerve; cavernous plexus; celiac ganglion; cervical branch of facial nerve; cervical plexus; chorda tympani; ciliary ganglion; cochlear nerve; common peroneal nerve; common palmar digital nerve of median nerve; deep radial nerve; deep peroneal nerve; deep temporal nerve; diagonal band of Broca; digastric branch of facial nerve; dorsal branch of ulnar nerve; dorsal nerve of clitoris; dorsal nerve of penis; dorsal nerve of scapula; esophageal plexus; ethmoid nerve; external laryngeal nerve; external nasal nerve; facial nerve; femoral nerve; frontal nerve; gastric plexus; geniculate ganglion; genital branch of genitofemoral nerve; genitofemoral nerve; glossopharyngeal nerve; greater auricular nerve; greater occipital nerve; greater petrosal nerve Nerves; Hepatic plexus; Hypoglossal nerve; Iliohypogastric nerve; Ilioinguinal nerve; Inferior alveolar nerve; Inferior anal nerve; Inferior cardiac nerve; Inferior cervical ganglion; Inferior gluteal nerve; Inferior hypogastric plexus; Inferior mesenteric plexus; Infrapalpebral nerve; Infraorbital nerve; Infraorbital plexus; Infratrochlear nerve; Intercostal nerve; Intercostobrachial nerve; Intermediate cutaneous nerve; Internal cervical plexus; Internal laryngeal nerve; Intermediate neuron; Cervical ganglion; Lacrimal nerve; Lateral cord; Lateral cutaneous nerve of the forearm; Lateral cutaneous nerve of the thigh; Lateral thoracic nerve; Lateral plantar nerve; Lateral pterygoid nerve; Lesser occipital nerve; Lingual nerve; Long ciliary nerve; Long root of ciliary ganglion; Long thoracic nerve; Inferior subscapular nerve; Lumbar nerve; Lumbar plexus; Lumbar splanchnic nerve; Lumboinguinal nerve; Lumbosacral plexus; Lumbosacral trunk; Mandibular nerve; Marginal mandibular branch of facial nerve; Masseteric nerve; Maxillary nerve;medial cord; medial brachial cutaneous nerve; medial forearm cutaneous nerve; medial cutaneous nerve; medial thoracic nerve; medial plantar nerve; median nerve; Meissner's plexus; mental nerve; medial cardiac nerve; middle cervical ganglion; middle meningeal nerve; motor nerve; muscular branch of radial nerve; musculocutaneous nerve; Mylohyoid nerve; nasociliary nerve; nasopalatine nerve; pterygoid nerve; internal obturator nerve; quadratus femoris nerve; piriformis nerve; stapedial nerve; subclavian nerve; nervus intermedius; spinous nerve; nodose ganglion; obturator nerve; oculomotor nerve; Olfactory nerve; ophthalmic nerve; optic nerve; otic ganglion; ovarian plexus; palatine nerve; palmar branch of median nerve; palmar branch of ulnar nerve; pancreatic plexus; patellar plexus; pelvic visceral nerve; perforating nerve; perineal branch of posterior femoral cutaneous nerve; perineal nerve; petrosal ganglion; Pharyngeal branch of vagus nerve; Pharyngeal branch of glossopharyngeal nerve; Ph ... plexus; Phrenic nerve; Phrenic plexus; Posterior auricular nerve; Posterior ramus of spinal nerve; posterior bundle; posterior brachial cutaneous nerve; posterior forearm cutaneous nerve; Posterior thigh cutaneous nerve; posterior scrotal nerve; posterior superior alveolar nerve; palmar digital nerve proper of the median nerve; prostatic plexus (nerve); pterygopalatine ganglion; pudendal nerve; pudendal plexus; pulmonary branch of the vagus nerve; Radial nerve; recurrent laryngeal nerve; renal plexus; sacral plexus; sacral visceral nerve; saphenous nerve; sciatic nerve; semilunar ganglion; sensory nerve; short ciliary nerve; sphenopalatine nerve; splenic plexus; stylohyoid branch of facial nerve; subcostal nerve; submandibular ganglion; suboccipital nerve; superficial radial nerve; superficial peroneal nerve; superior cardiac nerve; superior cervical ganglion; superior glossopharyngeal ganglion; superior vagus ganglion; superior gluteal nerve; Superior hypogastric plexus; superior labial nerve; superior laryngeal nerve; lateral cutaneous nerve of the arm; Superior mesenteric nerve; superior rectal plexus; supraclavicular nerve; supraorbital nerve; adrenal plexus; suprascapular nerve; supratrochlear nerve; sural nerve; sympathetic trunk; temporal branch of facial nerve; third occipital nerve; thoracic aortic plexus; thoracic splanchnic nerves; thoracic abdominal nerves; thoracodorsal nerves; Tibial nerve; transverse cervical nerve; trigeminal nerve; trochlear nerve; tympanic nerve; ulnar nerve; superior subscapular nerve; uterovaginal plexus; vagus nerve; ventral branch; vesical plexus; vestibular nerve; vestibulocochlear nerve; zygomatic branch of facial nerve; zygomatic nerve; zygomaticofacial nerve; zygomaticotemporal nerve; foramen caecum; optic canal; superior orbital fissure; foramen rotundum; foramen ovale; foramen spinosum; foramen hiatus; carotid canal; internal auditory foramen; jugular foramen; hypoglossal canal; foramen magnum.

28. A computer-readable [storage] medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method defined in any one of claims 1 to 27.

29. A system for detecting landmarks in a medical image volume, wherein the system comprises: - a measuring device for collecting a computed tomography volume or a magnetic resonance volume of a human patient, and - A computer system adapted to perform the steps of the method as defined in any of claims 1-27.

30. The system of claim 29, wherein: The computer system is adapted to perform the steps of the method as defined in any of claims 14-27.

Images