A Deep Learning-Based Image Segmentation Method and System for Auricular Cartilage and its Substructures

Abstract

This invention relates to a deep learning-based image segmentation method and system for auricular cartilage and its substructures. The image segmentation method includes: acquiring a magnetic resonance imaging (MRI) image of the external ear contour using a UTE sequence; preprocessing the acquired MRI image of the external ear contour using the UTE sequence; and training a deep learning-based segmentation model for auricular cartilage and its substructures using manually segmented auricular cartilage images and manually segmented auricular cartilage substructure images as label images, thereby segmenting the auricular cartilage and its substructures respectively. The deep learning-based image segmentation method and system for auricular cartilage and its substructures can automatically and efficiently acquire auricular cartilage models suitable for 3D bioprinting.

Description

Technical Field

[0001] This invention relates to the technical field of medical image processing, and in particular to a method and system for image segmentation of auricular cartilage and its substructures based on deep learning. Background Technology

[0002] Congenital maxillofacial malformations are a common type of congenital maxillofacial deformity, with an incidence rate of 5.18 / 10,000 in China, according to epidemiological statistics. This condition typically manifests as severe auricular hypoplasia, significantly impacting the patient's appearance, function, and mental health.

[0003] Currently, reconstructing the auricle using autologous rib cartilage sculpted scaffolds is one of the main plastic surgical treatments for congenital pediatric malformations. However, this procedure causes additional harm to the child during rib cartilage harvesting, and the aesthetic outcome largely depends on the surgeon's experience. With the development of tissue engineering and 3D bioprinting technologies, this method, which allows for precise allocation of cells, matrix, and biomaterials, can not only improve the construction of auricular cartilage scaffolds but also precisely control their structure. 3D bioprinting completes the scaffold construction task by layer-by-layer histological assembly based on a three-dimensional model designed using biomedical images. However, how to automatically and efficiently obtain auricular cartilage models suitable for 3D bioprinting remains a challenge.

[0004] The above description of the background technology is only for the purpose of facilitating a deeper understanding of the technical solution of the present invention (the technical means used, the technical problems solved, and the technical effects produced, etc.), and should not be regarded as an admission or in any form an implication that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] To address the practical problems existing in the prior art, this invention provides a solution for constructing a morphological structure model of ear cartilage using magnetic resonance imaging, suitable for tissue engineering and 3D bioprinting.

[0006] According to one embodiment of the present invention, a deep learning-based image segmentation method for auricular cartilage and its substructures is provided, comprising the following steps: acquiring magnetic resonance images of the UTE sequence of the outer ear contour; preprocessing the acquired magnetic resonance images of the UTE sequence of the outer ear contour; using the preprocessed magnetic resonance images of the UTE sequence as input, and using manually segmented auricular cartilage images and manually segmented auricular cartilage substructure images as label images respectively, training a deep learning-based segmentation model for auricular cartilage and its substructures, thereby segmenting the auricular cartilage and auricular cartilage substructures respectively.

[0007] Preferably, the deep learning-based segmentation model for auricular cartilage and its substructures includes a segmentation network with an encoder-decoder U-shaped framework as its backbone network; the segmentation network includes an encoder and a decoder; the segmentation network has a residual structure at each layer of the encoder, and the skip connections in the residual structure skip one or more convolutional layers to perform identity mapping; the segmentation network uses convolutional pooling instead of pooling operations, and uses convolutional kernels to convolve the feature map; the segmentation network has multi-scale fusion output; the segmentation network assigns weights to each auricular cartilage substructure based on its proportion in the auricular cartilage image; the segmentation network uses a dense conditional random field and a weighted Dice loss function to optimize substructure segmentation.

[0008] Preferably, the deep learning-based segmentation model for auricular cartilage substructures further includes a localization operation, which uses manually segmented auricular cartilage images and manually segmented auricular cartilage substructure images as masks, determines the image edge positions from six directions, thereby locating the auricular cartilage in the preprocessed UTE sequence magnetic resonance image and further cropping it. After localization, it is multiplied with the mask to further remove the brain bone region in the image; the image after localization is multiplied with the preprocessed UTE sequence magnetic resonance image, and then the resulting image is input into the segmentation network to train the segmentation network.

[0009] Preferably, when acquiring the magnetic resonance image of the UTE sequence of the outer ear contour, the echo time is set to 0-1ms depending on the MRI scanning equipment used; the magnetic resonance image of the UTE sequence includes axial image, sagittal image and coronal image; the preprocessing of the acquired magnetic resonance image of the UTE sequence of the outer ear contour includes: format conversion, renaming, reorientation and rigid registration of the image.

[0010] Preferably, the manually segmented auricular cartilage image is obtained based on the operator's manual operation; when segmenting the auricular cartilage image based on the operator's manual operation, the auricular cartilage, surrounding fat and connective tissue are all presented as bright signals to varying degrees in the preprocessed UTE sequence magnetic resonance image, while the skin and other tissues are relatively dark, thereby segmenting the boundary of the auricular cartilage.

[0011] Preferably, the manually segmented auricular cartilage substructure images are obtained through manual operation by the operator; when segmenting the auricular cartilage substructure images according to the operator's manual operation, a total of twelve auricular cartilage substructure images are segmented in the following order: segmentation of the antihelix, triangular fossa and crus of the antihelix, segmentation of the helix and crus of the helix, segmentation of the cymba conchae, segmentation of the scaphoid fossa, segmentation of the tragus, segmentation of the antitragus and intertragusal notch, and segmentation of the external auditory canal and conchae cavity.

[0012] According to one embodiment of the present invention, a deep learning-based image segmentation system for auricular cartilage and its substructures is provided, comprising the following modules: an acquisition module, which acquires magnetic resonance images of the UTE sequence of the outer ear contour; a preprocessing module, which preprocesses the acquired magnetic resonance images of the UTE sequence of the outer ear contour; and a training module, which takes the preprocessed magnetic resonance images of the UTE sequence as input, and uses manually segmented auricular cartilage images and manually segmented auricular cartilage substructure images as label images to train a deep learning-based segmentation model for auricular cartilage and its substructures, thereby segmenting the auricular cartilage and auricular cartilage substructures respectively.

[0013] Preferably, the deep learning-based segmentation model for auricular cartilage and its substructures includes a segmentation network with an encoder-decoder U-shaped framework as its backbone network; the segmentation network includes an encoder and a decoder; the segmentation network has a residual structure at each layer of the encoder, and the skip connections in the residual structure skip one or more convolutional layers to perform identity mapping; the segmentation network uses convolutional pooling instead of pooling operations, and uses convolutional kernels to convolve the feature map; the segmentation network has multi-scale fusion output; the segmentation network assigns weights to each auricular cartilage substructure based on its proportion in the auricular cartilage image; the segmentation network uses a dense conditional random field and a weighted Dice loss function to optimize substructure segmentation.

[0014] Preferably, the deep learning-based segmentation model for auricular cartilage and its substructures further includes a localization operation, which uses manually segmented auricular cartilage images and manually segmented auricular cartilage substructure images as masks, determines the image edge positions from six directions, thereby locating the auricular cartilage in the preprocessed UTE sequence magnetic resonance image and further cropping it. After localization, it is multiplied with the mask to further remove the brain bone region in the image; the image after localization is multiplied with the preprocessed UTE sequence magnetic resonance image, and then the resulting image is input into the segmentation network to train the segmentation network.

[0015] Preferably, the manually segmented auricular cartilage image is obtained based on the operator's manual operation; when segmenting the auricular cartilage image based on the operator's manual operation, based on the fact that in the preprocessed UTE sequence magnetic resonance image, the auricular cartilage, surrounding fat, and connective tissue all appear as bright signals to varying degrees, while the skin and other tissues are relatively dark, thus segmenting the auricular cartilage boundary; the manually segmented auricular cartilage substructure image is obtained based on the operator's manual operation; when segmenting the auricular cartilage substructure image based on the operator's manual operation, a total of twelve auricular cartilage substructure images are segmented in the following order: antihelix, triangular fossa and crus of antihelix, helix and crus of antihelix, cymba conchae, scaphoid fossa, tragus, antitragus and intertragusal notch, and external auditory canal and conchae cavity.

[0016] The present invention adopts the above technical solution, which has the following beneficial effects:

[0017] Based on ultrashort echo time-series imaging of auricular cartilage and its manual segmentation results, this invention uses a deep learning network to obtain a model that can automatically segment auricular cartilage and its substructures. This method and system provide an artificial intelligence solution for automatically and efficiently obtaining auricular cartilage models that can be used for 3D bioprinting. Attached Figure Description

[0018] The exemplary embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. For clarity, the same components in different drawings are shown with the same reference numerals. It should be noted that the drawings are for illustrative purposes only and are not necessarily drawn to scale. In these drawings:

[0019] Figure 1 This is a flowchart of an image segmentation method and system for auricular cartilage and its substructures based on deep learning, according to an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram illustrating the preprocessing process of the image segmentation method and system for auricular cartilage and its substructures based on deep learning according to an embodiment of the present invention.

[0021] Figures 3a to 3d This is a schematic diagram of the preprocessing process for magnetic resonance images obtained from UTE sequences;

[0022] Figures 4a to 4d An image of auricular cartilage segmented from a magnetic resonance imaging (MRI) image based on a UTE sequence, according to one embodiment of the present invention, is shown.

[0023] Figures 5a to 5d An image of auricular cartilage segmented from a magnetic resonance imaging (MRI) image based on a UTE sequence, according to one embodiment of the present invention, is shown.

[0024] Figure 6 A flowchart illustrating the manual segmentation of auricular cartilage substructures according to one embodiment of the present invention is shown;

[0025] Figure 7a and Figure 7b The image shows a segmented auricular cartilage image based on a UTE sequence and a schematic diagram of auricular cartilage substructures;

[0026] Figure 8 A schematic diagram of the architecture of a deep learning-based segmentation model for auricular cartilage according to an embodiment of the present invention is shown.

[0027] Figure 9a and Figure 9bA schematic diagram of the residual structure of the segmentation network in a deep learning-based segmentation model of auricular cartilage and its substructures according to an embodiment of the present invention is shown.

[0028] Figure 10 A schematic diagram of multi-scale fusion of segmentation networks in a deep learning-based segmentation model of auricular cartilage and its substructures according to an embodiment of the present invention is shown.

[0029] Figure 11a and Figure 11b as well as Figure 12a and Figure 12b A 3D visualization of ear cartilage segmentation using a deep learning network and manual segmentation is shown, using two examples of data.

[0030] Figure 13 An architecture diagram of a deep learning-based segmentation model for ear cartilage substructures according to an embodiment of the present invention is shown;

[0031] Figure 14a and Figure 14b as well as Figure 15a and Figure 15b The diagram shows a 3D representation of deep learning network segmentation and manual segmentation of ear cartilage substructure using two examples. Detailed Implementation

[0032] The following provides a detailed description of the embodiments of the present invention. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0033] Figure 1 This is a flowchart illustrating the image segmentation method and system for auricular cartilage and its substructures based on deep learning according to an embodiment of the present invention. The image segmentation method for auricular cartilage and its substructures based on deep learning according to the present invention mainly consists of three steps or modules.

[0034] The acquisition step (module) acquires magnetic resonance images of the UTE sequence of the outer ear contour.

[0035] The preprocessing step (module) preprocesses the magnetic resonance images of the UTE sequence of the acquired outer ear contour.

[0036] The training step (module) takes the preprocessed UTE sequence magnetic resonance image as input, and uses manually segmented auricular cartilage images and manually segmented auricular cartilage substructure images as label images to train a deep learning-based segmentation model of auricular cartilage and its substructures, thereby segmenting the auricular cartilage and auricular cartilage substructures respectively.

[0037] The manually segmented images of auricular cartilage and its substructures were obtained through manual manipulation by operators. Operators in this paper may include doctors, nurses, radiologists, etc.

[0038] The following describes in detail each step (module) of the image segmentation method and system for auricular cartilage and its substructures based on deep learning of the present invention.

[0039] Magnetic resonance imaging (MRI) is an imaging modality that utilizes no ionizing radiation, provides excellent soft tissue contrast, and offers a rich array of imaging sequences. The signal intensity of the transverse relaxation time (TE) of auricular cartilage falls between that of muscle or adipose tissue and bone tissue. It exhibits exponential decay within both ultra-short echo time (UTE) (<1 ms) and short echo time (1–10 ms), with particularly dramatic decay within the UTE range. Outside of long echo time (>10 ms), the signal intensity change tends to level off. In other words, setting the echo time (TE) below the short echo time range results in better enhancement of auricular cartilage imaging. In embodiments of this invention, when acquiring the UTE sequence, the TE can be set to 0–1 ms, preferably 0.1–0.5 ms, depending on the MRI scanner used.

[0040] Table 1 below shows exemplary parameter selections for magnetic resonance images of the UTE sequence of the acquired outer ear contour, but the present invention is not limited thereto.

[0041] [Table 1]

[0042] parameter UTE sequence Repeat time TR (ms) 5.58 Echo time TE (ms) 0.14 Flip angle 15 Layer thickness Th (mm) 1.2 Slice spacing (mm) 0.6 In-plane resolution (mm) 0.25×0.25 Matrix size 800×800×80

[0043] Figure 2 The present invention illustrates the process of preprocessing magnetic resonance images of acquired UTE sequences using an image segmentation method and system based on deep learning for auricular cartilage and its substructures according to the present invention.

[0044] like Figure 2 As shown, the image segmentation method and system for auricular cartilage and its substructures based on deep learning according to the present invention can be applied to magnetic resonance images of UTE sequences of the outer ear contour. The data types can include, but are not limited to, DICOM (Digital Imaging and Communications in Medicine) and NIFTI (Neuroimaging Informatics Technology Initiative).

[0045] Preprocessing of the acquired UTE sequence magnetic resonance images of the outer ear contour includes: format conversion, renaming, reorientation, and rigid registration of the images.

[0046] According to an embodiment of the present invention, if the usable data is in DICOM format, it is converted to NIFTI format; otherwise, no format conversion is performed. Rotation involves turning the image to a standard space and standardizing the voxel size. Rigid registration involves selecting an image with a clear auricular structure and appropriate size as the fixed image, and using the remaining images to be preprocessed as moving images, which are then rotated and translated to the same position as the fixed image.

[0047] Figures 3a to 3d This is a schematic diagram of the preprocessing process for magnetic resonance images obtained from UTE sequences. Figures 3a to 3d The images shown are: the original UTE sequence magnetic resonance images (axial, sagittal, and coronal from left to right); the images obtained after format conversion (axial, sagittal, and coronal from left to right); the images obtained after orientation (axial, sagittal, and coronal from left to right); and the images obtained after rigid registration (i.e., the images obtained after preprocessing, axial, sagittal, and coronal from left to right).

[0048] For the generation of labeled images of auricular cartilage and its substructures, this invention utilizes a manual method to segment the labeled images of auricular cartilage and its substructures. According to the embodiments of this invention, high-resolution imaging of auricular cartilage can be achieved using a combination of multiple sequences, primarily ultra-shortwave sequences, based on magnetic resonance imaging (MRI) technology. This allows for manual segmentation of the auricular cartilage image, and further, more detailed segmentation of substructures can be performed based on the segmented auricular cartilage.

[0049] In the magnetic resonance images of the UTE sequence, cartilage, surrounding fat, and connective tissue all appear as bright signals to varying degrees, while skin and other tissues appear relatively dark, thus allowing the boundaries of the ear cartilage to be fully delineated.

[0050] Therefore, based on the imaging effect of UTE sequence magnetic resonance images of auricular cartilage and earlobe, the auricular cartilage image can be manually segmented.

[0051] Figures 4a to 4d and Figures 5a to 5d Two examples are shown below, illustrating ear cartilage images segmented from magnetic resonance images based on UTE sequences according to one embodiment of the present invention. Figure 4a and Figure 5a This shows an axial image of manually segmented auricular cartilage. Figure 4b and Figure 5b This shows a sagittal image of manually segmented auricular cartilage; Figure 4c and Figure 5c This shows a coronal image of manually segmented auricular cartilage; Figure 4d and Figure 5d A three-dimensional schematic diagram of manually segmented ear cartilage is shown.

[0052] Therefore, according to an embodiment of the present invention, scanning can be performed using a UTE sequence to achieve high-definition, non-invasive imaging of the auricular cartilage.

[0053] In addition, various sequences, including 3D-T2W, PDW, and PROSET, can be acquired. Preprocessing of MRI images from 3D-T2W, PDW, and PROSET sequences is similar to that from UTE sequences and will not be repeated here. In 3D-T2W sequences, the auricular cartilage appears as a dark signal, while the earlobe (bright signal) is clearly distinguishable from surrounding tissues, making it useful for identifying the earlobe's location. Combined with UTE, the auricular cartilage can be completely segmented. PROSET sequences have a low signal-to-noise ratio, with the auricular cartilage appearing as a bright signal and the earlobe as a dark signal, making them suitable for earlobe segmentation. PDW sequences have low tissue contrast but a high signal-to-noise ratio, making them suitable for segmenting the outer ear.

[0054] In the following text, the segmented auricular cartilage image will be further segmented into twelve substructures, namely the antihelix, crus of the antihelix, triangular fossa, helix, crus of the helix, cymba conchae, scaphoid fossa, tragus, antitragus, intertragic notch, external auditory canal, and conchae cavity.

[0055] According to an embodiment of the present invention, such as Figure 6 As shown, the segmentation of the auricular cartilage substructures mainly follows six steps: segmentation of the antihelix, triangular fossa and crus of the antihelix, segmentation of the helix and crus of the helix, segmentation of the cymba conchae, segmentation of the scaphoid fossa, segmentation of the tragus, segmentation of the antitragus and intertragic notch, and segmentation of the external auditory canal and cavity of the conchae. Specifically, the crus of the helix is ​​used as the dividing line to divide the auricular cartilage substructures into upper and lower halves. In the segmentation of the upper half, the boundaries of the two important mechanical support structures, the helix and antihelix, are clearly defined to ensure the correct segmentation of the remaining substructures. Then, based on the anatomical definition of each structure and its corresponding features in the label, combined with MRI images, twelve substructures are segmented sequentially: antihelix, crus of the antihelix, triangular fossa, helix, crus of the helix, cymba conchae, scaphoid fossa, tragus, antitragus, intertragic notch, external auditory canal, and cavity of the conchae. Steps one through four are the segmentation steps for the upper half of the auricular cartilage substructures, while steps five and six are the segmentation steps for the lower half.

[0056] Figure 7a and Figure 7bThe diagram illustrates an image of auricular cartilage segmented from a magnetic resonance imaging (MRI) image based on a UTE sequence, along with a schematic diagram of auricular cartilage substructures. According to an embodiment of the present invention, high-resolution imaging of auricular cartilage can be achieved using a combination of multiple sequences, primarily UTE, based on MRI technology. This allows for manual segmentation of the auricular cartilage image, and further, manual segmentation of fine substructures based on the segmented auricular cartilage.

[0057] According to an embodiment of the present invention, manually segmented images of auricular cartilage and its substructures can be used as label images to train a three-dimensional deep learning network based on encoding and decoding structures, thereby obtaining a deep learning network model that can automatically segment auricular cartilage and its substructures.

[0058] Figure 8 A schematic diagram of the architecture of a deep learning-based segmentation model for ear cartilage according to one embodiment of the present invention is shown. Figure 8 As shown, a deep learning-based segmentation model for auricular cartilage and its substructures can include a localization operation and a segmentation network. The segmentation network can also be used independently to perform the same task.

[0059] The localization operation is performed using preprocessed UTE sequence images of ear cartilage. The localized image is then multiplied by the preprocessed UTE sequence magnetic resonance image, and the resulting image is input into the segmentation network to train the segmentation network.

[0060] The localization operation further locates the preprocessed image, allowing for image cropping to reduce the background image. The localization operation uses manually segmented auricular cartilage images and manually segmented auricular cartilage substructure images as masks, determining the image edge positions from six directions. This allows for the localization of the auricular cartilage in the preprocessed UTE sequence MRI image and further cropping. After localization, a dot product is performed with the mask to further remove the cranial bone region from the image.

[0061] The segmentation network structure in the deep learning-based segmentation model of auricular cartilage and its substructures according to an embodiment of the present invention is a U-shaped architecture of encoding and decoding. To address the complexity of auricular cartilage structures (e.g., complex and blurred edges, varied shapes, and small size), this invention improves the 3D-Unet network model by introducing residual modules in the encoding layer, using convolution instead of pooling, and incorporating multi-scale fusion. This allows the entire image to be used as input and output, and also improves optimization, further enhancing segmentation accuracy.

[0062] Figure 9a and Figure 9bA schematic diagram of the residual structure of a segmentation network in a deep learning-based segmentation model of ear cartilage and its substructures according to an embodiment of the present invention is shown. The segmentation network according to the embodiment of the present invention adds a residual structure to each encoding layer. The "skip connections" in the residual structure can skip one or more convolutional layers to perform identity mappings. This avoids problems such as gradient vanishing caused by network deepening, allowing for better network optimization. The residual structure includes two input layers (IN), two LReLU layers, two convolutional layers (Conv), and a random dropout layer. To accelerate network convergence, normalization is also added to the convolutional layers. Introducing a random dropout layer into the residual module deactivates some neurons during training, avoiding parameter overfitting and thus improving the network's generalization performance.

[0063] In the deep learning-based segmentation model of auricular cartilage and its substructures according to an embodiment of the present invention, the segmentation network further uses convolutional pooling instead of pooling operations. Convolutional kernels are used to convolve the feature maps; preferably, the stride size can be set to 2 to achieve the pooling effect. Convolutional pooling can maintain network performance while consuming less memory during training.

[0064] The segmentation network in the deep learning-based segmentation model of auricular cartilage and its substructures according to an embodiment of the present invention also has multi-scale fusion output. Figure 10 This diagram illustrates a multi-scale fusion of a segmentation network in a deep learning-based segmentation model of auricular cartilage and its substructures according to an embodiment of the present invention. At output, feature maps from multiple decoding layers are fused to optimize the output, yielding more feature information. The final result is obtained through multi-scale fusion. Figure 10 As shown, the segmentation network of the present invention integrates the results of the last three decoding layers, and can also integrate the results of more layers.

[0065] When using a deep learning-based segmentation model of auricular cartilage and its substructures to segment only a single auricular cartilage image, the preprocessed UTE sequence of the auricular cartilage image can be used as the input of the segmentation model, and the manually segmented auricular cartilage image can be used as the auricular cartilage label image to train the segmentation model of the present invention, thereby obtaining a segmentation model of the auricular cartilage image.

[0066] Figure 11a and Figure 11b as well as Figure 12a and Figure 12b A 3D visualization of ear cartilage segmentation using a deep learning network and manual segmentation is shown, using two examples. Figure 11a and Figure 12a A 3D representation of manually segmented ear cartilage; Figure 11b and Figure 12b A 3D representation of a deep learning network segmenting ear cartilage.

[0067] In this invention, the Dice similarity coefficient (DSC) is used to evaluate the results of auricular cartilage segmentation.

[0068]

[0069] The calculated DSC (Digital Subtraction Function) index for ear cartilage segmentation using a deep learning network is 0.86.

[0070] When using a deep learning-based segmentation model for auricular cartilage and its substructures to segment auricular cartilage substructure images, the preprocessed UTE sequence of the auricular cartilage image is used as the input to the segmentation model, and manually segmented auricular cartilage substructure images are used as auricular cartilage substructure label images. For the segmentation of auricular cartilage substructure images, this invention further improves the segmentation model for auricular cartilage images, such as... Figure 13 As shown, it illustrates the architecture of a deep learning-based segmentation model for ear cartilage substructures according to an embodiment of the present invention.

[0071] In an embodiment of the present invention, the architecture diagram of the deep learning-based segmentation model for auricular cartilage substructures according to the embodiment of the present invention is based on the architecture diagram of the auricular cartilage segmentation model, and adds a multi-structure weighted DICE loss function and a dense conditional random field (Dense CRF).

[0072] Each substructure was assigned a weight based on its proportion in the ear cartilage image. The background was given a very small weight to reduce its influence, while smaller structures were given a larger weight. The weighted DICE loss function, which addresses the inter-class imbalance problem, was used to enable the model to focus more on segmenting smaller structures.

[0073] Furthermore, since adjacent voxels share a large amount of spatial context information, the segmentation results produced by the network are often smooth. However, local minima during training and noise in the input image can still lead to some spurious outputs, with small isolated regions, holes, or discontinuities in the prediction. Therefore, this invention utilizes a dense conditional random field (Dense CRF) as a post-processing step to achieve more structured predictions.

[0074] Figure 14a and Figure 14b as well as Figure 15a and Figure 15b The image shows a 3D representation of deep learning network segmentation and manual segmentation of the ear cartilage substructure, using two examples. Figure 14a and Figure 15a A 3D representation of manually segmented ear cartilage substructure; Figure 15b and Figure 15b A 3D representation of a deep learning network segmenting the substructure of the ear cartilage.

[0075] In this invention, the Dice similarity coefficient (DSC) and the 95% Hausdorff surface distance (HSD95) can be used to evaluate the segmentation results of auricular cartilage substructures. The HSD95 is calculated as follows:

[0076]

[0077] For a given structure, S is the set of points representing that structure in the segmentation graph, and R is the set of points representing that structure in the standard annotation graph. and These are the corresponding sets of edge points. Where |V| represents the number of points in a set V, and d... m (v,V) represents the minimum Euclidean distance between point v and all points in the point set V, infinity. 5％,v∈V (d m ) represents the top 5% of d values ​​within the point set V. m The lower bound of the value.

[0078] The calculation results of the deep learning network for ear cartilage substructure of the present invention for segmenting different parts of ear cartilage substructure are shown in Table 2 below.

[0079] The deep learning-based segmentation model for auricular cartilage and its substructures of this invention can achieve a segmentation accuracy of nearly 90% for auricular cartilage and an average segmentation accuracy of nearly 80% for auricular cartilage substructures. The segmentation results can be directly converted into STL format models for general 3D printing and 3D bioprinting.

[0080] The deep learning-based segmentation model for auricular cartilage and its substructures of this invention can be trained on the entire existing image. The initial convolutional kernel weights are obtained from a Gaussian distribution, and then the model is optimized using a stochastic gradient descent optimizer. The designed weight Dice coefficients are applied to the final feature map to calculate the loss function. With the continuous addition of new data, the network can be optimized and trained to improve its generalization and robustness, and the accuracy of the results will be further improved.

[0081] [Table 2]

[0082] Ear cartilage structure DSC HSD95 1 Helix 0.90 1.32 2 ear boat 0.79 2.70 3 pairs of helix legs 0.76 2.04 4 pairs of helixes 0.90 1.91 5 triangular sockets 0.76 1.62 6 tragus 0.84 2.09 7. Intertragus notch 0.90 1.62 8 pairs of tragus 0.78 1.71 9 helix feet 0.67 2.74 10-ear armored boat 0.84 2.03 11. Concha 0.71 4.33 12 External auditory canal 0.81 2.91

[0083] Although the exemplary methods of the present invention described above are represented as a series of operations for clarity of description, they are not intended to limit the order of execution of the steps, and each step may be performed simultaneously or in a different desired order. To implement the method according to the invention, the steps shown may further include other steps, including steps other than certain steps, or additional steps other than certain steps.

[0084] The various embodiments of the present invention are not an exhaustive list of all possible combinations, but are intended to describe representative aspects of the invention, and the contents described in the various embodiments can be applied independently or in two or more combinations.

[0085] Furthermore, various embodiments of the present invention can be implemented through hardware, firmware, software, or a combination thereof. The hardware can be implemented using one or more of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a general-purpose processor, a controller, a microcontroller, a microprocessor, etc.

[0086] The scope of this invention is intended to include software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) and non-volatile computer-readable media, the software or machine-executable instructions causing operations according to various embodiments to be performed on a device or computer, and the non-volatile computer-readable media being executable on a device or computer storing such software or instructions, etc.

[0087] The descriptions of the exemplary embodiments presented above are merely illustrative of the technical solutions of the present invention and are not intended to be exhaustive or to limit the invention to the precise forms described. Obviously, those skilled in the art can make many changes and variations based on the above teachings. The exemplary embodiments were chosen and described to explain the specific principles of the invention and its practical applications, thereby enabling others skilled in the art to understand, implement, and utilize the various exemplary embodiments of the invention and their various alternatives and modifications. The scope of protection of the present invention is intended to be defined by the appended claims and their equivalents.

Claims

1. A deep learning-based image segmentation method for auricular cartilage and its substructures, characterized in that, Includes the following steps: Obtain magnetic resonance images of the UTE sequence of the outer ear contour; The magnetic resonance images of the UTE sequence of the acquired outer ear contour were preprocessed; Using preprocessed UTE sequence magnetic resonance images as input, and manually segmented auricular cartilage images and manually segmented auricular cartilage substructure images as label images, a deep learning-based segmentation model for auricular cartilage and its substructures is trained to segment the auricular cartilage and its substructures respectively. Among them, the segmentation model of auricular cartilage and its substructures based on deep learning includes a segmentation network, which uses an encoder-decoder U-shaped frame as the backbone network; The segmentation network includes an encoder and a decoder; The segmentation network has a residual structure in each layer of the encoder, and the skip connections in the residual structure skip one or more convolutional layers to perform identity mapping. The segmentation network uses convolutional pooling instead of pooling operations, and uses convolutional kernels to convolve the feature maps. The segmentation network has multi-scale fusion output; The segmentation network assigns weights based on the proportion of each auricular cartilage substructure in the auricular cartilage image. The segmentation network utilizes a dense conditional random field and a weighted Dice loss function to optimize substructure segmentation.

2. The image segmentation method for auricular cartilage and its substructures based on deep learning according to claim 1, characterized in that, The deep learning-based segmentation model for auricular cartilage and its substructures further includes a localization operation. It uses manually segmented auricular cartilage images and manually segmented auricular cartilage substructure images as masks, determines the image edge positions from six directions, thereby locating the auricular cartilage in the preprocessed UTE sequence magnetic resonance image and further cropping it. After localization, it multiplies with the mask to further remove the brain bone region in the image. The image after localization is multiplied by the magnetic resonance image of the preprocessed UTE sequence, and then the resulting image is input into the segmentation network to train the segmentation network.

3. The image segmentation method for auricular cartilage and its substructures based on deep learning according to claim 1, characterized in that, When acquiring magnetic resonance images of the UTE sequence of the outer ear contour, the echo time is set to 0-1ms depending on the MRI scanning equipment used; The magnetic resonance images of the UTE sequence include axial images, sagittal images, and coronal images; Preprocessing of the acquired UTE sequence magnetic resonance images of the outer ear contour includes: format conversion, renaming, reorientation, and rigid registration of the images.

4. The image segmentation method for auricular cartilage and its substructures based on deep learning according to claim 1, characterized in that, The manually segmented auricular cartilage images were obtained based on manual operation by the operator. When segmenting auricular cartilage images based on manual operation by the operator, the boundaries of auricular cartilage are segmented based on the fact that in the preprocessed UTE sequence magnetic resonance images, auricular cartilage, surrounding fat and connective tissue all appear as bright signals to varying degrees, while skin and other tissues are relatively dark.

5. The image segmentation method for auricular cartilage and its substructures based on deep learning according to claim 1, characterized in that, The manually segmented auricular cartilage substructure images were obtained based on manual operation by the operator. When segmenting the auricular cartilage substructure images according to the operator's manual operation, a total of twelve auricular cartilage substructure images are segmented in the following order: antihelix, triangular fossa and crus of antihelix, helix and crus of helix, cymba conchae, scaphoid fossa, tragus, antitragus and intertragusal notch, and external auditory canal and conchae cavity.

6. An image segmentation system for auricular cartilage and its substructures based on deep learning, characterized in that, Includes the following modules: The acquisition module acquires magnetic resonance images of the UTE sequence of the outer ear contour; The preprocessing module preprocesses the magnetic resonance images of the UTE sequence of the acquired outer ear contour. The training module takes preprocessed UTE sequence magnetic resonance images as input and uses manually segmented auricular cartilage images and manually segmented auricular cartilage substructure images as label images to train a deep learning-based segmentation model for auricular cartilage and its substructures, thereby segmenting the auricular cartilage and its substructures respectively. Among them, the segmentation model of auricular cartilage and its substructures based on deep learning includes a segmentation network, which uses an encoder-decoder U-shaped frame as the backbone network; The segmentation network includes an encoder and a decoder; The segmentation network has a residual structure in each layer of the encoder, and the skip connections in the residual structure skip one or more convolutional layers to perform identity mapping. The segmentation network uses convolutional pooling instead of pooling operations, and uses convolutional kernels to convolve the feature maps. The segmentation network has multi-scale fusion output; The segmentation network assigns weights based on the proportion of each auricular cartilage substructure in the auricular cartilage image. The segmentation network utilizes a dense conditional random field and a weighted Dice loss function to optimize substructure segmentation.

7. The image segmentation system for auricular cartilage and its substructures based on deep learning according to claim 6, characterized in that, The deep learning-based segmentation model for auricular cartilage and its substructures further includes a localization operation. It uses manually segmented auricular cartilage images and manually segmented auricular cartilage substructure images as masks, determines the image edge positions from six directions, thereby locating the auricular cartilage in the preprocessed UTE sequence magnetic resonance image and further cropping it. After localization, it multiplies with the mask to further remove the brain bone region in the image. The image after localization is multiplied by the magnetic resonance image of the preprocessed UTE sequence, and then the resulting image is input into the segmentation network to train the segmentation network.

8. The image segmentation system for auricular cartilage and its substructures based on deep learning according to claim 6, characterized in that, The manually segmented auricular cartilage images were obtained based on manual operation by the operator. When segmenting auricular cartilage images based on manual operation by the operator, the auricular cartilage, surrounding fat and connective tissue are presented as bright signals to varying degrees in the magnetic resonance images of the preprocessed UTE sequence, while the skin and other tissues are relatively dark, thus segmenting the boundaries of the auricular cartilage. The manually segmented auricular cartilage substructure images were obtained based on manual operation by the operator. When segmenting the auricular cartilage substructure images according to the operator's manual operation, a total of twelve auricular cartilage substructure images are segmented in the following order: antihelix, triangular fossa and crus of antihelix, helix and crus of helix, cymba conchae, scaphoid fossa, tragus, antitragus and intertragusal notch, and external auditory canal and conchae cavity.

Images