An inner ear labyrinth multi-level labeling pseudo-label generation and segmentation method based on spatial position structure prior

Abstract

This invention relates to a method for generating and segmenting multi-level pseudo-labels for the inner ear labyrinth based on prior spatial location and structure information, belonging to the field of medical image processing. The method proposes a technique for generating multi-level target structure pseudo-labels on the central osseous line based on prior information about the spatial location and structure of the inner ear labyrinth. This generates pseudo-labels comparable to those manually annotated by professional physicians, serving as the training and validation set for an automatic instance segmentation network with multi-level annotation. Leveraging existing semantic segmentation networks that have manually annotated a large number (68 cases) of single structures in the inner ear labyrinth, a small-sample (10 cases) initial model for multi-level segmentation of the inner ear labyrinth is achieved, enabling automatic segmentation of the horizontal semicircular canals, superior semicircular canals, posterior semicircular canals, vestibule, and cochlea within the inner ear labyrinth. Then, based on an active learning paradigm, the most valuable samples in the initial model are selected to update and optimize the initial multi-level segmentation model.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing, and specifically relates to a method for automatic segmentation of the multi-level structure of the inner ear labyrinth by combining prior information on spatial location and structure. Background Technology

[0002] The inner ear, also known as the inner ear labyrinth, contains auditory and balance receptors. It is functionally important and structurally complex, comprising the three semicircular canals, the vestibule, and the cochlea. Bone lesions of the inner ear can cause sensorineural hearing loss, vestibular vertigo, and other conditions. In ear diseases, approximately 40% of vertigo patients have pathological changes located in the vestibular system of the inner ear. CT is the primary means of visualizing the bony structures of the ear and their lesions. Traditional CT has a spatial resolution of 0.3–0.6 mm, which is insufficient for displaying numerous anatomical structures closely related to major ear diseases, such as the vestibular aqueduct, which are 0.1–1 mm in size. This makes them difficult for the human eye to identify and even more difficult to accurately quantify, posing significant challenges to clinical diagnosis, otological research, and automated computer analysis. Ultra-High Resolution CT (U-HRCT) equipment achieves a minimum voxel resolution of 0.05 mm × 0.05 mm × 0.05 mm. Compared to traditional CT, U-HRCT significantly enhances the visualization of minute structures in the inner ear, providing a novel technological approach for the diagnosis and treatment of ear diseases. The images provided by U-HRCT reveal entirely new signs that differ from those of traditional CT, requiring expert interpretation by experienced head and neck radiologists. This places a heavy workload on radiologists. Furthermore, the uneven experience among radiologists leads to missed diagnoses and misdiagnoses of ear diseases.

[0003] In recent years, artificial intelligence, represented by deep learning, has made significant breakthroughs in medical image analysis. Deep learning-based medical image analysis and processing has thus become a hot topic in the biomedical engineering field, with various new technologies, methods, and products emerging in an endless stream. Intelligent medical image analysis based on artificial intelligence is changing the traditional medical diagnostic model. However, deep learning relies on big data, and a large amount of labeled sample data is crucial for its success. Due to the special nature of medical images, the annotation of medical image data often requires experienced radiologists. Given the scarcity of medical resources, it is difficult to obtain a large amount of labeled medical image data. Although medical images generate a large amount of data each year, the amount of labeled data is relatively small. Specifically, in this task, based on the existing segmentation of the overall structure of the inner ear labyrinth, multi-level segmentation of its local structures is required. Conventional segmentation algorithms first need to annotate a sufficient number of multi-level labeled samples and then retrain the multi-level instance segmentation model. These algorithms do not utilize the relative positional relationships between the structures of the inner ear labyrinth; instead, they treat each structure to be located as an independent individual. The relative positional relationships between anatomical structures are a key characteristic of medical images and can provide guidance for multi-level segmentation. In addition, there is currently no way to directly apply the relative positional information between anatomical structures to deep learning and transform it into data that can rival human annotation.

[0004] Therefore, this invention proposes an algorithm based on extracting the 3D central ossicle line of the inner ear labyrinth, then using the prior spatial location structure of the inner ear labyrinth to generate pseudo-labels, and finally using transfer learning and active learning methods to achieve multi-level segmentation of the inner ear labyrinth. Summary of the Invention

[0005] The purpose of this invention is to alleviate the heavy reliance of existing medical image segmentation methods on manual annotation. For instance segmentation tasks involving multiple structures of the inner ear labyrinth, conventional instance segmentation methods require a large number of manually annotated samples from radiologists for each type of structure, without extracting relevant prior information from existing inner ear labyrinth segmentation models. Furthermore, in multi-level segmentation tasks of the inner ear labyrinth, the spatial relative positions between target structures can provide prior guidance for instance segmentation. For example, the planes containing the three semicircular canals are approximately perpendicular to each other, and the vestibule connects the three semicircular canals and the cochlear structure anteriorly and posteriorly, respectively. Utilizing this prior knowledge can significantly improve the instance segmentation performance of the inner ear labyrinth. Current traditional segmentation networks introduce attention mechanisms to better capture contextual information, but do not utilize information about the entire anatomical structure. Moreover, for situations where there is only a small amount of labeled data and the amount of unlabeled data is far greater, the labeled data cannot accurately represent the true distribution of the data. This difference in data distribution leads to a decrease in the generalization ability of the segmentation model, making it unable to meet the needs of clinical applications on large datasets.

[0006] To address the aforementioned issues, a method is proposed to generate multi-level target structure pseudo-labels on the central bony line based on prior information about the spatial location structure of the inner ear labyrinth. This generates pseudo-labels for the multi-level structures of the inner ear labyrinth comparable to those manually annotated by professional physicians. These pseudo-labels serve as the training and validation sets for an automatic instance segmentation network with multi-level annotation. Leveraging existing semantic segmentation networks based on a large number (68 cases) of manually annotated single structures of the inner ear labyrinth, an initial model for multi-level segmentation of the inner ear labyrinth with a small sample size (10 cases) is achieved, enabling automatic segmentation of the horizontal semicircular canals, superior semicircular canals, posterior semicircular canals, vestibule, and cochlea within the inner ear labyrinth. Then, based on an active learning paradigm, the most valuable samples from the initial model are selected to update and optimize the initial multi-level segmentation model.

[0007] This invention is achieved using the following technical means:

[0008] A method for generating and segmenting pseudo-labels for the inner ear labyrinth based on prior spatial location structure is proposed. The method consists of three parts: generating an initial model based on pseudo-labels, selecting the most valuable labeled samples based on the initial model, and updating and optimizing the multi-level segmentation model based on incremental learning. Figure 1 As shown.

[0009] The method specifically includes the following steps:

[0010] 1) Initialization model generation based on pseudo-labels:

[0011] The first step is to segment the inner ear labyrinth from the image data using an existing U-HRCT-based whole inner ear labyrinth segmentation network. The segmentation network used in this step is TransUNet, and the input data to the model is two-dimensional. The training dataset of this model consists of 64 complete U-HRCT image data and corresponding whole inner ear labyrinth annotation data.

[0012] The second step involves extracting the central skeletal line from the inner ear labyrinth segmentation mask. This skeletal line extraction, also known as binary image thinning, is an algorithm that refines a connected region to a width of one pixel for feature extraction and target topological representation. This method allows for further extraction of inner ear labyrinth structural features.

[0013] The third step involves using the DBSCAN clustering algorithm to extract the vestibular structures from the inner ear labyrinth skeleton. DBSCAN is a density-based spatial clustering algorithm. This algorithm divides regions with sufficient density into clusters and can discover clusters of arbitrary shapes in noisy spatial databases. A cluster is defined as the largest set of density-connected points. The algorithm flowchart is shown below. Figure 2As shown. It is worth noting that DBSCAN requires manually setting two values: radius (Eps) and density (MinPts). Because in the inner ear labyrinth structure, the vestibule connects anteriorly to the three semicircular canals and posteriorly to the cochlea, the inner ear labyrinth structure is as follows... Figure 3 As shown, after the vestibule separates, the cochlea can automatically separate. The semicircular canals include the horizontal, superior, and posterior semicircular canals, each shaped like a "C," approximately two-thirds the circumference of a circle. Each semicircular canal connects to the vestibule with two bony crura. One crura is called the ampulla, and each of the three canals has a swollen ampulla. The other crura of the superior and posterior semicircular canals are combined to form the common crura, while the other crura of the superior semicircular canal is called the single crura. Therefore, the three semicircular canals have a total of five openings into the vestibule. The three bony semicircular canals are approximately perpendicular to each other in spatial position. The horizontal semicircular canal arches upwards, perpendicular to the long axis of the pyramid, and is buried deep within the arch-shaped bulge; the superior semicircular canal arches posterolaterally, forming the lateral semicircular canal bulge at the entrance of the mastoid sinus; the posterior semicircular canal arches posterolaterally, approximately parallel to the long axis of the pyramid. Furthermore, according to relevant medical definitions, a continuous structure from the vestibule to the semicircular canals is considered a semicircular canal when it forms a tubular structure. Based on this medical definition and the structure of the inner ear labyrinth, the horizontal semicircular canals are directly connected to the vestibule at both ends, while the superior and posterior semicircular canals share a portion of their structure and are connected to the vestibule, with their other ends extending and connecting to the vestibule. In this case, the DBSCAN algorithm needs to be used again to select the smaller spatial density value for clustering, thereby separating the horizontal semicircular canals into two different clusters: the horizontal semicircular canals and the superior and posterior semicircular canals.

[0014] Fourthly, to standardize the process, the common area of ​​the superior and posterior semicircular canals is uniformly classified as the superior semicircular canal structure. Based on the prior knowledge that the three semicircular canal planes are approximately perpendicular, Principal Component Analysis (PCA), a statistical method for dimensionality reduction, uses an orthogonal transformation to convert the original random vectors whose components are correlated into new random vectors whose components are uncorrelated. This is a mathematical method for dimensionality reduction. Using PCA projection, the dimensionality of the three-dimensional superior and posterior semicircular canal skeletal point clouds is reduced and projected onto a two-dimensional plane, as shown below. Figure 4 As shown, they are separated according to the different intervals in which their projections fall on the two-dimensional plane.

[0015] Step five, following steps three and four, yields the point cloud clusters of ossicular lines representing various structures in the inner ear labyrinth. The process of converting ossicular line pseudo-labels into inner ear labyrinth pseudo-labels utilizes the KNN nearest neighbor classification algorithm. For any m-dimensional input vector (m=3 here), the nearest neighbor algorithm corresponds to the three-dimensional coordinates of a point in the feature space, outputting the category label corresponding to that feature vector. The nearest neighbor algorithm partitions the feature vector space using training data and uses the partitioning result as the final algorithm model. Here, the training data consists of the three-dimensional coordinates of each point on the ossicular lines, with labels corresponding to various structures in the inner ear labyrinth. Based on the trained final algorithm model, inputting the three-dimensional coordinates of each point in the inner ear labyrinth automatically outputs the corresponding structural labels, thus generating multi-level pseudo-labels for the inner ear labyrinth structures.

[0016] Step 6: Due to the insufficient robustness of the aforementioned machine learning clustering methods based on structural priors, they fail to achieve good clustering results for some inner ear labyrinth structures, and the process is cumbersome and time-consuming. Therefore, deep learning methods are still needed to train neural network models, enabling direct acquisition of multi-level inner ear labyrinth structure segmentation results from U-HRCT images. Considering existing work, a transfer learning method is chosen to transfer the TransUNet inner ear labyrinth single-class segmentation network to an inner ear labyrinth multi-level segmentation network. The two share most parameters, but because the segmentation task changes from single-class to multi-class, the output layer structure of the TransUNet segmentation network differs. Therefore, during training, all parameters of the TransUNet single-class segmentation network, except for the segmentation result output layer, are first loaded into the multi-class segmentation network as pre-training weights. Then, the final segmentation result output layer in the multi-class segmentation network is randomly initialized. During the TransUNet multi-class segmentation training phase, all network parameters are allowed to backpropagate and update, with an initial learning rate of 0.01. Regarding the dataset, 10 pseudo-labels were generated using the aforementioned machine learning methods, with 9 used as the training set and 1 as the validation set. The initial model for multi-level segmentation of the inner ear labyrinth was obtained through the above training method, and the initial model was further optimized.

[0017] 2) Select the most valuable labeled samples:

[0018] The first step involves shape modeling of the inner ear labyrinth multi-level segmentation results using a VAE variational autoencoder. This method uses an encoding network to learn features and perform statistical analysis on the inner ear labyrinth structure; the decoder reconstructs the inner ear labyrinth structure based on quantified descriptive features. Applying the variational autoencoder inner ear labyrinth morphological model to the no-reference evaluation of the segmentation model algorithm serves as a measure of the quality of the inner ear labyrinth segmentation results, thus providing a reference method for selecting the most valuable labeled samples for the initial multi-level segmentation model. This method can accurately identify segmentation results with significant incompleteness or poor segmentation quality. The VAE model is a deep learning-based generative model that can encode and reconstruct input shape data to represent high-dimensional complex shapes with low-dimensional implicit vectors. Unlike the AE autoencoder, the VAE model assumes that the latent space follows a known prior distribution (usually a Gaussian distribution). Specifically, for a set of input data X, the posterior distribution p(z|x) of the implicit vector Z(z1, z2, ..., zn) can be approximated by the variational distribution q(z|x). In variational inference, the evidence lower bound (ELBO) is maximized during training using data, as shown in Equation (1):

[0019]

[0020] Where p(x) is the evidence probability, p(z) is the prior, and p(x|z) is the likelihood probability. The first term on the right side of formula (1) is the regularization term for calculating the KL divergence, which constrains the posterior to approximate the prior; the second term is the likelihood of reconstructing the original input based on implicit vectors. VAE assumes that the prior p(z) and the approximate posterior q(z|x) follow a Gaussian distribution, i.e., p(z) ~ N(0, I). In addition, reparameterization is used to simulate the approximate posterior q(z|x). That is, by assuming z ~ N(μ, σ 2 The implicit vector z can be obtained by reparameterization using formula (2). The mean μ and standard deviation σ are calculated by gradient optimization during training of the VAE model.

[0021]

[0022] in Let ζ be a normal distribution with a mean of 0 and a variance of I, and let ζ follow this distribution.

[0023] Since identical anatomical structures in medical images should have similar shapes, the parameters of the implicit vectors in normal segmentation results, after being encoded by the VAE model, should fall within the parameter distribution range of the established model. Based on this idea, this step uses the established shape average statistical code as the gold standard, and achieves referenceless segmentation quality assessment by comparing the cosine similarity between the result to be evaluated and the gold standard, as shown in formula (3):

[0024]

[0025] Where A i and B i These represent the components of the encoding vector A for the result to be evaluated and the encoding vector B for the gold standard, respectively.

[0026] Lower cosine similarity values ​​correspond to significantly incomplete samples; therefore, this method can effectively screen out poorly segmented samples to a certain extent. The segmentation quality assessment process based on VAE is as follows: Figure 5 As shown.

[0027] The second step involves using the TransUNet multi-level segmentation network to predict 648 unlabeled U-HRCT data points. In the decision layer of the segmentation model, probability matrices for various foreground channels are extracted. A maximum connectivity mask is used to filter the probabilities of discrete point regions, resulting in the maximum connectivity probability map for each structural category, denoted as P. c , where c∈(0,4), and then the sample labeling value is quantified based on uncertainty assessment. For a specific foreground category c, for a maximum connectivity probability map containing m slices, each with n pixels, the uncertainty of the model prediction is defined by entropy, as shown in formula (4):

[0028]

[0029] Where, p ij This represents the probability that each pixel belongs to the foreground. When the probability is close to 0.5, it indicates that the segmentation model has difficulty determining the category of that pixel. Therefore, higher uncertainty means that there are more difficult-to-segment voxels in the sample. Labeling these samples can provide more valuable information for improving the model's segmentation performance.

[0030] Thirdly, although uncertainty can assess areas that are difficult for the model to predict in some cases, the probabilities generated at the decision level are not entirely reliable. In some cases, even if the segmentation model has a high confidence level for a certain pixel, it may still produce incorrect predictions with low uncertainty. For this reason, the strategy of selecting the most valuable labeled samples is combined with the mask shape modeling quality assessment strategy of the inner ear labyrinth segmentation results in step one and the uncertainty based on the segmentation model prediction in step two to jointly measure the labeling value of unlabeled samples. The sample value index Q is defined to evaluate the labeling value of unlabeled samples, as shown in formula (5):

[0031] Q=λ1×Norm(U)+λ2×Norm(1-sim) (5)

[0032] Here, λ1 and λ2 are balancing coefficients between uncertainty and cosine similarity, used to balance the contributions of uncertainty and cosine similarity to the sample value measurement. sim and U are the cosine similarity and uncertainty obtained in the first and second steps, respectively. In the proposed method, uncertainty and cosine similarity are both of the same order of magnitude. Given the current amount of data, we treat the contributions of uncertainty and cosine similarity equally; therefore, λ1 and λ2 are both set to 0.5 in this chapter.

[0033] 3) Multi-level segmentation model update and optimization based on incremental learning

[0034] The first step involves selecting the most valuable labeled samples, which are often the most difficult samples for the initial multi-level segmentation model to learn. These samples are relatively few in the data distribution and have a small amount of data. Therefore, data augmentation operations are needed for these difficult samples, including translation, rotation, horizontal flipping, and vertical flipping.

[0035] The second step, to address the problems caused by imbalanced samples during training, introduces Focalloss and Top-kloss loss functions in the multi-level segmentation model optimization stage. Focalloss is a loss function based on binary cross-entropy (CE), a dynamically scaled cross-entropy loss. Through a dynamic scaling factor, it dynamically reduces the weights of easily distinguishable samples during training, focusing the loss function on the difficult-to-classify samples from the perspective of sample difficulty, thus solving the model training problems caused by imbalanced samples. The formula for Focalloss is as follows:

[0036] FL(P t )=-(1-P t ) γ log(p t (6)

[0037] γ is a parameter ranging from [0,5]. When γ is 0, it becomes the cross-entropy loss function. P t P represents the degree of proximity to the ground truth and category y. t The larger the value, the closer it is to category y. (1-P) t ) γThis can reduce the loss contribution of easily separable samples, thereby increasing the loss proportion of difficult-to-separate samples. When the data distribution is uneven or a certain class of data has typical and atypical distributions, minimizing the average loss will ignore the data of the minority class distribution and obtain suboptimal results; while the maximum loss is very sensitive to sample noise and outliers. Even if there is only one outlier in the data, it may cause the model to learn a very poor decision boundary. Therefore, compared with the maximum loss, the k-th largest loss can better protect minority class samples and is more robust to noise than the maximum loss. Therefore, when optimizing the multi-level segmentation model, the average Top-k loss is used as a new aggregate loss. The average Top-k loss is defined as the average of the k largest losses on the sample set z, as shown in the following formula:

[0038]

[0039] Where L z (f) represents the ensemble loss, which is the sum of the individual losses over the training samples. [i] (f) represents the loss for the i-th sample.

[0040] The third step involves combining the most valuable labeled samples mentioned above with optimization training based on the loss function constraints from step two, to obtain the final multi-level segmentation model of the inner ear labyrinth.

[0041] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

[0042] This invention proposes a method to generate multi-level structures based on the relative spatial relationships of anatomical structures in medical images as prior information to guide the generation of multi-level labeled pseudo-labels for each structure in the inner ear labyrinth, which are then used to train a multi-level segmentation network for the inner ear labyrinth. By utilizing the relative positional relationships and morphological features of the semicircular canals, vestibule, and cochlea within the inner ear labyrinth, and combining various machine learning clustering methods, the structures of the inner ear labyrinth are classified at the central bony line feature level. Then, the nearest neighbor algorithm is used to generate multi-level segmentation mask pseudo-labels for the inner ear labyrinth corresponding to the bony lines. Building upon existing work on single-segmentation networks for the inner ear labyrinth in U-HRCT, a transfer learning method is used to train the initial multi-level segmentation network for the inner ear labyrinth. To improve the accuracy and robustness of multi-level segmentation, an initial segmentation network was optimized based on active learning. Regarding sample selection, two quantitative indicators were considered: the uncertainty of model prediction results and the similarity of segmentation structure shape modeling. This allowed for the selection of the most valuable samples for the initial network in the inner ear labyrinth multi-level segmentation. Then, during model training optimization, focal loss and Top-k loss were introduced to make the model focus more on difficult samples, mitigating the slow updates caused by sample imbalance during training. The generation of pseudo-labels, transfer learning, and the application of active learning paradigms further reduced the manual annotation cost, shortened training time, and improved the model's segmentation robustness in the inner ear labyrinth multi-level segmentation task.

[0043] Features of this invention:

[0044] 1. A method for generating multi-level pseudo-labels for various structures in the inner ear labyrinth based on prior information about the relative spatial relationships of anatomical structures is proposed. This method is used to segment the horizontal semicircular canals, superior semicircular canals, posterior semicircular canals, vestibule, and cochlea within the inner ear labyrinth. Conventional label generation methods require manual annotation by radiologists with specialized knowledge, a time-consuming and labor-intensive process. Traditional unsupervised methods cannot effectively utilize the spatial structural location information of the inner ear labyrinth, resulting in poor segmentation of each structure. In contrast, this invention fully utilizes prior information related to the inner ear labyrinth structure and employs various machine learning methods to automatically generate multi-level pseudo-labels for each structure with quality comparable to manual annotation by doctors.

[0045] 2. Building upon existing single-target segmentation networks for the inner ear labyrinth, we employ transfer learning to train a multi-level segmentation network for the inner ear labyrinth. During the training phase, we fully utilize the pre-trained weight information already learned by the segmentation network, thereby significantly shortening the training time and improving the model's segmentation accuracy. Attached Figure Description

[0046] Figure 1 Flowchart of the multi-level segmentation model of the inner ear labyrinth

[0047] Figure 2 DBSCAN Algorithm Flowchart

[0048] Figure 3 Schematic diagram of the structures of the inner ear labyrinth

[0049] Figure 4 Point cloud images of the superior and posterior semicircular canals before and after dimensionality reduction by projection.

[0050] Figure 5 Flowchart of VAE-based segmentation quality assessment Detailed Implementation

[0051] The following description, in conjunction with the accompanying drawings, illustrates various embodiments of the present invention:

[0052] This invention uses the temporal bone U-HRCT dataset for training and prediction. A total of 10 cases were used, with multi-level pseudo-labels for the inner ear labyrinth generated based on spatial location structural priors, resulting in five anatomical structures: the horizontal semicircular canal, superior semicircular canal, posterior semicircular canal, vestibule, and cochlea. Initially, 9 cases were used as the training set and 1 case as the validation set. The initial model used unlabeled data, totaling 652 cases.

[0053] The data preprocessing used in this invention includes desensitizing patient information in U-HRCT data and normalizing the Hu values ​​of images.

[0054] The process of the method for generating and segmenting pseudo-labels for multi-level annotation of the inner ear labyrinth based on spatial location structure priors is as follows:

[0055] a) Initialization model generation based on pseudo-labels

[0056] Step 1: Use the existing U-HRCT-based inner ear labyrinth global segmentation network to segment the inner ear labyrinth from the image data. The segmentation network used in this step is TransUNet. The input data to the model is two-dimensional. After the model has segmented a complete set of data, the two-dimensional prediction structure of the model is integrated into a three-dimensional inner ear labyrinth mask.

[0057] The second step involves extracting the central skeletal line from the inner ear labyrinth segmentation mask, and then generating pseudo-labels using algorithms such as DBSCAN and PCA. Specifically, the DBSCAN algorithm used Eps and MinPts values ​​of 3, 9, 6, 3, 2, and 3 respectively in three separate steps.

[0058] Third, considering the existing work, we choose to use the transfer learning method to transfer the TransUNet inner ear labyrinth single segmentation network to the inner ear labyrinth multi-level segmentation network. The two share most of the parameters. Therefore, except for the Conv2d and Identity layers in the final SegmentationHead module of TransUNet, the parameters in other layers are selected to be imported from the pre-trained model parameters of the inner ear labyrinth single segmentation network.

[0059] b) Select the most valuable labeled samples:

[0060] Step 1: In this step, the complete inner ear labyrinth region has a spatial resolution of 256×256×112. Due to computational resource limitations, this region is difficult to input into a 3D VAE model. Therefore, a 2D VAE model is used in the experiment to perform latent space encoding on 2D slices. The latent space dimension is initially set to 32 dimensions for training the VAE model until convergence. Then, PCA is used to reduce the latent space dimension, reducing the latent space dimension of a single image to 24 dimensions while retaining 95% of the information.

[0061] The second step is to sort the sample value index Q from high to low and select the 20% with the highest value from the unlabeled sample pool as high-value samples.

[0062] c) Multi-level segmentation model update and optimization based on incremental learning:

[0063] Step 1: Focalloss is a loss function based on binary cross-entropy (CE). It is a dynamically scaled cross-entropy loss. Through a dynamic scaling factor, the weights of easily distinguishable samples can be dynamically reduced during training.

[0064] Step 2: The average Top-k loss is defined as the average of the top k largest losses on the sample set z.

[0065] The quality of the pseudo-labels generated by the method was objectively evaluated against the gold standard manually labeled by experienced physicians. The evaluation results are shown in Table 1. Dice-Coefficient (DSC[%)), Average Symmetric Surface Distance (ASD[mm]), and Hausdorff Distance (HD[mm]) were used to evaluate the method. DSC is used to assess the degree of difference between the segmentation result P and the gold standard Q. The calculation method is shown in (8):

[0066]

[0067] Here, |·| represents the number of labeled voxels. The larger the DSC value, the more overlapping voxels there are between the predicted result and the gold standard.

[0068] ASD is a popular target surface-based metric used to measure the evaluation surface distance between symmetrical positions of two 3D objects. Its definition can be expressed as follows:

[0069]

[0070] Where S(P) and S(G) represent the surface voxel sets of P and G, respectively. For ASD values, the smaller the better.

[0071] Hausdorff distance (HD) is also a commonly used segmentation evaluation metric, primarily used to measure the accuracy of boundary segmentation. HD is a measure of the similarity between two sets of points; it is a definition of the distance between two sets of points: Assume there are two sets A = {a1, ..., a2}. p}, B={b1,…b p}, then the HD between these two sets of points is defined as:

[0072] H(A,B)=max(h(A,B),h(B,A))#(10)

[0073]

[0074]

[0075] (10) is called bidirectional Hausdorff distance, which is the most basic form of HD. In (11) and (12), h(A,B) and h(B,A) are called unidirectional Hausdorff distances from set A to set B and from set B to set A, respectively. Because it is sensitive to outliers, and outliers in medical segmentation are uncertain, in actual calculations, the final value is multiplied by 95% to eliminate the influence of a very small subset of outliers in order to maintain the stability of the overall value. This is also called HD. 95 .

[0076] Table 1 Evaluation Results

[0077]

Claims

1. An inner ear labyrinth multi-level labeling pseudo-label generation and segmentation method based on spatial position structure prior, characterized in that: First, the existing U-HRCT-based inner ear labyrinth whole segmentation network is used to segment the inner ear labyrinth from the image data, and the segmentation network adopted is TransUNet, and the data input into the model is two-dimensional; Second, the center bone line extraction is carried out from the inner ear labyrinth segmentation mask, and a connected region is refined to a pixel width for feature extraction and target topological representation; Third, the DBSCAN clustering algorithm is used to extract the vestibular structure in the inner ear labyrinth skeleton; Fourth, the PCA principal component analysis projection method is used to reduce the dimension of the three-dimensional superior and posterior semicircular canal bone line point cloud and project it onto a two-dimensional plane, and according to its different intervals in the two-dimensional plane projection, it is separated; Fifth, through steps three and four, the inner ear labyrinth structure bone line point cloud cluster is obtained, and the KNN nearest neighbor classification algorithm is used in the process of converting the bone line pseudo-label into the inner ear labyrinth pseudo-label; the nearest neighbor algorithm outputs the category label corresponding to the feature vector for any m-dimensional input vector m=3, which corresponds to the three-dimensional coordinates of a point in the feature space; the nearest neighbor algorithm divides the feature vector space using training data, and uses the division result as the final algorithm model, where the training data is the three-dimensional coordinates of the bone line points, and the label corresponds to the structures of the inner ear labyrinth; according to the trained final algorithm model, the three-dimensional coordinates of each point of the inner ear labyrinth are input, and the model automatically outputs the corresponding structure label of each point, thereby generating the inner ear labyrinth multi-level structure pseudo-label; Sixth, the transfer learning method is selected to transfer the TransUNet inner ear labyrinth single segmentation network to the inner ear labyrinth multi-level segmentation network, first load all parameters of the TransUNet single segmentation network except the segmentation result output layer to the multi-class segmentation network as pre-training weights, then randomly initialize the last segmentation result output layer of the multi-class segmentation network; during the training of the TransUNet multi-class segmentation, all network parameters are allowed to be updated through back propagation, and the initial learning rate is 0.01; in terms of data set, a total of 10 pseudo-labels are generated by using the above machine learning method, of which 9 are used as training set and 1 is used as validation set; the inner ear labyrinth multi-level segmentation initial model is obtained through the above training method, and the initialization model is further optimized; 1) Select the most valuable labeled samples: First, the VAE variational autoencoder shape model of the inner ear labyrinth multi-level segmentation result is established, the feature learning and statistical analysis of the inner ear labyrinth structure are carried out through the encoding network; the decoder is used to reconstruct the inner ear labyrinth structure according to the quantitative description features; the variational autoencoder inner ear labyrinth shape model is applied to the reference-free evaluation of the segmentation model algorithm, which is used to measure the goodness of the inner ear labyrinth segmentation result to a certain extent; For a set of input data X, the posterior distribution p(z| x) of the latent vector Z(z1, z2, …, zn) is approximated by the variational distribution q(z| x); in variational inference, the data is used to maximize the lower bound of evidence in training, as shown in equation (1): where p(x) is the evidence probability, p(z) is the prior, and p(x|z) is the likelihood probability; the first term on the right side of equation (1) is a regularization term for calculating the KL divergence, which constrains the posterior to be similar to the prior; the second term is the likelihood of reconstructing the original input based on the implicit vector; the VAE assumes that the prior p(z) and the approximate posterior q(z|x) follow a Gaussian distribution, i.e., p(z) ~ N(0, I); in addition, reparameterization is used to simulate the approximate posterior q(z|x); that is, by assuming z ~ N(μ, σ 2 ), the implicit vector z can be reparameterized by equation (2); where the mean μ and the standard deviation σ are calculated by gradient optimization during training of the VAE model; wherein is a normal distribution with mean 0 and variance I, and ζ is distributed according to this distribution; The shape average statistical encoding established as the gold standard, by comparing the evaluation results with the gold standard encoding cosine similarity to achieve no reference segmentation quality evaluation, see equation (3): where A i and B i represent the respective components of the result encoding vector A and the gold standard encoding vector B, respectively; Secondly, the TransUNet multi-level segmentation network is used to predict the unlabeled U-HRCT data, and the probability matrix of each foreground channel is extracted in the decision layer of the segmentation model. The maximum connected mask is used to filter the probability of discrete point area, and the maximum connected probability map of each structure category is obtained, denoted as P c where c∈(0,4), and then the sample labeling value is quantified based on the uncertainty evaluation. For a specific foreground class c, for the maximum connected probability map of each n pixel containing m slices, the uncertainty of the model prediction is defined by entropy, see equation (4): where p ij represents the probability that each pixel belongs to the foreground; Step 3, the strategy of screening the most valuable labeled samples combines the shape modeling quality evaluation strategy of the inner ear labyrinth segmentation result mask in step 1 and the uncertainty based on the segmentation model prediction in step 2 to jointly measure the labeling value of unlabeled samples, defines the sample value index Q to evaluate the labeling value of unlabeled samples, see equation (5): Q = λ1 x Norm(U) + λ2 x Norm(1-sim) (5) Where λ1 and λ2 are the balance coefficients between uncertainty and cosine similarity, used to balance the contribution of uncertainty and cosine similarity to sample value measurement, sim and U are the cosine similarity and uncertainty obtained in the first step and the second step respectively; λ1 and λ2 are both 0.5; 3) Multi-level segmentation model update optimization based on incremental learning Step 1, the most valuable labeled samples screened in the above steps are often difficult samples for the initial multi-level segmentation model, so data enhancement operations need to be performed on these difficult samples first; Step 2, Focalloss and Top-kloss loss functions are introduced in the multi-level segmentation model optimization stage; the formula of Focalloss is as follows: FL(P t ) = - (1 - P t ) γ log(p t ) (6) gamma is a parameter, range in [0, 5], when gamma is 0, it becomes cross-entropy loss function; P t represents the closeness to groundtruth and class y, P t the larger, the closer to class y; In optimizing the multi-level segmentation model, average Top-kloss is used as a new aggregate loss; the average Top-kloss loss is defined as the average of the top k largest losses on the sample set z, as shown in the following formula: where L z (f) is the collective loss is the accumulation of all individual losses on the training samples, l [i] (f) is the loss for the i-th example.

2. The method of claim 1, wherein: Temporal bone U-HRCT data set is used for training and prediction; a total of 10 cases of inner ear labyrinth multi-level labeled pseudo labels generated based on spatial position structure prior, a total of 5 anatomical structures of horizontal semicircular canal, superior semicircular canal, posterior semicircular canal, vestibule and cochlea are generated; in the initial training process, 9 cases of data are used as the training set and 1 case of data is used as the validation set; the initial model uses unlabeled data; The data preprocessing adopted includes patient information desensitization and image Hu value normalization of U-HRCT data; The process of generating and segmenting inner ear labyrinth multi-level labeled pseudo labels based on spatial position structure prior is as follows: a) Initialization model generation based on pseudo labels Step 1: Use the existing U-HRCT inner ear labyrinth overall segmentation network to segment the inner ear labyrinth from the image data; the segmentation network used is TransUNet, the input model data is two-dimensional, and after the model segments a complete case of data, the two-dimensional prediction structure of the model is integrated into a three-dimensional inner ear labyrinth mask; Secondly, the center bone line is extracted from the inner ear labyrinth segmentation mask, and then pseudo labels are generated using DBSCAN and PCA algorithms; wherein the Eps and MinPts values of the DBSCAN algorithm are Eps=3, MinPts=9, Eps=6, MinPts=3 and Eps=2, MinPts=3 respectively; Thirdly, the transfer learning method is selected to migrate the TransUNet inner ear labyrinth single segmentation network to the inner ear labyrinth multi-level segmentation network, and the parameters in other layers are selected to import the pre-training model parameters of the inner ear labyrinth single segmentation network except for the Conv2d and Identity layers in the last SegmentationHead module of TransUNet; b) Screening the most valuable labeled samples: First step: In this step, the complete inner ear labyrinth region spatial resolution is 256×256×112, and the 2D VAE model is used to encode the hidden space of the 2D slice; the hidden space dimension is initially trained with a 32-dimensional VAE model to convergence; then, the hidden space is reduced in dimension by PCA, and the hidden space dimension of a single image is reduced to 24 dimensions under the condition that 95% of the information is retained by PCA dimension reduction; Second step: After sorting the sample value index Q from high to low, the top 20% of the highest value from the unlabeled sample pool is selected as the high value sample; c) Multi-level segmentation model update optimization based on incremental learning: First step: Focalloss is a loss function based on binary cross-entropy CE, which dynamically reduces the weight of easily distinguishable samples in the training process through a dynamic scaling factor; Second step: The average Top-kloss loss is defined as the average of the top k largest losses on the sample set z.

Images