Inner ear lost form field representation and classification method based on form difference example guidance

Abstract

The invention discloses an inner ear lost form field representation and classification method based on form difference example guidance, and belongs to the field of medical image processing. According to the method, inner ear lost is represented and analyzed, and the established deformation field model is applied to an inner ear form classification task. The method is mainly characterized in that difference information is compared, a model learns morphological difference between normal data and variation data, and specifically, the data is represented and analyzed through a deformation field; secondly, a variable hyper-parameter embedded network is designed, a unique training strategy is designed for the variable hyper-parameter embedded network, so that the model can learn a potential relationship between a deformation field and hyper-parameters, and finally, the established inner ear lost deformation field representation method is applied to an inner ear lost form classification task. Experimental results show that the method can effectively express the detail difference of the inner ear lost local structure, and has a good distinguishing capability for data of different forms and types.

Description

Technical Field

[0001] The invention belongs to the field of medical image processing, and in particular relates to a method for representing and classifying the morphology of the inner ear labyrinth by combining a registration deformation field as a representation means. Background Art

[0002] Congenital inner ear malformation, also known as congenital labyrinth malformation, is caused by gene mutation, deletion or other genetic factors in the early embryonic development (3rd to 23rd week of embryonic development), as well as non-genetic factors such as viruses, physical and chemical factors during the mother's pregnancy, which lead to the cessation or variation of inner ear development. It is an important cause of congenital deafness. Congenital inner ear malformation can occur alone or be accompanied by external ear and middle ear malformations. Some cases are accompanied by facial organs, eyes, mouth, teeth malformations and (or) limb and visceral malformations. Clinical statistics show that 20% of patients with congenital deafness have inner ear morphological variations. The basis of inner ear morphological classification is the anatomical differences of various inner ears. Similar inner ear morphologies are classified into one category. They reflect similar clinical manifestations and treatment methods. Therefore, unified classification of inner ear morphology is particularly important.

[0003] The ear region is mainly composed of bone cavities, air cavities and other structures. CT images are the preferred way to intuitively display their structures and variant diseases. They are of great value in cochlear implantation, ear inflammation changes, cholesteatoma and postoperative evaluation. Since CT images are volume acquisitions, the target structure development, quantitative measurement and statistical analysis based on CT images have more accurate advantages than MR images. Traditional ear CT analysis and quantitative measurement are mainly manual, which is cumbersome to operate and has large data differences. It is not suitable for statistical analysis of batches of large cases. CT images are mainly browsed in continuous two-dimensional tomograms. In order to perform three-dimensional data analysis, the two-dimensional data obtained by computer tomography can be converted into 3D models through three-dimensional representation, anatomical experiments and other methods to present the bone and soft tissue structures of the outer ear, middle ear and inner ear, so as to increase the understanding of the anatomical structure of the human inner ear and middle ear, and at the same time focus on observing specific structures and quantitatively measuring the distance and angle of the target anatomical structure.

[0004] However, due to the wide variety of inner ear morphologies, the different definitions and standard scales between different types, and the limited amount of data for each type, it is of great significance to efficiently use a small number of example samples to obtain a more accurate and robust classification algorithm.

[0005] This paper takes the inner ear labyrinth structure morphology as the research object by targeting the latest temporal bone U-HRCT images developed by Beijing Friendship Hospital affiliated to Capital Medical University, and designs a method for representing and classifying the inner ear labyrinth morphology based on the registration deformation field. Using the self-supervised contrastive learning method, an efficient and accurate classification method is proposed to improve the classification accuracy of different types of inner ears. Finally, a multi-category inner ear morphology classification algorithm is realized. Summary of the invention

[0006] The purpose of the present invention is to construct an intelligent, fast, accurate and robust U-HRCT inner ear morphology classification algorithm. For the U-HRCT inner ear, due to the wide variety of inner ear morphologies, the diverse definitions and standard scales between different types, and the limited amount of data for each type, conventional manual detection methods have strong subjective differences. Therefore, it is necessary to assist in interpretation with the help of methods based on objective judgment criteria. The existing shape representation methods based on deep learning are mainly end-to-end statistical representation methods based on autoencoders or variational autoencoders (applying implicit priors). Compared with the autoencoder, the variational autoencoder sets the input data to obey a certain prior distribution, and the encoder extracts features from the input data to obtain the mean and standard deviation of the distribution, and then obtains its implicit expression vector by random sampling. Compared with the autoencoder, the variational autoencoder can generate results that obey its data distribution but are different from any input data. The features learned by the statistical structural morphological representation are sensitive to dimensions with large statistical variation variance in the training data, and the inner ear structure may also show variations in local areas in addition to overall morphological changes. The statistical representation of the overall morphology is difficult to effectively quantify the local morphological changes of medical concern through statistical energy. Therefore, how to establish a multi-scale morphological quantification model that is sensitive to clinical focus points and effectively parameterize the local morphology of the inner ear is a key issue that needs to be solved.

[0007] To address the above problems, we can design a method for representing the inner ear labyrinth morphology guided by difference signs to simulate the diagnostic learning ability of learning morphological difference examples, so that the model can learn the knowledge of morphological classification with only a small number of target sample examples. In addition, by changing the information of morphological representation, the generated data can be modified as expected, so that the model can autonomously simulate and generate the change process from the original data to the target morphological data, thereby generating a large number of pseudo-target morphological samples.

[0008] In the specific implementation of morphological representation, a stereo deformation field network structure based on variable parameter embedding is proposed. The network architecture adopts a deformable registration network. The input image includes a normal image sequence and a test sequence. The network output is the deformation field of the morphological change from the target sequence to the normal sequence. By adjusting the deformation field amplitude parameter, the morphological diffusion sample from the normal to the target sequence can be generated. The model can simulate the change process from normal to target morphology, thereby generating sequence pseudo samples. With the help of a large amount of existing (517 cases) U-HRCT temporal bone image data, the stereo deformation field network model based on variable parameter embedding is trained to complete the morphological expression of the inner ear labyrinth. Then, different morphological samples are classified by performing dimensionality reduction analysis on the deformation field representation information.

[0009] The present invention is achieved by adopting the following technical means:

[0010] A deformation field-based inner ear representation and classification method. The method is divided into three parts: segmentation and localization alignment of inner ear labyrinth substructures, parametric expression based on variable morphology diffusion network, and classification of target structure morphology, such as Figure 1 shown.

[0011] The method specifically comprises the following steps:

[0012] 1) Segmentation and alignment of inner ear labyrinth substructures:

[0013] In the first step, the existing U-HRCT-based inner ear labyrinth substructure segmentation network is used to segment the inner ear labyrinth substructure to be calibrated from the image data. The segmentation network used in this step is TransUNet, and the input data of the model is two-dimensional slice data along the Z-axis direction. The training data set of this model is 10 complete U-HRCT image data and the corresponding inner ear labyrinth substructure annotation data.

[0014] Second, during the CT scan, the image is affected by the deviation and rotation of the human head, and the imaging differences between different data are large. Therefore, the angle and spatial position of the anatomical structure of the inner ear labyrinth vary greatly, such as Figure 2 In the process of inner ear labyrinth morphology classification, in order to avoid receiving images of factors other than shape, such as angles and spatial positions, all data need to be aligned and calibrated to the same standard.

[0015] ROI positioning and alignment are performed from the segmentation mask of the inner ear labyrinth substructure to be calibrated, and the ROI region of interest is positioned on the original image according to the center point of the segmentation mask. In this paper, a data alignment method based on affine transformation is proposed for the alignment of the inner ear labyrinth structure. The coordinates of the center points of the vestibule, horizontal semicircular canal, superior semicircular canal and posterior semicircular canal are extracted using the substructure segmentation mask. The formula of 3D affine transformation is as follows:

[0016]

[0017] Among them, x ′ ,y ′ and z ′ T represents the coordinates after affine transformation, and x, y and z represent the coordinates before affine transformation. X ,T Y and T Z Represents the translation in the X, Y and Z directions respectively, is the 3D rotation matrix. When selecting the coordinates of four non-coplanar points, the right is a fourth-order square matrix, and there is a reversible matrix. Therefore, the affine transformation matrix can be obtained according to the coordinates of the corresponding points before and after the affine transformation, that is,

[0018]

[0019] In addition, due to the small field of view of U-HRCT imaging, each sequence changes from bilateral ear imaging of ordinary CT to unilateral imaging, and each data needs to be classified into left and right ears. The left and right ear data can be approximately understood as mirror data, and the horizontal mirror affine transformation matrix is ​​as follows:

[0020]

[0021] Where, width is the image size in the x direction. X The translation voxel value will not exceed 50 pixels, so T can be determined by calculating the affine matrix obtained by the calibrated data and the left and right template data. X ,T Y and T Z The value determines the left and right sides of the data to be calibrated. The calibration result is as follows: Figure 3 shown.

[0022] 2) Parameterized representation based on variable morphology diffusion network:

[0023] In the first step, the network architecture used in this paper is a convolutional neural network architecture similar to Unet, which consists of an encoder and a decoder with skip connections. The input 3D image f (fixed image) and m (image to be transformed) are spliced ​​into a single input data with 2 channels. The hyperparameter λ is multiplied with the feature map before each convolution layer of the input encoder as attention after dimension expansion and full connection layer transformation. In the encoder structure, there is a LeakyReLU activation function after each convolution layer. The spatial dimension of each layer is halved by using strided convolution to effectively reduce the spatial resolution of the output feature map to achieve downsampling effect, while adjusting the spatial receptive field of the feature map to better capture the local and global features of the input image. The output of the encoder is a high-dimensional feature tensor containing an abstract representation of the input image. The task of the decoder is to map the features extracted by the encoder to a deformation field, which describes how to transform from the input image to the target image. The decoder usually includes multiple convolutional layers and upsampling layers (deconvolutional layers) to gradually restore the resolution of the deformation field. At each step of the decoder, the upsampled feature map is concatenated (skip connection) with the feature map of the corresponding layer of the encoder. This structure helps to retain low-level details and avoid loss of spatial information, thereby improving accuracy. After upsampling, the decoder continues to extract features through convolutional layers to further improve the resolution. These convolutional layers are 3x3 convolution kernels to further refine the feature map and extract local features. After the convolutional layers of the decoder, the ReLU activation function is used to increase nonlinearity and help the network learn more complex feature representations, so that the features learned in the encoding stage can be directly passed to the decoding stage to generate the registration field. The last layer of the decoder is the output layer, which is used to generate the prediction of the deformation field. This deformation field describes how to transform from the input image to the target image, usually in a four-dimensional form to represent the displacement of each spatial voxel in the three directions of X, Y, and Z in the spatial coordinate system. At the same time, there is a jump connection between the encoder and the decoder to fuse the low-level and high-level features extracted by the encoder to better guide the generation of the deformation field. This helps the network capture the local and global information of the input image. The specific encoding and decoding network framework is as follows Figure 4 shown.

[0024] In the second step, the role of the sampler in the spatial transformer network is to apply the transformation to the input image and align it with the target image, thereby achieving medical image registration generation. This is very useful for tasks such as aligning medical images of different patients to the same template space or performing time series image registration.

[0025] The method proposed in the present invention minimizes The difference between and f is used to learn the optimal parameter value. Specifically, m is the 3D image to be transformed, Its transformation function is represents the transformed image, the transformation operation is performed by the spatial transformer network (STN), and f is a fixed 3D image. In order to use the standard gradient-based transformation method, the present invention uses a differentiable operation based on the spatial transformer network to calculate For each voxel p of the 3D image m before transformation, calculate the transformed voxel position p in m ′ =p+u(p), where p is the voxel position before transformation and u(p) is the transformation function Transform offset for position p. Since the pixel coordinates of an image are defined only as integers, linear interpolation is required between the eight neighboring voxels of each voxel:

[0026]

[0027] Among them, z(p ′ ) is p ′ The set of adjacent voxels, voxel q is the voxel point in the set, m(q) is the voxel value of the input feature map m at voxel q, and d is the dimension of the iterative spatial domain Ω.

[0028] The core idea of ​​STN is to allow neural networks to learn how to transform input data spatially (such as rotate, scale, or distort) to enhance the model's robustness to geometric transformations. A major advantage of STN is that it is differentiable, which means it can be trained via regular back-propagation. Therefore, when STN is embedded in a larger neural network, the entire network (including STN) can be trained end-to-end at the same time. In short, the spatial transformer network provides neural networks with explicit spatial transformation capabilities, allowing the model to better handle spatial distortions and transformations of images.

[0029] The third step is to design the loss function in the model training phase. In this method, the loss function we use can be divided into two parts: the similarity loss function between the constrained transformed image and the fixed image. And the smoothness loss function that constrains the spatial variation of the deformation field in The calculation formula is as follows:

[0030]

[0031] We define the similarity loss function as Two commonly used loss functions are adapted. The first is the mean square error (MSE) loss function, which is used to measure the difference between the predicted value and the true value. MSE is calculated by square the difference between the predicted value and the true value and taking the average. Specifically, for a 3D spatial transformation task, assuming that the predicted value voxel value is The true voxel value is h, and the number of voxels is n. The calculation formula of MSE is as follows:

[0032]

[0033] One of the advantages of MSE is that it amplifies the square of the prediction error, so that larger errors get more weight in the loss function. This helps capture the larger difference between the predicted value and the true value and optimize it more sensitively.

[0034] The second one is and the local cross-correlation loss function of f, The definition of f is the same as in the second step. Compared with some other similarity measures, cross-correlation is more robust to certain types of noise and distortion, which is particularly important in medical imaging because medical images are often affected by various artifacts, noise and equipment differences. and Respectively represent the image f around position p 3 The local average intensity value within the voxel volume and the transformed image Around position p n 3 The local average intensity value within the voxel volume. The specific formula is where p j Iterate over n around o 3 The volume of f(p j ) is the voxel p j Voxel value at image f. In the present invention, n=9. The local cross-correlation function of and f is as follows:

[0035]

[0036] A higher CC indicates a higher degree of similarity, so the local cross-correlation loss function is:

[0037] minimize This will encourage the model to Approximate to f, making the generated image closer to the target image. At the same time, the model may produce more exaggerated distortion than reality. In essence, it is generated by the network model. This is caused by excessive displacement gradients in some areas. Therefore, diffusion regularization is used on the spatial gradient of displacement u to constrain the network model to generate a smooth displacement field. :

[0038]

[0039] Here The difference between voxels connected to voxel p is used to approximate the spatial gradient. approximate and Here x, y, and z represent the three directions of voxel p in the spatial coordinate system.

[0040] The fourth step is to train the diffusion network based on variable hyperparameter embedding. In order to enable the model to generate different degrees of deformation field information according to the input hyperparameter λ, a training method with dynamic variable hyperparameters is adopted in the training stage. Specifically, in each epoch, a real number λ between 0 and 1 is randomly generated. random (a float value in this invention) is used as the hyperparameter of this epoch. After that, the model first obtains the hyperparameter setting of this epoch through the hyperparameter embedding method in the first step above, and then outputs information such as the deformation field. When calculating the loss function, the following loss function is used:

[0041]

[0042] To constrain the model. In each epoch, different λ random value to penalize the change in the model output deformation field. random When λ is larger (close to 1), it means that the loss function has a greater constraint on the deformation field, forcing the model to generate a deformation field with a more gradual change. random When it is small (close to 0), the penalty constraint on the deformation field is small, and the model can generate a more drastic deformation field, making Smaller but also ensures The value of the hyperparameter λ can be predicted by the model through encoder embedding in the inference phase, which encourages the model to adapt to different λ values. random Value, come in and , and then generate a morphological diffusion network with controllable deformation. It should be noted that compared with no random generation of λ random Alternatively, in each epoch stage, a training strategy in which λ is fixed, the diffusion network with variable hyperparameter embedding proposed in the present invention requires more epochs (3000-5000 rounds) to train the model in the training stage so that the model can learn the intrinsic relationship between λ and the deformation field.

[0043] 3) Morphological classification of target structure:

[0044] In the first step, the deformation field changes between normal images after different alignments can be divided into two parts. One part is the local misaligned offset changes in the segmentation and positioning alignment of the inner ear labyrinth substructures mentioned above, and the other part can be summarized as the individual intra-class differences between the inner ear labyrinth structures of different people, which are manifested in the form of fluctuations in the diameter and curvature of the semicircular canals within the normal range. The differences between the above two parts can be collectively referred to as intra-class differences between normal data, and the deformation field transformation can be understood as the superposition of their intra-class differences. In addition to the two parts mentioned above, the deformation field changes from morphological difference data to normal data also have class-to-class differences in the local target structure compared to the normal structure. Therefore, the deformation field model of the transformation from morphological variation data to normal data is different in the expression of the local target area compared to the normal structure. For example Figure 5 and 6 The figure shows the arrow vector field visualization of the deformation field of the normal image and the variant image and their corresponding layers in the X and Z directions in the spatial coordinate system, as well as the heat map of the change amplitude in the X, Y and Z directions. The longer the arrow of the vector field, the greater the change amplitude. Similarly, the darker the color of the heat map, the more drastic the change. Among them, the heat map in the X direction mainly reflects the change of the deformation field between the layers of the CT image. The heat maps in the Y and Z directions can be combined into arrow vector maps in the Y and Z directions, which mainly reflect the change of the image in the direction within the layer. As can be seen from the purple box in the figure above, the deformation field representation of the normal image is significantly different from that of the variant image in the local area of ​​the target, which further shows that the inner ear labyrinth morphological representation model proposed in this paper can effectively express the differences in the local area of ​​the target.

[0045] In the second step, in the previous work on the segmentation of the inner ear labyrinth substructure, an unsupervised method based on substructure voxel statistics was introduced to perform morphological classification tasks on the inner ear labyrinth, and different standard deviation multiples were set to effectively classify the target structure. However, since the voxel value contains less information, the local structure of some cases with small morphological differences cannot be effectively detected. After analyzing the cases that failed to be classified in the previous classification work, it was found that the failed cases were mainly reflected in the morphological difference data of the fusion of the semicircular canals and the vestibule. This method is used to represent these data with deformation fields, and based on the substructure segmentation results obtained in the previous work, the semicircular canals are located in ROI, and the three-dimensional deformation field information of the target structure is extracted. The ROI size is [3,30,80,80]. The three-dimensional deformation field information of the variant target structure and the normal target structure is reduced by t-SNE, and its two-dimensional visualization is shown as follows. Figure 7 In addition, the information is also subjected to PCA (principal component analysis) dimensionality reduction, and the visualization results are shown as follows Figure 8As shown in the figure. The experimental results show that the deformation field representation method proposed in this paper can effectively reflect the difference information of the target structure information and cluster them together after dimensionality reduction. At the same time, compared with the t-SNE method, PCA dimensionality reduction can not only reduce the dimensionality of similar morphological data and cluster them, but also has a more obvious boundary distinction between it and normal data. Combining the unsupervised normalized standard deviation statistical variance and the deformation field representation analysis proposed in this paper, the structural morphology is classified.

[0046] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

[0047] The present invention proposes a three-dimensional deformation field network to represent and analyze the inner ear labyrinth, a key structure in the temporal bone region in U-HRCT, and applies the established deformation field model to the inner ear morphology classification task. The proposed method mainly compares the difference information, allowing the model to learn the morphological differences between normal data and variant data, specifically through the deformation field to represent and analyze the data; secondly, a variable hyperparameter embedding network is designed, and a unique training strategy is designed for it, so that the model can learn the potential relationship between the deformation field and the hyperparameters. Experimental results show that this training strategy enables the model to weigh the value of each λ. and The optimal solution under the current conditions is obtained, which further proves that the training strategy can effectively control the deformation field changes of different degrees. Finally, the established inner ear labyrinth deformation field representation method is applied to the inner ear labyrinth morphology classification task. The experimental results show that this method can effectively express the differences in the local structural details of the inner ear labyrinth and has a good ability to distinguish different morphological types of data.

[0048] Features of the present invention:

[0049] 1. A method for representing the inner ear labyrinth morphology based on differential signs is proposed, which enables the model to learn the knowledge of inner ear labyrinth morphology difference classification with only a small number of sample examples. The model can autonomously simulate and generate the change process from normal data to target morphology data, thereby generating a large number of pseudo samples.

[0050] 2. In terms of the specific implementation technology of the inner ear labyrinth morphological representation, a network architecture based on a three-dimensional deformation field is proposed. The network architecture uses a deformation registration network. The input image includes a normal image sequence and a test sequence. The network output result is a deformation field of the morphological change from the test sequence to the normal sequence. By adjusting the deformation field amplitude parameter, a morphological diffusion sample from the normal to the test sequence can be generated. The model can simulate the change process from normal to target morphology, thereby generating sequence pseudo samples.

[0051] 3. During the model training phase, a diffusion network training strategy with variable hyperparameter embedding was adopted, which enables the model to be trained at random λ random The optimal solution under the current conditions is obtained under the given value. This further shows that the training strategy proposed by this method that changes the hyperparameters with the training epoch can effectively control the shape changes of different degrees.

[0052] 4. Due to the different inner ear morphologies, the corresponding registration deformation fields are also quite different. By statistically analyzing the deformation field information of the target structure area, the inner ear labyrinth morphology classification can be achieved with a small number of example samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 , Overall framework of the inner ear morphology classification method based on deformation field

[0054] Figure 2 , the inner ear labyrinth structure under different human head scans

[0055] Figure 3 , Calibrated data (left) and template (right)

[0056] Figure 4 , UNet network architecture used in the present invention

[0057] Figure 5 , Comparison of normal image before and after generation

[0058] Figure 6 , Comparison of target morphology images before and after generation

[0059] Figure 7 , t-SNE 2D visualization of the three-dimensional deformation field

[0060] Figure 8 , PCA 2D visualization of the three-dimensional deformation field DETAILED DESCRIPTION

[0061] The following is a description of the implementation examples of the present invention in conjunction with the accompanying drawings:

[0062] The present invention uses the temporal bone U-HRCT data set for training and prediction. A complete scan result is composed of bilateral ear data, the unilateral data contains 370 images axially, the scanning receptive field is 65mm×65mm×37mm, the spatial resolution is 650×650×370, the layer thickness is 0.1mm, the layer spacing is 0.1mm, the tube voltage is 100KV, and the tube current is 3.5mA. We retrospectively collected 517 unilateral U-HRCT data for training and generating a three-dimensional deformation field network. The data was reviewed by the Life Ethics Committee of Beijing Friendship Hospital Affiliated to Capital Medical University (2020-P2-156-01, 2020-P2-061-02), and the data set consisted of 269 left ears and 248 right ears. Among them, the training set was divided into 448 cases, the test set was divided into 68 cases, and the template fixed data was 1 case, and the right data was mirrored at the same time.

[0063] The data preprocessing adopted by the present invention includes desensitizing the patient information of U-HRCT data and normalizing the image Hu value.

[0064] The process of the inner ear labyrinth morphological field representation and morphological classification method based on difference example guidance is as follows:

[0065] a) Segmentation and alignment of inner ear labyrinth substructures

[0066] The first step is to use the existing U-HRCT-based inner ear labyrinth substructure segmentation network to segment the inner ear labyrinth substructure from the image data. The segmentation network used in this step is TransUNet, and the input model data is two-dimensional. After the model segments a whole case of data, the two-dimensional predicted structure of the model is integrated into a three-dimensional inner ear labyrinth mask.

[0067] In the second step, the segmentation mask is used to locate the target area ROI, and then the 3D image to be moved is aligned with the fixed image through rigid transformation.

[0068] b) Parametric expression based on variable morphology diffusion network:

[0069] Step 1: The network architecture used in the present invention is a convolutional neural network architecture similar to Unet, which consists of an encoder and a decoder with jump connections. The input 3D images f and m are spliced ​​into a single input data with a channel number of 2. The hyperparameter λ is multiplied as attention with the feature map before each convolution layer of the input encoder after dimension expansion and full connection layer transformation. In this experiment, the data size of the input encoder is Batch size×2×160×192×224, where 160×192×224 is the size of the input 3D image.

[0070] In the encoder structure, 3D convolution with a kernel size of 3 and a stride of 2 is used. Each convolution layer is followed by a LeakyReLU activation function with a slope of 0.2. The slope of 0.2 here is selected based on many studies and applications in the field of deep learning. By using strided convolution with a stride of 2, the spatial dimension of each layer is halved to effectively reduce the spatial resolution of the output feature map to achieve a downsampling effect, while adjusting the spatial receptive field of the feature map to better capture the local and global features of the input image. The output of the encoder is a high-dimensional feature tensor that contains an abstract representation of the input image.

[0071] The task of the decoder is to map the features extracted by the encoder to a deformation field that describes how to transform from the input image to the target image. The decoder usually includes multiple convolutional layers and upsampling layers (deconvolutional layers) to gradually restore the resolution of the deformation field. This allows the features learned in the encoding phase to be directly passed to the decoding phase that generates the registration field. The last layer of the decoder is the output layer, which generates a prediction of the deformation field. This deformation field describes how to transform from the input image to the target image, usually in a four-dimensional form, representing the displacement of each spatial voxel in the three directions of X, Y, and Z.

[0072] At the same time, there is a jump connection between the encoder and the decoder to fuse the low-level and high-level features extracted by the encoder to better guide the generation of the deformation field. This helps the network capture the local and global information of the input image. The specific encoding and decoding network framework is as follows Figure 4 shown.

[0073] In the second step, the role of the sampler in the spatial transformation network is to apply the transformation to the input image and align it with the target image, thereby achieving medical image registration generation. Multi-level segmentation model update optimization based on incremental learning: By inputting the image to be moved and the fixed 3D image into the variable morphology diffusion network, the 3D deformation field of the image to be moved to the fixed image is obtained.

[0074] In the third step, in order to control the different similarities between the data to be moved and the fixed image after diffusion, a hyperparameter λ is introduced. By simultaneously embedding the variable morphology diffusion network and the loss function constraint, the whole process of achieving smooth change from the image to be moved to the fixed image is simulated.

[0075] c) Classification of the inner ear labyrinth morphology

[0076] Step 1: Obtain the deformation field information between the data to be detected and the fixed normal data through the above method, and compare and analyze the deformation field between normal images and the deformation field between the target image and the normal image. The deformation field changes between normal images after different alignments can be divided into two parts. One part is the local misaligned offset change, and the other part can be summarized as the individual intra-class differences between the inner ear labyrinth structures of different people, which are manifested in the form of fluctuations in the diameter and curvature of the semicircular canals within the normal range. The above two parts of the change differences can be collectively referred to as the intra-class differences between normal data, and the deformation field transformation can be understood as the superposition of their intra-class differences. In addition to the two parts mentioned above, the deformation field changes from the target morphological data to the normal data also have class-to-class differences in its local variant structure compared to the normal structure. Therefore, the deformation field model of the transformation from variant data to normal data is different in the local area compared to the normal structure. Such as Figure 5 , 6 shown.

[0077] The second step: the present invention represents the deformation field of the inner ear labyrinth CT data, and according to the substructure segmentation results obtained in the previous work, performs ROI positioning on the semicircular canal, and extracts the three-dimensional deformation field information of the target structure. The ROI size is [3,30,80,80], where 3 represents the number of channels contained in each voxel, that is, the three spatial dimensions of X, Y, and Z, and [30,80,80] is the ROI size of the target structure. The three-dimensional deformation field information of the target structure and the normal target structure is subjected to t-SNE and PCA dimensionality reduction, thereby performing morphological classification on the data.

Claims

1. A method for representing and classifying the inner ear labyrinth morphological field based on morphological difference example guidance, characterized in that The following steps are involved: 1) Segmentation and alignment of inner ear labyrinth substructures: The first step is to use the existing segmentation network to segment the inner ear labyrinth substructure to be calibrated from the image data. The segmentation network used in this step is TransUNet, and the input data of the model is the two-dimensional slice data along the Z-axis direction. The training data set of the model is more than 10 complete U-HRCT image data and the corresponding inner ear labyrinth substructure annotation data; In the second step, ROI positioning and alignment are performed from the segmentation mask of the inner ear labyrinth substructure to be calibrated. The ROI region of interest is positioned on the original image according to the center point of the segmentation mask. For the alignment of the inner ear labyrinth structure, a data alignment method based on affine transformation is proposed. The coordinates of the center points of the vestibule, horizontal semicircular canal, superior semicircular canal and posterior semicircular canal are extracted using the substructure segmentation mask. The formula of 3D affine transformation is as follows: Among them, x ′ ,y ′ and z ′ Represent the coordinates after affine transformation, x, y and z represent the coordinates before affine transformation; T X ,T Y and T Z Represents the translation in the X, Y and Z directions respectively, is a 3D rotation matrix; when the coordinates of four non-coplanar points are selected, the right is a fourth-order square matrix, and there is a reversible matrix. Therefore, the affine transformation matrix can be obtained according to the coordinates of the corresponding points before and after the affine transformation, that is, In addition, due to the small field of view of U-HRCT imaging, each sequence changes from bilateral ear imaging of ordinary CT to unilateral imaging, and each data needs to be classified into left and right ears; the left and right ear data are understood as mirror data, and the horizontal mirror affine transformation matrix is ​​as follows: Where width is the size of the image in the x direction; T of the same side data X The translation voxel value will not exceed 50 pixels, so T can be determined by calculating the affine matrix obtained by the calibrated data and the left and right template data. X , T Y and T Z The size of the value determines the left and right sides of the data to be calibrated; 2) Parameterized representation based on variable morphology diffusion network: In the first step, the network architecture used consists of an encoder and a decoder with jump connections; the input 3D image fixed image f and the image to be transformed m are spliced ​​into a single input data with a channel number of 2. The hyperparameter λ is multiplied with the feature map before each convolution layer of the input encoder as attention after dimension expansion and full connection layer transformation; in the encoder structure, there is a LeakyReLU activation function after each convolution layer; by using strided convolution, the spatial dimension of each layer is halved to effectively reduce the spatial resolution of the output feature map to achieve downsampling effect, and at the same time adjust the spatial receptive field of the feature map to better capture the local and global features of the input image; After upsampling, the decoder will continue to extract features through convolutional layers. These convolutional layers are 3×3 convolutional kernels. After the convolutional layers of the decoder, the ReLU activation function is used. The last layer of the decoder is the output layer, which is used to generate the prediction of the deformation field. This deformation field represents the displacement of each spatial voxel in the three directions of X, Y, and Z in the spatial coordinate system in a four-dimensional form. At the same time, there is a jump connection between the encoder and the decoder to fuse the low-level and high-level features extracted by the encoder together. In the second step, the role of the sampler in the spatial transformer network is to apply the transformation to the input image and align it with the target image, thereby achieving medical image registration generation; By minimizing The difference between and f is used to learn the optimal parameter value. Specifically, m is the 3D image to be transformed, Its transformation function is represents the transformed image, the transformation operation is performed by the spatial transformer network (STN), f is a fixed 3D image; for each voxel p of the 3D image m before transformation, the transformed voxel position p′=p+u(p) in m is calculated, where p is the voxel position before transformation and u(p) is the transformation function Transformation offset of position p; since the pixel coordinates of the image are defined only as integers, linear interpolation is required at the eight neighboring voxels of each voxel: Where z(p′) is the set of neighboring voxels of p′, voxel q is the voxel point in the set, m(q) is the voxel value of the input feature map m at voxel q, and d is the dimension of the iteration space domain Ω; STN transforms the input data by rotating, scaling or twisting it to enhance the robustness of the model to geometric transformations. The third step is to design the loss function in the model training phase The loss function can be divided into two parts: the similarity loss function between the constrained transformed image and the fixed image And the smoothness loss function that constrains the spatial variation of the deformation field in The calculation formula is as follows: is the similarity loss function Two commonly used loss functions are adapted; the first is the mean square error loss function MSE; MSE is calculated by squaring the difference between the predicted value and the true value and taking the average; specifically, for a 3D spatial transformation task, assuming that the predicted value voxel value is The true voxel value is h, and the number of voxels is n. The calculation formula of MSE is as follows: The second one is and the local cross-correlation loss function of f, The definition of f is the same as step 2; and Respectively represent the image f around position p 3 The local average intensity value within the voxel volume and the transformed image Around position p n 3 The local average intensity value within the voxel volume; The specific formula is where p j Iterate n around p 3 The volume of f(p j ) is the voxel p j The voxel value at image f; in the present invention, n=9; The local cross-correlation function of and f is as follows: The local cross-correlation loss function is: Diffusion regularization is used on the spatial gradient of displacement u to constrain the network model to generate a smooth displacement field. Here The difference between voxels connected to voxel p is used to approximate the spatial gradient. and Here x, y, and z represent the three directions of voxel p in the spatial coordinate system; The fourth step is diffusion network training based on variable hyperparameter embedding. In the training phase, a dynamic hyperparameter training method is used; specifically, a real number λ between 0 and 1 is randomly generated in each epoch. random As the hyperparameter of this epoch; then, the model obtains the hyperparameter setting of this round through the hyperparameter embedding method in the first step above, and then outputs the deformation field; When calculating the loss function, the following loss function is used: To constrain the model; During the training phase, the model is trained with 3000-5000 epochs.

2. The method according to claim 1, characterized in that Also includes: 3) Morphological classification of target structure: In the first step, the deformation field changes between normal images after different alignments are divided into two parts. One part is the offset change of local misalignment in the segmentation and positioning alignment of the inner ear labyrinth substructures mentioned above, and the other part can be summarized as the individual intra-class differences between the inner ear labyrinth structures of different people, which are manifested in the fluctuations of the semicircular canal diameter and curvature within the normal range. The above two parts of the difference are collectively referred to as the intra-class differences between normal data, and the deformation field transformation is the superposition of their intra-class differences. In addition to the two parts mentioned above, the deformation field change of morphological difference data to normal data also includes the class difference change of its local target structure compared with the normal structure; In the second step, the semicircular canal is located in ROI and the three-dimensional deformation field information of the target structure is extracted. The ROI size is [3, 30, 80, 80]. The three-dimensional deformation field information of the variant target structure and the normal target structure is reduced in dimension by t-SNE. The information is also reduced in dimension by PCA (principal component analysis) and visualized.

Images