An adapter-based medical image substructure extension segmentation method

Abstract

This invention discloses an adapter-based method for expanding and segmenting substructures in medical images, belonging to the field of medical image analysis and segmentation technology. Addressing the problems of limited labeled samples, high training costs, low segmentation accuracy of newly added small structures, and uneven accuracy between new and old structures in expanding and segmenting substructures in medical images, this invention inserts a lightweight adapter with self-attention after each feature block of the pre-trained TransUNet ViT encoder. It employs a hierarchical parameter freezing strategy, fine-tuning only 1 / 10 of the model parameters, and dynamically increases the loss weight of small structures using a difficulty-aware hybrid loss function. This allows for rapid expansion of substructure segmentation tasks, with all structural segmentation accuracy reaching clinically usable standards. It can be applied to scenarios such as assisted diagnosis, surgical planning, and postoperative evaluation.

Description

Technical Field

[0001] This invention belongs to the field of medical image analysis and deep learning medical image segmentation technology, specifically an adapter-based medical image substructure expansion segmentation method. Background Technology

[0002] Precise segmentation of medical images is a core foundation for clinical diagnosis, surgical planning, and medical research. With the increasing demands for granular image analysis in clinical practice, substructure-based segmentation has become a common requirement in the field of medical image segmentation. This often involves adding finer-grained substructure segmentation tasks to an already trained basic model capable of segmenting large, coarse-grained structures, without needing to re-annotate the entire training data or retrain the entire model. This need is widespread in image analysis scenarios across multiple clinical departments: for example, in temporal bone CT segmentation in otology, the segmentation has expanded from five major inner ear structures, such as the cochlea and vestibule, to include smaller substructures like the common crus; in lung CT segmentation in respiratory medicine, the segmentation has expanded from large lung lobe structures to include lung segment substructures; and in brain MRI segmentation in neurology, the segmentation has expanded from large brain regions to include substructures like brain nuclei.

[0003] Currently, the main technical bottlenecks for extended segmentation tasks targeting this type of substructure are as follows: First, the annotation of fine structures in medical images is a high-barrier and costly process, requiring manual annotation by professional physicians with clinical anatomical knowledge in the relevant field. The number of available high-quality substructure annotation samples is extremely limited. If the traditional method of full-scale fine-tuning of deep learning models is used, not only does it require updating all model parameters, resulting in long training times and high hardware resource requirements, but it is also highly susceptible to overfitting due to insufficient sample size, leading to poor model generalization ability. Furthermore, full-scale fine-tuning can destroy the general features and coarse-grained large-structure segmentation capabilities already learned by the basic model, causing a decrease in the accuracy of the original large-structure segmentation and an imbalance in the accuracy between old and new structures.

[0004] Second, the general adapter fine-tuning methods that have emerged in recent years can significantly reduce the number of training parameters by inserting lightweight adapter modules into the pre-trained model, training only the adapter and task-related layer parameters, and freezing the backbone network. However, these methods are designed for natural image scenes and do not take into account the spatial structural correlation features of medical images (i.e., substructures usually exist in the fixed anatomical positions of the corresponding large structures and have clear spatial constraints). They are not good at extracting features of small structures in medical images and cannot meet the needs of medical image substructure segmentation.

[0005] Third, medical imaging substructures generally suffer from significant class imbalance. The newly added fine-grained substructures typically have a low pixel ratio. Traditional loss functions are dominated by the background and coarse-grained large structures, which account for a very high percentage. The model does not focus on learning small structure features, which can easily lead to missed detections and incomplete segmentation of small structures, making it difficult to meet the clinical requirements for accuracy across all structures.

[0006] Therefore, an adapter fine-tuning method is needed to adapt to multi-scale and multi-structure segmentation of medical images, and to solve the problem of uneven segmentation accuracy between old and new target classes when the segmentation model expands the substructure division target. Summary of the Invention

[0007] To address the problems existing in the background technology, this invention proposes an adapter-based medical image substructure expansion and segmentation method. This method solves the problems of high cost of full-scale fine-tuning training, easy overfitting, damage to the original coarse-grained structure segmentation capability, general adapters not adapting to medical image spatial features, insufficient small structure feature extraction capability, and class imbalance leading to low segmentation accuracy of newly added fine-grained substructures and uneven accuracy between new and old structures in medical image substructure expansion and segmentation scenarios. It achieves efficient and stable transfer from pre-trained coarse-grained segmentation models to fine-grained substructure segmentation models, and provides multi-scenario medical image substructure expansion and segmentation services with low annotation requirements, low computational overhead, and full-structure accuracy.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: An adapter-based method for expanding and segmenting substructures in medical images includes the following steps: S1. Obtain a pre-trained medical image segmentation basic model, which is used to segment the first group of foreground structures; obtain the target segmentation dataset after substructure expansion, wherein the annotation of the target segmentation dataset includes the second group of foreground structures, and the second group of foreground structures is the set of structures after adding at least one target substructure to the first group of foreground structures. S2. Insert an adapter module after each feature extraction module of the encoder in the basic medical image segmentation model, and adjust the segmentation output layer according to the total number of categories after the substructure expansion to obtain the reconstructed fine-tuned model. S3. Freeze all pre-trained parameters of the encoder of the medical image segmentation basic model, and set the parameters of the adapter module, decoder, and adjusted segmentation output layer as trainable parameters. S4. Construct a hybrid loss function, which calculates the loss for all foreground structures and dynamically assigns loss weights based on the historical segmentation accuracy of each foreground structure. S5. Train the fine-tuned model using the target segmentation dataset to obtain the substructure extended segmentation model; S6. Input the medical image to be segmented into the substructure extended segmentation model and output the segmentation result containing all the second group of foreground structures.

[0009] Specifically, the adapter module performs feature transformation according to the following steps: S21. Perform layer normalization on the input features: ; in This is the input feature vector of the adapter module. These are the feature vectors after layer normalization. S22. Perform down-projection dimensionality reduction on the layer-normalized feature vectors: ; in Here is the weight matrix for the lower projection layer. This is the bias vector for the lower projection layer. The bottleneck feature vector obtained by dimensionality reduction; S23. After processing the bottleneck feature vector by inputting it into the Gaussian error linear activation unit, input it into the multi-head self-attention unit to extract spatial correlation features, as shown in the formula: ; in, Let Gaussian error be the linear activation function. For bullish self-attention operations, The feature vector after spatial feature enhancement; S24. Perform an up-projection dimensionality increase operation on the feature vector after spatial feature enhancement to restore it to the dimension of the adapter input features. The formula is as follows: ; in Here is the weight matrix of the upper projection layer. This is the bias vector of the upper projection layer. These are the feature vectors after dimensionality increase; S25. Perform a residual connection between the upgraded feature vector and the original input feature vector of the adapter, and output the feature vector processed by the adapter, as shown in the formula: ; in The scaling factor for the residual connection. This is the output feature vector of the adapter module.

[0010] Specifically, the bottleneck dimension of the downprojection dimensionality reduction operation satisfies ,in Bottleneck feature vector Dimensions Input feature vectors for the adapter The original hidden layer dimension; the number of attention heads in the multi-head self-attention operation is set to 4; the scaling factor of the residual connection is 1.0.

[0011] Specifically, the pre-trained medical image segmentation base model is the TransUNet model. The encoder of the TransUNet model consists of a ResNet50 convolutional backbone network and a visual Transformer feature extraction layer stacked together. The visual Transformer feature extraction layer is composed of several Block feature extraction modules stacked together. The adapter module is inserted into the output of each Block feature extraction module. The decoder of the TransUNet model is a multi-level upsampling feature fusion structure. The feature transformation formula for adjusting the segmentation output layer based on the total number of categories after substructure expansion is as follows: ; in, This is the fused feature map output by the decoder. The kernel size is The number of output channels is Convolution operation, The total number of categories after expanding the substructure. This is the multi-class predicted feature map output by the adjusted segmentation output layer.

[0012] Specifically, step S3 includes: S31. Load the pre-trained weights of the TransUNet model, delete the parameters of the original segmentation output layer and the parameters corresponding to the adapter module in the pre-trained weights, and transfer the remaining pre-trained weights to the reconstructed fine-tuned model. S32. Freeze all parameters of the ResNet50 convolutional backbone network and the visual Transformer feature extraction layer of the TransUNet encoder, keeping them fixed and unupdated during fine-tuning training. S33. Set all inserted adapter modules, decoders, and adjusted segmentation output layer parameters as trainable parameters.

[0013] Specifically, the hybrid loss function is composed of a weighted average of cross-entropy loss and difficulty-aware Dice loss, and the total loss is calculated using the following formula: ; in, Cross-entropy loss is used to learn the category boundary features of each structure; Difficulty-aware Dice loss is used to dynamically adjust the loss weights for different foreground structures to alleviate the class imbalance problem; The formula for calculating the cross-entropy loss is: ; in, This represents the total number of pixels in a single batch of imported medical images. This represents the total number of categories after expanding the substructures, including the background category and all categories corresponding to the second group of foreground structures. For pixels Corresponding category Unique hot-coded real tags, Predict pixels for the model Category The probability of.

[0014] Specifically, the difficulty-aware Dice loss is calculated only for all second-group foreground structures, skipping the background category. The calculation steps include: S41. For each second group of foreground structures, calculate the segmentation Dice coefficient for that category during a single batch of training. The calculation formula is as follows: ; in, For the category index of the second group of foreground structures, The category predicted by the model The set of pixels, Category in real labels The set of pixels, The number of pixels in the set. This is a smoothing term used to avoid calculation errors where the denominator is 0; The corresponding single-class Dice loss is ; S42. Maintain a historical queue of Dice coefficients for each second group of foreground structures, and calculate the historical average Dice coefficient using the following formula: ; in, The length of the historical queue, For the first In the next training iteration, the class Dice coefficient; S43. Use the difference between 1 and the historical average Dice coefficient as the difficulty coefficient for the category. : ; After normalizing the difficulty coefficients of all foreground structures, the results are clipped to a preset range to obtain the final loss weights. The calculation formula is: ; ; in, For category Normalized difficulty coefficient This represents the total number of foreground structures in the second group. This is a numeric clipping function used to restrict input values ​​to a preset range. Inside; S44. Calculate the difficulty-perceived Dice loss, the formula is as follows: .

[0015] Specifically, the model fine-tuning training process in step S5 performs the following operations: After each round of training, the average Dice coefficient of all second-group foreground structures is calculated on the validation set as a model accuracy metric. If the average Dice coefficient of the validation set does not improve after a preset number of rounds, training is terminated early, and the weight of the model with the highest average Dice coefficient on the validation set is saved as the final weight of the substructure expansion segmentation model.

[0016] Specifically, the reasoning and segmentation process in step S6 includes: S61. Perform preprocessing operations on the medical image to be segmented, consistent with the training set of the target segmentation dataset. After inputting the substructure to expand the segmentation model, forward propagation is used to obtain multi-channel predicted feature maps. S62. Perform Softmax normalization on the multi-channel prediction feature map to obtain the prediction probability of each target category for each pixel; S63. Select the category with the highest predicted probability for each pixel as its label and generate an initial segmentation mask; perform morphological post-processing on the initial segmentation mask to remove discrete noise points and obtain the final segmentation result.

[0017] Specifically, the method is applied to the extended segmentation of the inner ear substructure in temporal bone CT. The first group of foreground structures consists of five inner ear structures: cochlea, vestibule, superior semicircular canal, lateral semicircular canal, and posterior semicircular canal. The second group of foreground structures consists of six inner ear structures after adding the common foot structure on the basis of the first group of foreground structures. The target segmentation dataset is a temporal bone CT image dataset labeled with six inner ear structures and the background.

[0018] In summary, the beneficial technical effects of the present invention are as follows: 1. Strong model transfer adaptability and excellent small structure feature extraction capability: The lightweight adapter module with multi-head self-attention designed in this invention is inserted into the output of each feature extraction module of the encoder. On the one hand, it completely preserves the general image features and coarse-grained large structure segmentation capability learned by the pre-trained model through residual connections, avoiding the degradation of the original segmentation capability. On the other hand, it captures the inherent spatial constraint relationship between structures in medical images through the multi-head self-attention module. Compared with the general non-attention adapter, it can significantly improve the feature extraction capability and segmentation accuracy of fine-grained small substructures.

[0019] 2. Low training cost and low requirement for labeled sample size: This invention adopts a hierarchical parameter freezing strategy, setting only the adapter module, decoder, and adjusted output layer parameters as trainable parameters. The total number of trainable parameters accounts for only a very small part of the total parameters of the original model. Compared with the full fine-tuning scheme, it can significantly shorten the training time and reduce the hardware computing power threshold. Moreover, it does not require re-labeling the entire coarse-grained structure dataset. Only a small number of labeled samples with newly added substructures are needed to complete the training, effectively avoiding the overfitting problem under small sample size. Both labeling cost and training cost are greatly reduced.

[0020] 3. Balanced accuracy across all structures, eliminating the issue of accuracy deviation between old and new structures: The difficulty-aware hybrid loss function designed in this invention calculates dynamic weights only for foreground structures. It automatically assigns loss weights based on the historical segmentation accuracy of each structure, with higher loss weights for structures with lower segmentation accuracy. This forces the model to learn the features of all old and new structures in a balanced manner, effectively solving the problem of missed detection and incomplete segmentation of small structures caused by class imbalance. There is no significant difference in the segmentation accuracy of all foreground structures, completely avoiding the imbalance problem of high accuracy for large structures and low accuracy for small structures in existing solutions.

[0021] 4. High versatility and adaptability to various clinical scenarios: The solution of this invention does not depend on specific image modalities and anatomical locations. It can be quickly adapted to all medical image segmentation scenarios with substructure expansion requirements, such as temporal bone CT inner ear segmentation, lung CT lung segmentation, and brain MRI nucleus segmentation. It can be implemented without structural adjustments to the core solution. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is an example diagram of the method of the present invention; Figure 3 This is an example diagram of an adapter for the method of the present invention; Figure 4 This is a comparison image of the 3D segmentation results before and after in an embodiment of the present invention. Detailed Implementation

[0023] To make the technical means, creative features, objectives and effects of this invention clearer and easier to understand, the invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0024] Example like Figure 1 As shown, the present invention provides a medical image substructure expansion and segmentation method based on an adapter, which specifically includes the following steps: S1. Preparation of pre-trained models and datasets; This step selects a pre-trained encoder-decoder structure medical image segmentation model as the base model, which has achieved stable segmentation of the first set of coarse-grained foreground structures. A target segmentation dataset with substructure expansion is constructed. The dataset annotation includes the second set of foreground structures. The second set of foreground structures is the set of structures after adding at least one fine-grained target substructure to the first set of foreground structures. Gray-level normalization and size uniformity preprocessing are performed on all images, and data augmentation operations are performed on the training set to improve the model's generalization ability.

[0025] Specifically, this embodiment uses the R50-ViT-B_16 version of TransUNet as the pre-trained base model. This model has achieved stable segmentation of the first group of foreground structures (five major inner ear structures: cochlea, vestibule, superior semicircular canal, lateral semicircular canal, and posterior semicircular canal). Its encoder consists of a ResNet50 convolutional backbone and stacked ViT-B_16 feature extraction layers, the decoder is a multi-level upsampling feature fusion structure, and the segmentation output layer is a 3×3 convolutional layer. The overall structure is as follows: Figure 2 As shown.

[0026] For dataset construction: 427 cases of temporal bone ultra-high resolution CT images were included, divided proportionally into a training set of 359 cases, a validation set of 40 cases, and a test set of 28 cases. All CT images underwent preprocessing, including grayscale value normalization to the [0,1] range and cropping to 224×224 pixels. The training set underwent additional data augmentation operations, including random rotation and random left / right flipping. The dataset labels are 7 categories of pixel-level segmentation masks manually labeled by physicians, corresponding to the second group of foreground structures (6 inner ear structures after adding the total foot structure on the basis of the first group of structures) + background. Single image slices are stored in npz format. At the same time, training, validation, and test list files are constructed to store the image paths, which facilitates batch loading of models.

[0027] S2, Model Reconstruction with Adapter; This step inserts a lightweight adapter module after each feature extraction module of the base model encoder. The number of output channels of the segmentation output layer is adjusted according to the total number of categories after the substructure expansion. This adapts the substructure expansion task without changing the backbone structure of the base model, resulting in a reconstructed fine-tuned model.

[0028] Specifically, in this embodiment, after the 12 Block feature extraction modules of the TransUNet ViT encoder, the following are inserted respectively: Figure 3 The adapter module shown performs feature transformation according to the following steps: S21. Perform layer normalization on the input features to stabilize the feature distribution: ; in This is the input feature vector of the adapter module. The original hidden layer dimension of TransUNet These are the feature vectors after layer normalization. S22. Perform down-projection dimensionality reduction on the layer-normalized feature vectors, mapping high-dimensional features to a low-dimensional bottleneck space to reduce computational cost: ; in Here is the weight matrix for the lower projection layer. Given the bias vector of the lower projection layer, the bottleneck dimension of the lower projection dimensionality reduction operation satisfies... , The bottleneck feature vector obtained by dimensionality reduction. Bottleneck feature vector The dimension; S23. After processing the bottleneck feature vector by inputting it into a Gaussian error linear activation unit, it is then input into a multi-head self-attention unit to extract spatial correlation features, thereby improving the ability to extract small structure features. The formula is as follows: ; in, Let Gaussian error be the linear activation function. For multi-head self-attention operation, this embodiment sets it to 4. The feature vector after spatial feature enhancement; S24. Perform an up-projection dimensionality increase operation on the enhanced spatial feature vectors to restore them to the dimension of the adapter input features, in order to adapt to the input requirements of subsequent network layers. The formula is as follows: ; in Here is the weight matrix of the upper projection layer. This is the bias vector of the upper projection layer. These are the feature vectors after dimensionality increase; S25. Perform a residual connection between the upgraded feature vector and the original input feature vector of the adapter, preserving the original feature capabilities of the pre-trained model, and output the feature vector processed by the adapter, as shown in the formula: ; in This is the scaling factor for the residual join, with a value of 1.0. This is the output feature vector of the adapter module.

[0029] Subsequently, based on the total number of categories after substructure expansion (6 inner ear structures + background, a total of 7 categories), the number of 3×3 convolutional kernels in the original output layer was adjusted to 7, the number of output channels was matched with the number of target categories, and the feature transformation formula corresponding to the segmentation output layer was adjusted as follows: ; in, This is the fused feature map output by the decoder. The kernel size is Convolution operation with padding value of 1, The total number of categories after expanding the substructure. This is a 7-channel multi-class prediction feature map.

[0030] Finally, the model is configured, with the ViT patch size set to 16×16 and the number of skip connections set to 3, forming a complete TransUNet fine-tuning model with an adapter. In this embodiment, the model is built on the PyTorch framework and runs in a CUDA accelerated environment.

[0031] S3, Layered parameter freezing and trainable parameter configuration; After loading the pre-trained weights of the base model in this step, all pre-trained parameters of the encoder are frozen, and only the parameters of the adapter module, decoder, and adjusted segmentation output layer are set as trainable parameters. This reduces the number of training parameters, preserves the pre-trained feature capabilities, and significantly reduces training overhead.

[0032] Specifically, the steps include: S31. Load the pre-trained weights of the TransUNet model, delete the parameters of the original segmentation output layer and the corresponding parameters of the adapter module in the pre-trained weights, and transfer the remaining pre-trained weights to the reconstructed fine-tuned model to realize the reuse of pre-trained features. S32. Freeze all parameters of the ResNet50 convolutional backbone network and the visual Transformer feature extraction layer of the TransUNet encoder. These parameters remain fixed and do not update during training, thus fully preserving the general feature extraction capability of the pre-trained model and the segmentation capability of the original five inner ear structures. S33. Set all inserted adapter modules, decoders, and parameters of the adjusted segmentation output layer as trainable parameters. The total number of trainable parameters is about 1 / 10 of the original total parameters of TransUNet, which greatly reduces the training computing power requirement and the risk of overfitting.

[0033] S4. Construction of a difficulty-aware hybrid loss function; This step constructs a hybrid loss function consisting of cross-entropy loss and difficulty-aware Dice loss weighted together. The difficulty-aware Dice loss is calculated only for the foreground structure. The loss weight is dynamically assigned according to the historical segmentation accuracy of each structure. The structure with lower segmentation accuracy has a higher weight, thereby solving the problems of class imbalance and uneven accuracy between new and old structures. Specifically, the hybrid loss function is composed of a weighted average of cross-entropy loss and difficulty-aware Dice loss, and the total loss is calculated as follows: ; in, Cross-entropy loss is used to learn the class boundary features of each structure, and its calculation formula is: ; in, This represents the total number of pixels in a single batch of imported medical images. The total number of categories after expanding the substructure. For pixels Corresponding category Unique hot-coded real tags, Predict pixels for the model Category The probability is the value after Softmax normalization; To achieve a difficulty-aware Dice loss, calculations are performed only for the six inner ear foreground structures, skipping background categories to avoid excessive background pixel proportions interfering with loss calculations. The calculation steps include: S41, For each second group of foreground structures Calculate the Dice coefficient for this category during a single batch of training. The formula is as follows: ; in, For the category index of the second group of foreground structures, The category predicted by the model The set of pixels, Category in real labels The set of pixels, The number of pixels in the set. This is a smoothing term used to avoid calculation errors where the denominator is 0; Therefore, the corresponding single-class Dice loss is ; S42, maintain a length of [length] for each foreground structure. The historical Dice score queue is used to calculate the historical average Dice coefficient. The calculation formula is as follows: ; in, The length of the historical queue, For the first In the next training iteration, the class The Dice coefficient; using historical averages can smooth out fluctuations in a single iteration and accurately reflect the actual learning difficulty of the structure; S43. Use the difference between 1 and the historical average Dice coefficient as the category. Difficulty level The higher the difficulty, the higher the weight: ; After normalizing the difficulty coefficients of all foreground structures, the weights are clipped to the range [0.5, 2.0] to ensure training stability, thus obtaining the final loss weights. The calculation formula is: ; ; in, For category Normalized difficulty coefficient This represents the total number of foreground structures in the second group. This is a numeric clipping function used to restrict input values ​​to a preset range; S44. Multiply the single-class Dice loss of each foreground structure by its corresponding final weight and then average them to obtain the difficulty-aware Dice loss: ; S5, Model fine-tuning training; This step uses the target segmentation dataset to train the fine-tuned model, only updating the trainable parameters. During training, the model accuracy is evaluated using the validation set, and an early stopping strategy is adopted to avoid overfitting, resulting in a substructure-extended segmentation model. The specific implementation of this embodiment is as follows: (1) Data loading: Build a dataset loader, set the DataLoader of the training set to shuffle=True, configure random seed in worker_init_fn to ensure reproducibility of training, set the batch size to 8, and batch load image slices and label masks; (2) Training process: Input the image slices into the model, forward propagation to obtain 7-channel predicted feature maps, input the mixed loss function to calculate the loss value, use the AdamW optimizer to backpropagate and update the trainable parameters, and set the learning rate to 1e-4.

[0034] (3) Early termination and weight preservation: After each round of training, the average Dice coefficient of the six inner ear structures is calculated on the validation set as the accuracy index. The training / validation loss and Dice curve are visualized through TensorBoard. If the average Dice coefficient of the validation set does not improve for 10 consecutive rounds, the training is terminated early, and the model weight with the highest accuracy on the validation set is preserved as the final substructure expansion segmentation model weight.

[0035] S6. Reasoning Segmentation and Result Output; The image to be segmented is preprocessed in the same way as the training set and then input into the model. After forward propagation, class probability calculation, and post-processing, the final segmentation result is obtained.

[0036] The process specifically includes: S61. Perform preprocessing operations on the medical image to be segmented that are consistent with the training set of the target segmentation dataset, including grayscale normalization and size cropping. Do not perform data augmentation. After inputting the substructure to expand the segmentation model, forward propagation is performed to obtain a 7-channel predicted feature map. S62. Perform Softmax normalization on the 7-channel predicted feature map to obtain the predicted probability of each pixel for each target category. The calculation formula is as follows: ; in, To predict pixels in the feature map Corresponding category The original output value; S63. Select the class with the highest predicted probability for each pixel as its label and generate an initial segmentation mask; S64. Post-processing and result output: Perform morphological opening and closing operations on the initial segmentation mask to remove discrete noise points and obtain the final segmentation result.

[0037] The segmentation results of this embodiment are verified as follows: ① Examples of 3D segmentation results Figure 4 As shown: the left side is the output of the original 5-structure segmentation model, and the right side is the output of the model in this embodiment. The total foot structure at the connection between the upper and posterior semicircular canals can be clearly identified, which meets the substructure expansion requirements.

[0038] ②The DSC indices for each structure in the 28 test cases are shown in the table below: Table 1 - Test Set Segmentation Results: DSC (%) Average Values As shown in the table above, the DSC data for the cochlea, vestibule, superior semicircular canal, posterior semicircular canal, lateral semicircular canal, and common crus all meet the clinically usable standards. Furthermore, the accuracy of the original five major structures did not decrease significantly, and the accuracy difference between the old and new structures was less than 5%, thus solving the problem of low accuracy of newly added small structures in traditional methods.

[0039] Therefore, this invention provides an adapter-based medical image substructure expansion segmentation method. Through a core design employing a lightweight adapter module with multi-head self-attention, a low-parameter fine-tuning strategy with hierarchical parameter freezing, and a difficulty-aware hybrid loss function with dynamic weight allocation, it achieves a highly efficient and stable transfer process from a pre-trained coarse-grained medical image segmentation model to a segmentation model with newly added fine-grained substructures, characterized by no capability degradation and low sample dependence. This solves the problems of high training costs, overfitting, and damage to the original coarse-grained structure segmentation capabilities associated with traditional substructure expansion segmentation schemes, as well as the limitations of general adapters in adapting to the spatial constraints of medical images, leading to small... The shortcomings of insufficient structure extraction capabilities and class imbalance leading to low segmentation accuracy of newly added fine-grained substructures and uneven accuracy between new and old structures have been addressed. This paper improves the segmentation accuracy of fine-grained substructures, the balance of overall structural accuracy, and the model's generalization ability. It significantly reduces the cost of fine structure annotation in medical images, the computational overhead of model training, and the technical threshold for upgrading segmentation tasks. It provides an efficient and feasible solution for multi-scale structure precision segmentation services in medical image analysis scenarios with substructure expansion needs, such as assisted diagnosis and surgical planning. It ensures the stable achievement of full-structure segmentation accuracy that meets clinical requirements under conditions of few labeled samples and limited clinical computing resources.

[0040] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that the parameters of this embodiment can be adaptively adjusted according to actual needs. For example, the adapter bottleneck dimension, the number of attention heads, the length of the historical queue, the loss weight ratio can be adjusted, or the basic model can be replaced with other encoder-decoder structure segmentation models such as SwinUNet or UNet. Alternatively, the solution can be applied to other clinical scenarios such as lung segment extension segmentation in lung CT and brain MRI nucleus extension segmentation. Such adjustments do not depart from the core concept of the present invention and fall within the protection scope of the present invention.

Claims

1. An adapter-based method for extending medical image substructure segmentation, used to add substructure segmentation tasks to an already trained basic segmentation model, characterized in that, The method includes the following steps: S1. Obtain a pre-trained medical image segmentation basic model, which is used to segment the first group of foreground structures; obtain the target segmentation dataset after substructure expansion, wherein the annotation of the target segmentation dataset includes the second group of foreground structures, and the second group of foreground structures is the set of structures after adding at least one target substructure to the first group of foreground structures. S2. Insert an adapter module after each feature extraction module of the encoder in the basic medical image segmentation model, and adjust the segmentation output layer according to the total number of categories after the substructure expansion to obtain the reconstructed fine-tuned model. S3. Freeze all pre-trained parameters of the encoder of the medical image segmentation basic model, and set the parameters of the adapter module, decoder, and adjusted segmentation output layer as trainable parameters. S4. Construct a hybrid loss function, which calculates the loss for all foreground structures and dynamically assigns loss weights based on the historical segmentation accuracy of each foreground structure. S5. Train the fine-tuned model using the target segmentation dataset to obtain the substructure extended segmentation model; S6. Input the medical image to be segmented into the substructure extended segmentation model and output the segmentation result containing all the second group of foreground structures.

2. The adapter-based medical image substructure expansion and segmentation method according to claim 1, characterized in that, The adapter module performs feature transformation according to the following steps: S21. Perform layer normalization on the input features: ； in This is the input feature vector of the adapter module. These are the feature vectors after layer normalization. S22. Perform down-projection dimensionality reduction on the layer-normalized feature vectors: ； in Here is the weight matrix for the lower projection layer. This is the bias vector for the lower projection layer. The bottleneck feature vector obtained by dimensionality reduction; S23. After processing the bottleneck feature vector by inputting it into the Gaussian error linear activation unit, input it into the multi-head self-attention unit to extract spatial correlation features, as shown in the formula: ； in, Let Gaussian error be the linear activation function. For multi-head self-attention operation, The feature vector is the feature vector after spatial feature enhancement; S24. Perform an up-projection dimensionality increase operation on the feature vector after spatial feature enhancement to restore it to the dimension of the adapter input features. The formula is as follows: ； in Here is the weight matrix of the upper projection layer. This is the bias vector of the upper projection layer. These are the feature vectors after dimensionality increase; S25. Perform a residual connection between the upgraded feature vector and the original input feature vector of the adapter, and output the feature vector processed by the adapter, as shown in the formula: ； in The scaling factor for the residual connection. This is the output feature vector of the adapter module.

3. The adapter-based medical image substructure expansion and segmentation method according to claim 2, characterized in that: The bottleneck dimension of the downprojection dimensionality reduction operation satisfies ,in Bottleneck feature vector Dimensions Input feature vectors for the adapter The original hidden layer dimension; the number of attention heads in the multi-head self-attention operation is set to 4; the scaling factor of the residual connection is 1.

0.

4. The adapter-based medical image substructure expansion and segmentation method according to claim 1, characterized in that: The pre-trained medical image segmentation base model is the TransUNet model. The encoder of the TransUNet model consists of a ResNet50 convolutional backbone network and a visual Transformer feature extraction layer stacked together. The visual Transformer feature extraction layer is composed of several Block feature extraction modules stacked together. The adapter module is inserted into the output of each Block feature extraction module. The decoder of the TransUNet model is a multi-level upsampling feature fusion structure. The feature transformation formula for adjusting the segmentation output layer based on the total number of categories after substructure expansion is as follows: ； in, This is the fused feature map output by the decoder. The kernel size is The number of output channels is Convolution operation, The total number of categories after expanding the substructure. This is the multi-class predicted feature map output by the adjusted segmentation output layer.

5. The adapter-based medical image substructure expansion and segmentation method according to claim 4, characterized in that, Step S3 specifically includes: S31. Load the pre-trained weights of the TransUNet model, delete the parameters of the original segmentation output layer and the parameters corresponding to the adapter module in the pre-trained weights, and transfer the remaining pre-trained weights to the reconstructed fine-tuned model. S32. Freeze all parameters of the ResNet50 convolutional backbone network and the visual Transformer feature extraction layer of the TransUNet encoder, keeping them fixed and unupdated during fine-tuning training. S33. Set all inserted adapter modules, decoders, and adjusted segmentation output layer parameters as trainable parameters.

6. The adapter-based medical image substructure expansion and segmentation method according to claim 1, characterized in that, The hybrid loss function is composed of a weighted average of cross-entropy loss and difficulty-aware Dice loss, and the total loss is calculated using the following formula: ； in, Cross-entropy loss is used to learn the category boundary features of each structure; Difficulty-aware Dice loss is used to dynamically adjust the loss weights for different foreground structures to alleviate the class imbalance problem; The formula for calculating the cross-entropy loss is: ； in, This represents the total number of pixels in a single batch of imported medical images. This represents the total number of categories after expanding the substructures, including the background category and all categories corresponding to the second group of foreground structures. For pixels Corresponding category Unique hot-coded real tags, Predict pixels for the model Category The probability of.

7. The adapter-based medical image substructure expansion and segmentation method according to claim 6, characterized in that, The difficulty-aware Dice loss is calculated only for all second-group foreground structures, skipping the background category. The calculation steps include: S41. For each second group of foreground structures, calculate the segmentation Dice coefficient for that category during a single batch of training. The calculation formula is as follows: ； in, For the category index of the second group of foreground structures, The category predicted by the model The set of pixels, Category in real labels The set of pixels, The number of pixels in the set. This is a smoothing term used to avoid calculation errors where the denominator is 0; The corresponding single-class Dice loss is ; S42. Maintain a historical queue of Dice coefficients for each second group of foreground structures, and calculate the historical average Dice coefficient using the following formula: ； in, The length of the historical queue, For the first In the next training iteration, the class Dice coefficient; S43. Use the difference between 1 and the historical average Dice coefficient as the difficulty coefficient for the category. : ； After normalizing the difficulty coefficients of all foreground structures, the results are clipped to a preset range to obtain the final loss weights. The calculation formula is: ； ； in, For category Normalized difficulty coefficient This represents the total number of foreground structures in the second group. This is a numeric clipping function used to restrict input values ​​to a preset range. Inside; S44. Calculate the difficulty-perceived Dice loss, the formula is as follows: 。 8. The adapter-based medical image substructure expansion and segmentation method according to claim 7, characterized in that, In step S5, the model fine-tuning training process performs the following operations: After each round of training, the average Dice coefficient of all second-group foreground structures is calculated on the validation set as a model accuracy metric. If the average Dice coefficient of the validation set does not improve after a preset number of rounds, training is terminated early, and the weight of the model with the highest average Dice coefficient on the validation set is saved as the final weight of the substructure expansion segmentation model.

9. The adapter-based medical image substructure expansion and segmentation method according to claim 1, characterized in that, The reasoning and segmentation process in step S6 specifically includes: S61. Perform preprocessing operations on the medical image to be segmented, consistent with the training set of the target segmentation dataset. After inputting the substructure to expand the segmentation model, forward propagation is used to obtain multi-channel predicted feature maps. S62. Perform Softmax normalization on the multi-channel prediction feature map to obtain the prediction probability of each target category for each pixel; S63. Select the category with the highest predicted probability for each pixel as its label and generate an initial segmentation mask; perform morphological post-processing on the initial segmentation mask to remove discrete noise points and obtain the final segmentation result.

10. The adapter-based medical image substructure expansion and segmentation method according to claim 1, characterized in that, The method is applied to the extended segmentation of the inner ear substructure in temporal bone CT scan. The first group of foreground structures consists of five inner ear structures: cochlea, vestibule, superior semicircular canal, lateral semicircular canal, and posterior semicircular canal. The second group of foreground structures consists of six inner ear structures after adding the common foot structure on the basis of the first group of foreground structures. The target segmentation dataset is a temporal bone CT image dataset labeled with six inner ear structures and the background.

Images