Ultrasonic image segmentation method based on fine tuning of dynamic adaptation method

Abstract

The invention relates to an ultrasonic image segmentation method based on fine tuning of a dynamic adaptation method, which comprises the following steps of: acquiring ultrasonic images to form a data set, and evaluating a parameter matrix of an existing pre-training model SAM; the parameter matrix is distinguished through a preset threshold value, and then different adaptation schemes are adopted for the image encoder; based on the adaptive scheme, adding a detail feature enhancement fusion module, and constructing a model HATSAM; and inputting the ultrasonic image data set into the model HATSAM for training, and outputting an ultrasonic image segmentation result. According to the ultrasonic image segmentation method based on dynamic adaptation method fine tuning, through dynamic adaptation of parameter space completeness, different fine tuning schemes are selected for different completeness parameters, disturbance to a pre-training model is reduced, the model calculation amount is reduced, and the ultrasonic image segmentation efficiency is improved. Only a small number of trainable parameters need to be added, SAM is migrated to ultrasonic image segmentation, and the problem that a specific task needs to be trained independently in a traditional segmentation scheme is solved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of image processing, in particular to an ultrasound image segmentation method based on dynamic adaptation method fine tuning. BACKGROUND

[0002] Medical image segmentation aims to accurately locate the boundaries of each organ tissue and lesion in medical images, which helps doctors to correctly diagnose the patient's condition. At present, there are many researches dedicated to this field. However, due to the huge difference between imaging devices and organ tissues, most current methods can only perform tasks for specific organs or tissues, and lack the generalization ability for multi-organ segmentation.

[0003] Segment Anything Model, SAM, is a basic model for visual segmentation, which has excellent performance in a wide range of natural image segmentation and zero-shot generalization ability. SAM can dynamically segment the corresponding target according to the points, frames or masks provided by the user. This way makes SAM can adaptively adjust the segmentation target for different segmentation tasks. This feature is beneficial to multi-organ segmentation of medical images, but the huge difference between medical images and natural images makes it difficult for SAM to be directly applied to the segmentation task of medical images. And because the pre-training model has not been fully analyzed in the adaptability of the parameter space and the feature space in the downstream task, it is easy to cause excessive disturbance to the model and destroy the original performance or add redundant parameters, which slows down the inference speed. SUMMARY

[0004] The technical problem of the present application is to propose a dynamic adaptation based on parameter space and a detail feature enhancement hybrid module, to apply different fine tuning schemes to different completeness parameters, and to add denoising detail features to the feature space of the mask decoder according to the characteristics of ultrasound images, to solve the problem that traditional segmentation schemes need to be trained separately for specific tasks, and to increase the application range of ultrasound image segmentation.

[0005] The purpose of the present application is to solve the above problems, and to propose an ultrasound image segmentation method based on dynamic adaptation method fine tuning, comprising the following steps: S1: collect ultrasound images to form a data set, and evaluate the parameter matrix of the existing pre-training model SAM; S2: discriminate the parameter matrix by a preset threshold, and then adopt different adaptation schemes for the image encoder; S3: based on the adaptation scheme, increase the detail feature enhancement fusion module, and construct the model HATSAM; S4: input the ultrasound image data set into the model HATSAM for training, and output the ultrasound image segmentation result.

[0006] Further, in step S1, the ultrasound image data comprises ultrasound images of multiple organs, including images of multiple parts of the heart, breast and thyroid.

[0007] Preferably, in step S1, the pre-trained model SAM is an image segmentation base model SAM trained using the SA-1B dataset.

[0008] Further, in step S1, the following sub-steps are included: S11: collecting an ultrasound image dataset and constructing a pre-trained model SAM; S12: inputting a training set in the dataset into the pre-trained model SAM for round training, and counting training gradients of different position parameters in the pre-trained parameters ; S13: based on the training gradients , calculating the under-completeness of the corresponding position parameters ; the calculation formula is: ; In the formula, represents the training gradient of the parameter at the nth position in the matrix, represents the under-completeness of the parameter at the n position; S14: a threshold value is preset for determining whether the parameter is complete, when is greater than the threshold value , it is determined to be under-complete, and when is less than the threshold value , it is determined to be relatively complete; S15: counting the number of under-complete parameters of each parameter matrix, if the number of under-complete parameters of a single parameter matrix is greater than the threshold value , it is considered that the corresponding parameter matrix is under-complete, otherwise it is relatively complete.

[0009] Preferably, in step S2, the adaptation scheme includes: for the original parameter matrix corresponding to the under-complete parameter matrix, adding an Adapter branch in the adaptation scheme and setting the branch as trainable; for the original parameter matrix corresponding to the relatively complete parameter matrix, using singular value decomposition and setting the diagonal matrix as trainable, and freezing the rest of the parameters.

[0010] Further, the singular value decomposition includes the following steps: 1) obtaining a relatively complete parameter matrix from the pre-trained SAM model; 2) using singular value decomposition to split the parameter matrix into a left singular vector matrix and a right singular vector matrix.

[0011] Preferably, in step S3, the model HATSAM comprises training using the DEFFM module, including the following sub-steps: 1) Obtain the model image encoder layer 2 output features , layer 5 output features , and the last layer output features ; 2) Obtain the prompt features output by the original pre-trained model SAM , and the decoded pixel features ; 3) Use the prompt features to denoise and through the guide module Guide to obtain and , the calculation formula is: ; In the formula, represents the layer 5 output features, represents the prompt features, represents the features after denoising, and i represents the counting unit.

[0012] 4) Fuse the features using the fuse module, the calculation formula is: ; In the formula, represents the fused features, represents the features after denoising, represents the features after denoising, represents the last layer output features, represents the decoded pixel features.

[0013] Further, in step S3, the guide module Guide inputs the output features through depth separable convolution, cross-attention, and deconvolution, including the following steps: 1) Input the layer 2 output features , layer 5 output features to the depth separable convolution; 2) Perform cross-attention mechanism on the convolution results and the prompt features ; 3) Perform deconvolution operation on the results of the previous step.

[0014] Preferably, the DEFFM module further comprises calculating using deconvolution, and calculating the fused feature output, the calculation formula is: ; ; In the formula, denotes a convolution operation, denotes an inverse convolution operation; 、 and are constant coefficients, indicating the proportion of the corresponding features in the mixed features.

[0015] Compared with the prior art, the beneficial effects of the present application include: 1) The ultrasound image segmentation method based on dynamic adaptation method fine-tuning proposed in the present application reduces the disturbance to the pre-trained model by dynamic adaptation of parameter space completeness, selects different fine-tuning schemes for different completeness parameters, reduces the model calculation amount, and increases the efficiency of ultrasound image segmentation.

[0016] 2) The ultrasound image segmentation method based on dynamic adaptation method fine-tuning proposed in the present application enhances the mixed module by referencing detailed features, and according to the characteristics of ultrasound images, uses prompt features to guide the original detailed feature information to complete the fusion of detailed features and deep features layer by layer, enhances effective features while suppressing irrelevant noise.

[0017] 3) The ultrasound image segmentation method based on dynamic adaptation method fine-tuning proposed in the present application only needs to add a small number of trainable parameters to migrate SAM to ultrasound image segmentation, solving the problem of separate training of traditional segmentation schemes for specific tasks. BRIEF DESCRIPTION OF DRAWINGS

[0018] The present application will be further described below in conjunction with the drawings and examples.

[0019] Figure 1 The present application is an ultrasound image segmentation method based on dynamic adaptation method fine-tuning.

[0020] Figure 2 The present application is an ultrasound image segmentation method based on dynamic adaptation method fine-tuning.

[0021] Figure 3 The present application is an ultrasound image segmentation method based on dynamic adaptation method fine-tuning. DETAILED DESCRIPTION The present application will be further described below in conjunction with the drawings and examples.

[0022] S1: Collect ultrasound image data sets and evaluate the parameter matrix of the existing pre-trained model SAM; In step S1, the ultrasound image data includes ultrasound images of multiple organs, covering multiple sites such as the heart, breast, and thyroid. The pre-trained model SAM is the image segmentation base model SAM trained using the SA-1B dataset.

[0023] Step S1 includes the following sub-steps: S11: Acquire ultrasound image datasets and construct a pre-trained model SAM; S12: Input the training set from the dataset into the pre-trained model SAM for multiple rounds of training, and calculate the training gradients of parameters at different positions in the pre-trained parameters. ; S13: Based on training gradients Calculate the incompleteness of the corresponding positional parameters. The calculation formula is: ; In the formula, This represents the training gradient of the parameter at the nth position in the matrix. This indicates the degree of incompleteness of the parameter at position n; S14: Preset threshold Used to determine whether the parameters are complete, when Greater than the threshold It was judged to be incomplete at that time. Less than the threshold It is judged to be relatively complete at that time; S15: Count the number of incomplete parameters for each parameter matrix. If the number of incomplete parameters for a single parameter matrix exceeds a threshold... If the parameter matrix is ​​incomplete, then the corresponding parameter matrix is ​​considered to be incomplete; otherwise, it is considered to be relatively complete.

[0024] S2: The parameter matrix is ​​judged by a preset threshold, and then different adaptation schemes are adopted for the image encoder.

[0025] In step S2, the adaptation scheme includes: for the original parameter matrix corresponding to the incomplete parameter matrix, adding an Adapter branch in the adaptation scheme and setting the branch as trainable; for the original parameter matrix corresponding to the relatively complete parameter matrix, using singular value decomposition, setting the diagonal matrix as trainable, and freezing the remaining parameters.

[0026] Singular value decomposition includes the following steps: 1) Obtain a relatively complete parameter matrix from the pre-trained SAM model; 2) Use singular value decomposition to split the parameter matrix into a left singular vector matrix and a right singular vector matrix.

[0027] S3: Based on the adaptation scheme, add a detailed feature enhancement fusion module to build the HATSAM model; In step S3, the model HATSAM includes training using the DEFFM module, comprising the following sub-steps: 1) Obtain the output features of the second layer of the model image encoder. 5th layer output features and the features of the last layer output ; 2) Obtain the cue features output by the original pre-trained model SAM. and decoded pixel features ; 3) Use prompt features through the Guide module. right and Obtain by performing noise reduction operation and The calculation formula is: ; In the formula, This represents the output features of the 5th layer. Indicates a prompt feature, This represents the feature after denoising, and i represents the counting unit.

[0028] 4) Feature fusion is performed using the Fuse hybrid module, calculated as follows: ; In the formula, Indicates the characteristics after fusion. Representation of features Features after denoising Representation of features Features after denoising This represents the feature of the last layer's output. This represents the pixel features after decoding.

[0029] The Guide module processes the input and output features through depthwise separable convolution, cross attention, and deconvolution, including the following steps: 1) Input the second layer output features 5th layer output features Input into a depthwise separable convolution; 2) Combine the convolution results with cue features Implement cross-attention mechanisms; 3) Perform a deconvolution operation on the result of the previous step.

[0030] The DEFFM module also includes calculations using deconvolution. And calculate the fused feature output, the calculation formula is: ; ; In the formula, This represents the convolution operation. Indicates the deconvolution operation; , and is a constant coefficient, representing the proportion of the corresponding feature in the mixed features.

[0031] S4: Input the ultrasound image dataset into the HATSAM model for training and output the ultrasound image segmentation results.

[0032] In this implementation example, the TN3K, TG3K, BUSI, CAMUS, DDTI, UDIAT, and EchoCP datasets were selected. The experimental data used in the example are shown in Table 1. Table 1

[0033] like Figure 1 As shown, based on the pre-trained model SAM, downsampling of position encoding is added. The image encoder part of the original SAM adopts the parameter space completeness dynamic adaptation PSCDA scheme to reduce the domain gap between SAM and ultrasound image segmentation. The mask decoder of the original SAM adds a detail feature enhancement hybrid module DEFFM to supplement detail features for more refined segmentation results.

[0034] like Figure 2 As shown, the pre-trained model parameters are incomplete and are evaluated. For relatively complete parameter matrices, SVD decomposition is used; for incomplete parameter matrices, the Adapter fine-tuning method is employed. Furthermore, DEFFM is added to freeze components except those that have been added or modified, and training is performed only on the components that have undergone the corresponding modifications.

[0035] like Figure 3 As shown, the feature outputs of layers 2, 5, and the last layer of the input image encoder are used in DEFFM. The features from layers 2 and 5 will be combined with the prototype for cue feature generation to generate the guided feature. The final layer of features is used as initial input, along with the image features (pixel features) obtained from the mask decoder. Layer-by-layer fusion yields output features with fine-grained characteristics.

[0036] As shown in Tables 2-5, the experimental results of this invention are compared with mainstream models in the prior art in terms of quantitative analysis of visible data for specific tasks, generalization characteristics of invisible data for specific tasks, quantitative analysis of visible data in joint training, and quantitative analysis of generalization performance of invisible data in joint training, intuitively demonstrating the technical effects of this invention.

[0037] Table 2

[0038] As shown in Table 2, the test results of the models trained on the thyroid dataset TN3K, the breast dataset BUSI, and the heart dataset CAMUS show that the present invention outperforms most models on specific tasks. However, the present invention's method has a slight gap with the state-of-the-art in terms of the HD index on the CAMUS-MYO dataset and the Dice index on the CAMUS-LA dataset.

[0039] Table 3

[0040] As shown in Table 3, through a comparison of generalization performance on data without prior knowledge of specific tasks, the model of this invention has significant advantages over other models in terms of HD and Dice metrics on the thyroid dataset DDTI, the breast dataset UDIAT, and the heart dataset Echo-CP.

[0041] Table 4

[0042] As shown in Table 4, through joint training and quantitative comparison of visible data, the model of this invention is significantly better than other models on the TN3K and BUSI datasets, and also has a certain advantage on CAMUS-LV. It only has a slight gap with SAMUS on the Dice metric of CAMUS-MYO and the HD metric of CAMUS-LA.

[0043] Table 5

[0044] As shown in Table 5, through joint training without quantitative comparison of generalization performance on any data, the generalization performance of the model of this invention is better than that of the other models on all three types of datasets.

[0045] This invention provides a universal ultrasound image segmentation model based on dynamic adaptation for fine-tuning. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for ultrasound image segmentation based on dynamic adaptation for fine-tuning, characterized in that, Includes the following steps: S1: Acquire ultrasound images to form a dataset and evaluate the parameter matrix of the existing pre-trained model SAM; S2: The parameter matrix is ​​judged by a preset threshold, and then different adaptation schemes are adopted for the image encoder; S3: Based on the adaptation scheme, add a detailed feature enhancement fusion module to build the HATSAM model; S4: Input the ultrasound image dataset into the HATSAM model for training and output the ultrasound image segmentation results.

2. The ultrasound image segmentation method based on dynamic adaptation fine-tuning according to claim 1, characterized in that, In step S1, the ultrasound imaging data includes ultrasound images of multiple organs, covering multiple sites such as the heart, breast, and thyroid.

3. The ultrasound image segmentation method based on dynamic adaptation fine-tuning according to claim 1, characterized in that, In step S1, the pre-trained model SAM is the image segmentation base model SAM trained using the SA-1B dataset.

4. The ultrasound image segmentation method based on dynamic adaptation fine-tuning according to claim 1, characterized in that, Step S1 includes the following sub-steps: S11: Acquire ultrasound image datasets and construct a pre-trained model SAM; S12: Input the training set from the dataset into the pre-trained model SAM for multiple rounds of training, and calculate the training gradients of parameters at different positions in the pre-trained parameters. ; S13: Based on training gradients Calculate the incompleteness of the corresponding positional parameters. The calculation formula is: ； In the formula, This represents the training gradient of the parameter at the nth position in the matrix. This indicates the degree of incompleteness of the parameter at position n; S14: Preset threshold Used to determine whether the parameters are complete, when Greater than the threshold It was judged to be incomplete at that time. Less than the threshold It is judged to be relatively complete at that time; S15: Count the number of incomplete parameters for each parameter matrix. If the number of incomplete parameters for a single parameter matrix exceeds a threshold... If the parameter matrix is ​​incomplete, then the corresponding parameter matrix is ​​considered to be incomplete; otherwise, it is considered to be relatively complete.

5. The ultrasound image segmentation method based on dynamic adaptation fine-tuning according to claim 1, characterized in that, In step S2, the adaptation scheme includes: for the original parameter matrix corresponding to the incomplete parameter matrix, adding an Adapter branch in the adaptation scheme and setting the branch as trainable; for the original parameter matrix corresponding to the relatively complete parameter matrix, using singular value decomposition, setting the diagonal matrix as trainable, and freezing the remaining parameters.

6. The ultrasound image segmentation method based on dynamic adaptation fine-tuning according to claim 1, characterized in that, The singular value decomposition includes the following steps: 1) Obtain a relatively complete parameter matrix from the pre-trained SAM model; 2) Use singular value decomposition to split the parameter matrix into a left singular vector matrix and a right singular vector matrix.

7. The ultrasound image segmentation method based on dynamic adaptation fine-tuning according to claim 1, characterized in that, In step S3, the model HATSAM includes training using the DEFFM module, comprising the following sub-steps: 1) Obtain the output features of the second layer of the model image encoder. 5th layer output features and the features of the last layer output ; 2) Obtain the cue features output by the original pre-trained model SAM. and decoded pixel features ; 3) Use prompt features through the Guide module. right and Obtain by performing noise reduction operation and The calculation formula is: ； In the formula, This represents the output features of the 5th layer. Indicates a prompt feature, This represents the feature after denoising, where i represents the counting unit. 4) Feature fusion is performed using the Fuse hybrid module, calculated as follows: ； In the formula, Indicates the characteristics after fusion. Representation of features Features after denoising Representation of features Features after denoising This represents the feature of the last layer's output. This represents the pixel features after decoding.

8. The ultrasound image segmentation method based on dynamic adaptation fine-tuning according to claim 7, characterized in that, In step S3, the Guide module processes the input output features through depthwise separable convolution, cross attention, and deconvolution, including the following steps: 1) Input the second layer output features 5th layer output features Input into a depthwise separable convolution; 2) Combine the convolution results with cue features Implement cross-attention mechanisms; 3) Perform a deconvolution operation on the result of the previous step.

9. The ultrasound image segmentation method based on dynamic adaptation fine-tuning according to claim 7, characterized in that, The DEFFM module also includes calculations using deconvolution. And calculate the fused feature output, the formula is: ； ； In the formula, This represents the convolution operation. Indicates the deconvolution operation; , and is a constant coefficient, representing the proportion of the corresponding feature in the mixed features.

Images