Three-dimensional breast ultrasound image semi-automatic segmentation method based on fine-tuning large model

Abstract

The invention relates to the field of deep learning, in particular to a three-dimensional breast ultrasound image semi-automatic segmentation method based on a fine-tuning large model, which comprises the following steps of: preprocessing acquired to-be-segmented three-dimensional breast ultrasound image data, and inputting an SAM-Med3D model fine-tuned based on a low-rank fine-tuning strategy to realize semi-automatic segmentation of a breast tumor, and a customization module MLoRA is embedded in each 3D attention block of the 3D image encoder of the SAM-Med3D model. According to the method, the customization module based on the low-rank fine tuning strategy is applied to the SAM-Med3D model, and a doctor is simulated to click an ABUS image lesion area, so that semi-automatic segmentation of the breast tumor is realized, the segmentation capability of the SAM-Med3D model for the three-dimensional ABUS image is effectively improved, and the segmentation efficiency of the SAM-Med3D model is improved. Therefore, the problems of lack of ABUS image training data, high training cost and semantic confusion of an existing medical image segmentation large model are solved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a semi-automatic segmentation method of three-dimensional breast ultrasound images based on a fine-tuned large model. Background Art

[0002] Breast cancer is one of the most common cancers in women and has a profound impact on women's health. Early screening for breast cancer can provide patients with timely and effective intervention measures. Automated breast ultrasound (ABUS) is an advanced breast imaging technology. With its advantages of high resolution, repeatability of results and high objectivity, it has become a common technology for early screening of breast cancer. However, early screening of breast cancer by manual observation of three-dimensional ABUS images is time-consuming and laborious, and there is a risk of missed diagnosis. Therefore, developing an efficient and accurate three-dimensional automatic breast ultrasound image screening method is an urgent problem to be solved in order to improve the early diagnosis process of breast cancer.

[0003] In recent years, the rapid development of image processing technology and deep learning has driven the development of the field of medical image segmentation. The introduction of computer-aided diagnosis (CAD) systems aims to assist doctors in complex and large-scale automated segmentation and annotation of three-dimensional breast ABUS images. At the same time, CAD systems improve the speed, accuracy and interpretability of image segmentation, helping radiologists and clinicians analyze tumor signs and develop surgical plans. However, the premise of combining CAD systems is accurate automatic / semi-automatic ABUS lesion segmentation, and due to the fuzziness of lesion boundaries, class imbalance and imaging artifacts, traditional medical image segmentation methods cannot achieve ideal results. Therefore, in recent years, deep learning technology based on large medical image segmentation models has attracted widespread attention in medical image segmentation.

[0004] The medical image segmentation big model is a way to transfer the powerful generation and understanding capabilities of big models such as GPT-4 and SAM to the field of medical image segmentation to perform accurate and efficient image segmentation. However, due to the special properties of 3D ABUS images, the segmentation method based on big models has the following shortcomings:

[0005] (1) Lack of medical image training data: Since the semantic segmentation of 3D ABUS images requires a lot of expertise and time for manual annotation, the cost of obtaining medical datasets is high and the number of datasets is insufficient.

[0006] (2) High training cost: Large models suitable for specific 3D ABUS images require further downstream generalization, and full parameter fine-tuning will result in high training cost.

[0007] (3) Semantic confusion: Due to the complexity, stereoscopic nature, and imaging artifacts of 3D ABUS images, the risk of semantic confusion in large models will increase, and the accuracy of image tumor segmentation will be affected. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a semi-automatic segmentation method for three-dimensional breast ultrasound images based on a fine-tuned large model. By applying a customized module based on a low-rank fine-tuning strategy to the SAM-Med3D model and simulating a doctor clicking on the lesion area of ​​the ABUS image, semi-automatic segmentation of breast tumors is achieved, which effectively improves the segmentation capability of the SAM-Med3D model for three-dimensional ABUS images, thereby overcoming the problems of lack of ABUS image training data, high training cost and semantic confusion in the existing large model of medical image segmentation.

[0009] In order to achieve the above object, the technical solution specifically adopted by the present invention is as follows:

[0010] A semi-automatic segmentation method of three-dimensional breast ultrasound images based on a fine-tuned large model comprises the following steps:

[0011] Step 1, obtaining three-dimensional breast ultrasound image data to be segmented;

[0012] Step 2: After preprocessing the acquired three-dimensional breast ultrasound image data to be segmented, the SAM-Med3D model fine-tuned based on the low-rank fine-tuning strategy is input to realize the semi-automatic segmentation of breast tumors. The SAM-Med3D model fine-tuned based on the low-rank fine-tuning strategy includes a 3D image encoder, a 3D hint encoder, a 3D mask decoder and a custom module M. LoRA , the custom module M LoRA Embedded in each 3D attention block of the 3D image encoder.

[0013] Furthermore, the SAM-Med3D model fine-tuned based on the low-rank fine-tuning strategy is trained and constructed by the following steps:

[0014] S1. Acquire three-dimensional breast ultrasound image data;

[0015] S2, preprocess the data set and divide it into training set, validation set and test set;

[0016] S3. Build a training model based on the SAM-Med3D model structure and embed a customized module M based on a low-rank fine-tuning strategy in the 3D image encoder of the SAM-Med3D model. LoRA , so that the SAM-Med3D model can be fine-tuned for ABUS images;

[0017] S4. Freeze the parameters of the SAM-Med3D model, input the training set into the model for iterative training, perform model validation on the validation set, and save the customized module M after training is completed. LoRA The weight parameter of

[0018] S5. Use the trained customized module M LoRA The weight parameters are combined with the pre-trained weights of the SAM-Med3D model to test the model performance on the test set.

[0019] Furthermore, the specific process of step S2 is as follows:

[0020] S2.1. Process, crop, and fill the acquired three-dimensional breast ultrasound data one by one into a uniform size, and clean and screen the data as samples for subsequent processing;

[0021] S2.2. Use ITK-SNAP to visualize the data, and have experienced ultrasound physicians annotate the tumor boundaries of the dataset, and use the annotated files as the true labels of the tumor area.

[0022] Furthermore, the specific process of step S3 is as follows:

[0023] S3.1. The SAM-Med3D model structure is divided into three parts: 3D image encoder, 3D mask decoder, and 3D hint encoder. A custom module M is embedded in each 3D attention block of the 3D image encoder. LoRA ;

[0024] S3.2, through the 3D cue encoder part, the randomly generated simulated click coordinates (x, y, z) in the lesion are converted into sparse cue embedding p point ;

[0025] S3.3. Calculate the image segmentation information m of the three-dimensional breast ultrasound image through the 3D mask decoder part.

[0026] Furthermore, the specific process of step S4 is as follows:

[0027] S4.1. Read and freeze the pre-trained parameters of the SAM-Med3D model from the local file and LoRA Initialize;

[0028] S4.2. Obtain a 3D ABUS image dataset and select n groups of ABUS images as training samples s from it each time. t ;

[0029] S4.3, training sample s t Input the model for model training;

[0030] S4.4, the simulated click p point and the image embedding v for each sample img Input 3D mask decoder for iterative training, retaining M LoRA The weight parameter of .

[0031] Furthermore, the specific process of step S5 is as follows:

[0032] S5.1, construct the training model described in step S3;

[0033] S5.2. Load the trained M LoRA Weight parameters and SAM-Med3D pre-trained weight parameters;

[0034] S5.3. Input the test samples to test the model performance and evaluate the model segmentation results based on the true labels.

[0035] Furthermore, in step S3.1, the custom module M LoRA The n×r tensor W a and the r×n tensor W b Composition, r can be customized during the training process, n×n tensor first passes through W a After dimensionality reduction, W b Dimensional reduction is performed to reduce the number of model parameters and retain key parameter information.

[0036] Furthermore, the specific steps of step S4.3 are as follows:

[0037] S4.3.1. The training sample s t Each sample is cropped or filled to a size of (H×H×H) according to the lesion area;

[0038] S4.3.2. Calculate the 3D image convolution embedding v corresponding to each sample i and a learnable 3D absolute position code p img i , the required v i and p img i Contains the spatial details of the sample, thus obtaining the image embedding v of the ABUS image img ;

[0039] S4.3.3. Randomly generate simulated click prompts in the lesion area prompt = (x j ,y j ,z j ),i∈(0,n p ), which is input into the 3D prompt encoder for processing to obtain the absolute position code p of the simulated click imgj , the jth simulated click p point j and image embedding v img Input 3D mask decoder for iterative training, where the model sets n p Simulate clicks, and select the unmarked lesion area for each output;

[0040] S4.3.4, per k l Calculate the loss function L once per round CE and L Dice , respectively the cross entropy loss function and the Dice loss function;

[0041] S4.3.5, per k v Round selection validation data set samples s v , N (H×H×H) lesion blocks are cut out from the lesion area and compared with the real lesions, and the Dice value is calculated for verification.

[0042] The present invention has the following characteristics and beneficial effects:

[0043] 1) The present invention adds a customized LoRA module to each 3D attention block of the 3D image encoder of the SAM-Med3D model, which can compress features into a low-rank space and reproject them, thereby improving the model's ability to segment breast tumors for three-dimensional ABUS images. The average Dice similarity coefficient on the TDSC-ABUS 2023 challenge public dataset reaches 0.79, exceeding the highest score in its segmented field rankings, and the average Dice similarity coefficient on the private breast cancer dataset is 0.76. At the same time, the large model hyperparameters are frozen and only a small part of the parameters of the LoRA module are updated. The storage of the relevant parameters occupies only 29.56MB, and the parameter amount is 2.6% of the SAM-Med3D model parameter amount. Therefore, the present invention reduces the computational overhead of model training while maintaining the segmentation performance, and has excellent results for the segmentation of lesions of different sizes and categories.

[0044] 2) The present invention uses a low-rank fine-tuning strategy algorithm to solve the downstream migration of the large medical image segmentation model to the three-dimensional breast ultrasound image segmentation problem, which greatly reduces the cost of applying the computer-aided diagnosis system to the early diagnosis of breast cancer.

[0045] 3) The present invention proposes a semi-automatic segmentation simulation method, which effectively enriches the details of the model segmentation by simulating the clicking operation of a radiologist inside the tumor, and improves the segmentation performance of the large segmentation model on three-dimensional breast ultrasound images. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0047] Figure 1 The invention discloses a model training framework for a semi-automatic segmentation method of three-dimensional breast ultrasound images based on a fine-tuned large model. DETAILED DESCRIPTION

[0048] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0049] The present invention provides a semi-automatic segmentation method for three-dimensional breast ultrasound images based on a fine-tuned large model. The segmentation method is implemented based on the SAM-Med3D model fine-tuned by a low-rank fine-tuning strategy. Figure 1 As shown in Figure 2, the SAM-Med3D model fine-tuned by the low-rank fine-tuning strategy mainly includes a 3D image encoder, a 3D hint encoder, a 3D mask decoder, and a custom module M. LoRA Four parts. The custom module M LoRA In the 3D image encoder part embedded in the SAM-Med3D model, each layer of 3D self-attention blocks in the main network of the 3D image encoder is module-added to learn the image information in the ABUS image. LoRA The input is the feature information v processed by 3D convolution and 3D self-attention module img i and a learnable 3D absolute position encoding p img , the characteristic matrix T of the medical information of the three-dimensional breast ultrasound image dataset in the low-rank space is obtained by calculation i Finally, the output feature matrix T i The feature map E after the output channel of the 3D self-attention block is aligned. In this embodiment, a 3D convolution kernel with a kernel size of (16, 16, 16) is used for feature extraction, the size of the low-rank space is set to 4, and 8 rounds of 3D attention modules A are set. n and M LoRA , perform 8 iterations of learning, and output the final 3D image embedding v containing the spatial information of the 3D ABUS image img .

[0050] The customized module M LoRA Each LoRA block in is attached as a trainable bypass to each 3D attention block of the frozen “3D image encoder”. By freezing the parameters of the 3D image encoder, only the custom module M is updated. LoRAThis not only reduces the computational overhead, but also reduces the deployment and storage complexity of the fine-tuned model while maintaining the segmentation performance.

[0051] The constructed medical image segmentation model is used in two different steps of image segmentation: training stage and inference stage.

[0052] A further configuration of this embodiment is a model training method, which mainly includes the following steps:

[0053] Step 1: Obtain three-dimensional ABUS image data through the Invenia automatic breast ultrasound diagnostic system, and clean and normalize the obtained data as samples for subsequent steps, as follows:

[0054] Step 1.1: An experienced radiologist annotates the 3D lesion area of ​​the original ultrasound data, and puts the original ultrasound file and the corresponding annotation file into two directories respectively, and uses the annotation file as the real label of the tumor area;

[0055] Step 1.2: Use the open source SimpleITK library tool to convert all the original files and ultrasound files from DICOM format to NIFTI format;

[0056] Step 1.3: Crop and fill all images and their annotation files to a uniform size of 840×440×330;

[0057] Step 1.4: Divide all acquired data into training set, validation set and test set in a ratio of 7:1:2;

[0058] Step 2: Download the pre-trained weight parameters of SAM-Med3D, freeze the pre-trained weight parameters of 3D image encoder S, 3D hint encoder T, and 3D mask decoder D, and customize the module M. LoRA To initialize, the specific steps are as follows:

[0059] Step 2.1, use the pre-trained weight parameters of SAM-Med3D to participate in the model training, but freeze the parameters of the 3D image encoder S, 3D hint encoder T, and 3D mask decoder D during the training process and do not participate in the update;

[0060] Step 2.2: Build custom module M LoRA It is attached as a bypass to each 3D attention module, specifically to the tensors Q and K of the 3D attention module. For the input N×N tensor, the formula processed by the LoRA module is:

[0061] X n×n ·WA n×r +WBr×n ·X n×n =X output

[0062] Among them, X n×n is the 3D absolute position code p img tensor, r is a custom low-rank parameter, which is set to 4 in this embodiment. WA and WB are the key tensors for vital sign extraction in the LoRA module, where WB is initialized to a zero tensor, and WA uses the initialization method proposed by He Kaiming to generate a uniformly distributed tensor. output It is M LoRA The generated feature tensor;

[0063] Step 3: Set the training hyperparameters, where Batch Size is set to 2, the initial learning rate is 0.0001, and the Loss function uses a combination of Cross-entropy loss and Dice loss. Train for 200 epochs, set the number of simulated clicks for each epoch to 11, verify the model performance once every 10 epochs, and the number of simulated clicks for the verification process is 5. Then, the SAM-Med3D-ABUS network model is obtained.

[0064] Step 4: Train the sample s t Input the SAM-Med3D-ABUS network model for model training, as follows:

[0065] Step 4.1: Train the sample s t For each sample, a block of size (128×128×128) was cut from the center of the lesion area;

[0066] Step 4.2: Randomly generate simulated click prompts in the block

[0067] prompt=(x_j,y_j,z_j),i∈(0,11]

[0068] Input to the 3D prompt encoder T for processing to obtain the absolute position code p of the simulated click point j , the jth simulated click p point j Input the 3D mask decoder D, and the output image prediction mask is re-input into the 3D prompt encoder for iterative training. The model is set up with 11 simulated clicks in total, and each output selects the unmarked lesion area;

[0069] Step 4.3: Calculate the loss and Dice value once in each round and record them;

[0070] Step 5: Perform model performance verification every 10 epochs. The verification is performed on the validation set data using the weight parameters obtained from the current training. The specific process is as follows:

[0071] Step 5.1, cutting out a number of adjacent blocks (128×128×128), which should cover the entire lesion area;

[0072] Step 5.2: Input each block into the model separately, obtain the segmentation results of all blocks, then splice all blocks, calculate the Dice value with the real label, and select the highest Dice value in the 5 simulated clicks as the verification data of the current sample;

[0073] Step 5.3: Calculate the average Dice data of all validation set samples as the final result of this validation. If the Dice value is higher than the results of the previous validations and exceeds 0.6, save the currently trained custom module M. LoRA Weight parameters and continue training for the next Epoch.

[0074] Step 6: Read the custom module M saved during training LoRA The weight parameters are used to perform inference tests on the model performance. The specific logic is the same as the verification process. The data uses the data of the test set to test the inference results to verify the final validity of the weight parameters and determine the weight parameter file that should be used in actual applications.

[0075] When the constructed medical image segmentation model is used for image segmentation, the following steps are included:

[0076] Step 1, obtaining three-dimensional breast ultrasound image data to be segmented;

[0077] Step 2: After preprocessing the acquired three-dimensional breast ultrasound image data to be segmented, the SAM-Med3D model fine-tuned based on the low-rank fine-tuning strategy is input to realize the semi-automatic segmentation of breast tumors. During preprocessing, the open source SimpleITK library tool is used to convert the format of the three-dimensional breast ultrasound image data to be segmented, from DICOM format to NIFTI format, and then cropped and filled to process it into a unified size of 840×440×330.

[0078] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A semi-automatic segmentation method for three-dimensional breast ultrasound images based on a fine-tuned large model, characterized in that: The steps include: Step 1, obtaining three-dimensional breast ultrasound image data to be segmented; Step 2: After preprocessing the acquired three-dimensional breast ultrasound image data to be segmented, the SAM-Med3D model fine-tuned based on the low-rank fine-tuning strategy is input to realize the semi-automatic segmentation of breast tumors. The SAM-Med3D model fine-tuned based on the low-rank fine-tuning strategy includes a 3D image encoder, a 3D hint encoder, a 3D mask decoder and a custom module M. LoRA , the custom module M LoRA Embedded in each 3D attention block of the 3D image encoder.

2. The method for semi-automatic segmentation of three-dimensional breast ultrasound images based on a fine-tuned large model as claimed in claim 1, characterized in that: The SAM-Med3D model fine-tuned based on the low-rank fine-tuning strategy is trained and constructed by the following steps: S1. Acquire three-dimensional breast ultrasound image data; S2, preprocess the data set and divide it into training set, validation set and test set; S3. Build a training model based on the SAM-Med3D model structure and embed a customized module M based on a low-rank fine-tuning strategy in the 3D image encoder of the SAM-Med3D model. LoRA , so that the SAM-Med3D model can be fine-tuned for ABUS images; S4. Freeze the parameters of the SAM-Med3D model, input the training set into the model for iterative training, perform model validation on the validation set, and save the customized module M after training is completed. LoRA The weight parameter of S5. Use the trained customized module M LoRA The weight parameters are combined with the pre-trained weights of the SAM-Med3D model to test the model performance on the test set.

3. The method for semi-automatic segmentation of three-dimensional breast ultrasound images based on a fine-tuned large model as claimed in claim 2, characterized in that: The specific process of step S2 is as follows: S2.

1. Process, crop, and fill the acquired three-dimensional breast ultrasound data one by one into a uniform size, and clean and screen the data as samples for subsequent processing; S2.

2. Use ITK-SNAP to visualize the data, annotate the tumor boundaries of the dataset, and use the annotated files as the true labels of the tumor areas.

4. The method for semi-automatic segmentation of three-dimensional breast ultrasound images based on a fine-tuned large model as claimed in claim 2, characterized in that: The specific process of step S3 is as follows: S3.

1. The SAM-Med3D model structure is divided into three parts: 3D image encoder, 3D mask decoder, and 3D hint encoder. A custom module M is embedded in each 3D attention block of the 3D image encoder. LoRA ; S3.2, through the 3D cue encoder part, the randomly generated simulated click coordinates (x, y, z) in the lesion are converted into sparse cue embedding p point ; S3.

3. Calculate the image segmentation information m of the three-dimensional breast ultrasound image through the 3D mask decoder part.

5. The method for semi-automatic segmentation of three-dimensional breast ultrasound images based on a fine-tuned large model as claimed in claim 2, characterized in that: The specific process of step S4 is as follows: S4.

1. Read and freeze the pre-trained parameters of the SAM-Med3D model from the local file and LoRA Initialize; S4.

2. Obtain a 3D ABUS image dataset and select n groups of ABUS images as training samples s from it each time. t ; S4.3, training sample s t Input the model for model training; S4.4, the simulated click p point and the image embedding v for each sample img Input 3D mask decoder for iterative training, retaining M LoRA The weight parameter of .

6. The method for semi-automatic segmentation of three-dimensional breast ultrasound images based on a fine-tuned large model as claimed in claim 2, characterized in that: The specific process of step S5 is as follows: S5.1, construct the training model described in step S3; S5.

2. Load the trained M LoRA Weight parameters and SAM-Med3D pre-trained weight parameters; S5.

3. Input the test samples to test the model performance and evaluate the model segmentation results based on the true labels.

7. The method for semi-automatic segmentation of three-dimensional breast ultrasound images based on a fine-tuned large model as claimed in claim 4, characterized in that: In step S3.1, the customized module M LoRA The n×r tensor W a and the r×n tensor W b Composition, r can be customized during the training process, n×n tensor first passes through W a After dimensionality reduction, W b Dimensional reduction is performed to reduce the number of model parameters and retain key parameter information.

8. The method for semi-automatic segmentation of three-dimensional breast ultrasound images based on a fine-tuned large model as claimed in claim 5, characterized in that: The specific steps of step S4.3 are as follows: S4.3.

1. The training sample s t Each sample is cropped or filled to a size of (H×H×H) according to the lesion area; S4.3.

2. Calculate the 3D image convolution embedding v corresponding to each sample i and a learnable 3D absolute position code p img i , the required v i and p img i The spatial details of the sample are contained, and the image embedding v of the ABUS image is obtained img ; S4.3.

3. Randomly generate simulated click prompts in the lesion area prompt = (x j ,y j ,z j ),i∈(0,n p ), which is input into the 3D prompt encoder for processing to obtain the absolute position code p of the simulated click img j , the jth simulated click p point j and image embedding v img Input 3D mask decoder for iterative training, where the model sets n p Simulate clicks, and select the unmarked lesion area for each output; S4.3.4, per k l Calculate the loss function L once per round CE and L Dice , respectively the cross entropy loss function and the Dice loss function; S4.3.5, per k v Round selection validation data set samples s v , N (H×H×H) lesion blocks are cut out from the lesion area and compared with the real lesions, and the Dice value is calculated for verification.

Images