A masking pre-training method for ultrasound image segmentation

Abstract

The present application belongs to the technical field of medical image processing, and specifically provides a masking pre-training method for ultrasound image segmentation, which is used to realize automatic segmentation of ultrasound images. The present application pre-trains an ultrasound image segmentation model by randomly masking part of the pixels in the input ultrasound image to predict unknown content, so as to adapt the model to the current fitting scene before formal training, and the parameter weights in the masking pre-training are used as the initial state of the subsequent segmentation task. Based on this, a large number of unlabeled images can greatly enhance the feature extraction capability, and in the case that the labeled data is very limited, the segmentation accuracy is greatly improved. In summary, the present application can be used to greatly enhance the feature extraction capability by using limited labeled images in the case of a large number of irrelevant unlabeled images, and realize accurate segmentation under a small sample data set.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image segmentation and deep learning, and specifically provides a masking pre-training method for ultrasound image segmentation. BACKGROUND

[0002] In modern medical imaging technology, ultrasound imaging technology is widely used worldwide due to its advantages of no radiation, low cost, high portability, and the ability to provide real-time images. As a key diagnostic tool, ultrasound imaging technology is particularly suitable for maternal care, cardiovascular disease assessment, and soft tissue and internal organ examination, providing valuable information for doctors and helping doctors quickly discover diseases for treatment. However, despite the many advantages of ultrasound imaging, its application is limited by image quality factors. Ultrasound images often exhibit low resolution and high noise levels due to technical limitations, making it difficult to provide clear tissue boundary information and detailed structural information, and this problem is particularly pronounced in situations that require high-precision image analysis.

[0003] In order to overcome the above limitations and improve the application value of ultrasound imaging in clinical diagnosis, computer-aided ultrasound image segmentation technology has become a research hotspot in recent years. By automatically dividing images into clinically meaningful regions using advanced algorithms, doctors can quickly and accurately diagnose. Among them, the U-Net model has achieved remarkable success in automatic medical image segmentation due to its unique structure and capabilities. The U-Net model uses its deep network structure to capture image features while preserving rich contextual information through specially designed skip connections, making it perform well in various medical image segmentation tasks. However, the model's performance is limited when faced with low spatial resolution and contrast, artifacts and noise, etc. of ultrasound images, especially in fine structure recognition and boundary clarity.

[0004] In addition, the development of ultrasound image segmentation technology faces an important bottleneck: the lack of high-quality, professionally annotated datasets, which is particularly prominent in specific medical applications such as the diagnosis of rare diseases. High-quality ultrasound image annotation not only requires a lot of time and professional knowledge, but also is limited by medical data privacy policies, further exacerbating the difficulty of obtaining such data. Since deep learning models often rely on large amounts of training data to learn complex image features and patterns, data shortages further limit the performance of deep learning models. Therefore, developing deep learning models that can effectively work with limited labeled data is a key requirement in the field of ultrasound image segmentation, which not only improves the model's generalization ability, but also reduces the model's dependence on large-scale labeled datasets, providing important support for improving global medical service quality and sustainable development. SUMMARY

[0005] The present application aims at the defects of the prior art, and provides a masking pre-training method for ultrasonic image segmentation, which can greatly enhance the feature extraction capability by using limited labeled images in the case of a large number of irrelevant unlabeled images, and realize accurate segmentation under a small sample data set, and is especially suitable for the ultrasonic image segmentation scene in intelligent medical treatment in the case of limited labeled images and a large number of unlabeled images.

[0006] To achieve the above object, the technical scheme adopted by the present application is:

[0007] A masking pre-training method for ultrasonic image segmentation, comprising the following steps:

[0008] Step 1, data preprocessing is performed on the ultrasonic image to obtain a training set, wherein the training sample is represented as (x, y), x represents the sample image, and y represents the sample label;

[0009] Step 2, the training set obtained in step 1 is used to perform masking pre-training on the ultrasonic image segmentation model, and the model pre-training parameters are saved;

[0010] Step 3, the model pre-training parameters obtained in step 2 are used as the model initialization parameters, and the training set obtained in step 1 is used to perform segmentation task training on the ultrasonic image segmentation model, to obtain a trained ultrasonic image segmentation model;

[0011] Step 4, the ultrasonic image to be processed is input into the trained ultrasonic image segmentation model obtained in step 3 after data preprocessing, and the ultrasonic image segmentation result is output by the ultrasonic image segmentation model.

[0012] Further, in step 1, the specific process of data preprocessing is as follows:

[0013] Step 1.1, the ultrasonic image is subjected to grayscale processing to convert the ultrasonic image into a single-channel grayscale image;

[0014] Step 1.2, the ultrasonic image processed in step 1.1 is subjected to scaling operation to unify the resolution of the ultrasonic image to a preset size;

[0015] Step 1.3, the ultrasonic image processed in step 1.2 is subjected to storage format conversion to obtain a binary file and store it.

[0016] Further, in step 2, the specific process of masking pre-training is as follows:

[0017] Step 2.1, a mask matrix M is generated for the sample image;

[0018] Step 2.2, generate a (1-M) matrix according to the mask matrix M, and perform a dot product operation between the (1-M) matrix and the sample image x to obtain an input image x1 of the ultrasound image segmentation model; perform a dot product operation between the mask matrix M and the sample image x to obtain a masked pre-training label y1;

[0019] Step 2.3, perform a dot product operation between the output image Y of the ultrasound image segmentation model and the mask matrix M to obtain an output image Y1; calculate the mean square error (MSE) between the output image Y1 and the masked pre-training label y1 as a loss, and train the model using an Adamax optimizer, where the parameters are set as learning rate lr = 0.001, β1 = 0.9, and β2 = 0.999 to complete the masked pre-training.

[0020] Further, in step 2.1, the mask matrix M generation process is as follows:

[0021] Step 2.1.1, perform zero padding and expansion on the sample image x to obtain an expanded image with an expansion size of p pixels;

[0022] Step 2.1.2, randomly select a pixel point of the expanded image as a starting point to generate a mask block with a side length of p.

[0023] Step 2.1.3, construct a mask matrix with the same size as the sample image x according to the mask block, and the mask matrix is a 0-1 matrix, where the masked part is 1 and the non-masked part is 0.

[0024] Further, in step 2.1.2, the probability q of each pixel being masked is:

[0025]

[0026] where w and h represent the length and width of the image.

[0027] Further, in step 2.1.2, the side length p of the generated mask block is in the range of 10 to 100 pixels, and the masking ratio is 0.2 or 0.6.

[0028] Further, in step 3, the specific process of the segmentation task training is as follows:

[0029] Step 3.1, use the model pre-training parameters obtained in step 2 as the model initialization parameters for the segmentation task training process;

[0030] Step 3.2, set the loss function as Tversky loss, and complete the segmentation task training using an Adamax optimizer.

[0031] Further, in step 3.2, the Tversky loss is expressed as:

[0032]

[0033] wherein y is a sample label, Y is an output image of the ultrasound image segmentation model, <,> represents a matrix dot product operation, and β is a weight coefficient.

[0034] Based on the above technical solution, the beneficial effects of the present application are as follows:

[0035] The present application provides a masking pre-training method for ultrasound image segmentation. In the case of limited labeled images and a large number of related unlabeled images in the domain, the ultrasound image segmentation model is pre-trained by randomly masking part of the pixels in the input ultrasound image to predict unknown content, so as to adapt the model to the current fitting scene before formal training, and the parameter weight in the masking pre-training is used as the initial state of the subsequent segmentation task. Based on this, a large number of unlabeled images can greatly enhance the feature extraction capability, and in the case of very limited labeled data, the segmentation accuracy is greatly improved. In particular, the trained ultrasound image segmentation model can also be used to segment the remaining unlabeled images to help the annotator determine the accurate region boundary. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 It is a flowchart of the masking pre-training method for ultrasound image segmentation in the present application.

[0037] Figure 2 It is a segmentation performance result graph of the ultrasound image segmentation model trained by the embodiment in the present application under various mask ratios. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and beneficial effects of the present application clearer, further detailed description of the present application will be given below in combination with the drawings and embodiments.

[0039] The present embodiment provides a masking pre-training method for ultrasound image segmentation, as shown in Figure 1 The masking pre-training and small sample pre-training method are introduced, which specifically includes the following steps:

[0040] Step 1, data preprocessing is performed on all ultrasound images to obtain the training set and test set data required in the subsequent training and test stages; wherein the training sample is represented as (x, y), x represents a sample image, and y represents a sample label;

[0041] The specific process is as follows:

[0042] All ultrasound images are divided into training set and test set according to a certain proportion, and then they are preprocessed in the same way;

[0043] The following preprocessing operations are performed on all images of the training set and test set:

[0044] Step 1.1, gray processing is performed on the original ultrasound image, and the ultrasound image is converted from the original RGB three-channel color image to a single-channel gray image;

[0045] Step 1.2, scaling operation is performed on the ultrasound image processed in step 1.1, and the resolution of the ultrasound image is unified to 256x256 format;

[0046] Step 1.3, storage format conversion operation is performed on the ultrasound image processed in step 1.2, and the ultrasound image file is converted into a ".npz" format binary file to speed up data reading speed;

[0047] Step 2, using the training set obtained in step 1, the ultrasound image segmentation model is masked pre-trained, and the model pre-training parameters are saved; the specific process of the masked pre-training is as follows:

[0048] Step 2.1, a mask matrix M is generated for each sample image, and the generation process is as follows:

[0049] Step 2.1.1, the sample image x is zero-padded and expanded upward and leftward, and the expansion size is p pixels, to obtain an expanded image, so as to ensure that each pixel has the same probability of being masked;

[0050] Step 2.1.2, randomly select a pixel point of the expanded image as a starting point, and generate a mask block with p as the side length, and the probability q of each pixel being masked is:

[0051]

[0052] Where w and h represent the length and width of the image, and p represents the side length of the square mask block;

[0053] Step 2.1.3, a mask matrix with the same size as the sample image x is constructed according to the mask block, and the mask matrix adopts a 0-1 matrix, wherein the masked part is 1 and the non-masked part is 0;

[0054] Step 2.2, generate (1-M) matrix according to the mask matrix M, and use (1-M) matrix and sample image x to do dot product operation, obtain the image retaining non-masked pixel points as the input image x1 of the ultrasound image segmentation model; use the mask matrix M and the sample image x to do dot product operation, obtain the image of the masked pixel points as the masked pre-training label y1;

[0055] Step 2.3, the output image Y of the ultrasound image segmentation model is dot multiplied with the mask matrix M, so that the output image only contains the pixel points of the masked part, and an output image Y1 is obtained; the mean square error (MSE) between the output image Y1 and the masked pre-training label y1 is calculated as the loss, and the model is trained using the Adamax optimizer, wherein the parameters are set as learning rate lr = 0.001, β1 = 0.9, and β2 = 0.999 to complete the masking pre-training;

[0056] Step 3, using the model pre-training parameters obtained in step 2 as the model initialization parameters, the ultrasound image segmentation model is trained for a segmentation task using the training set and the validation set obtained in step 1, and the final model parameters are saved; the specific process of the segmentation task training is as follows:

[0057] Step 3.1, using the model pre-training parameters obtained in step 2 as the model initialization parameters for the segmentation task training process;

[0058] Step 3.2, setting the loss function as Tversky loss, and using the Adamax optimizer to complete the segmentation task training; the Tversky loss is represented as:

[0059]

[0060] wherein y is a sample label, Y is an output image of the ultrasound image segmentation model, <,> represents a matrix dot product operation, and β is a weight coefficient;

[0061] Step 4, using the test set obtained in step 1 to test the performance of the ultrasound image segmentation model trained in step 3, and using Dice as the evaluation index.

[0062] The beneficial effects of the present application are described in detail below in combination with simulation tests. The simulation tests are performed on a central processing unit of Intel(R) Xeon(R) Silver 4210R CPU@2.40GHz, a graphics card of NVIDIA GeForce RTX 3090 24GB display memory, an Ubuntu 20.04.1LTS operating system, and TensorFlow as a back-end computing support. A TN3k dataset is used, which is proposed by Gong et al. in the literature "Gong H, Chen G, Wang R, et al. Multi-task learning for thyroid nodule segmentation with thyroid region prior [C] / / 2021IEEE 18th international symposium on biomedical imaging (ISBI). IEEE, 2021: 257-261." The dataset contains a total of 3493 thyroid nodule ultrasound images and corresponding segmentation labels from 2421 patients.

[0063] In this embodiment, MS-UNet is selected as the basic model, and the size (pixels) of the mask block is set to 10x10, 20x20, 40x40, 80x80, and 100x100, respectively, and a mixed mask of 40x40 and 100x100. In addition, the mask ratio refers to the ratio of the masked area to the entire image, which is set to 0.2 and 0.6, respectively. In each case, the image segmentation performance is shown in Table 1:

[0064] Table 1

[0065]

[0066] As can be seen from Table 1, the optimal size of the mask block in this embodiment is 40x40. On this basis, the size of the mask block is fixed at 40x40, and the segmentation performance results under each mask ratio are shown in Table 2: Figure 2As shown, it can be seen that good segmentation performance can be obtained when the masking ratio changes between 0.2 and 0.6, but the performance sharply decreases when the masking ratio is greater than 0.7; unlike the masking pre-training method proposed in the documents "Masked autoencoders are scalable vision learners, in Proceedings of CVPR, 2022, pp. 16000-16009" and "Simmim: A simple framework for masked image modeling, in Proceedings of CVPR, 2022, pp. 9653-9663", and the simMIM method, the latter proves that a high masking ratio (such as 0.75) can produce better performance in subsequent supervised tasks of natural image processing; due to the inherent differences between natural images and ultrasound images, the small amount of supervised training data and the single-channel and low-precision characteristics, it is difficult to infer the masked part from the known area, therefore, the relatively low masking rate is more suitable for the masking pre-training of ultrasound images.

[0067] Further, the segmentation performance of the fetal chest space-occupying lesion ultrasound image small sample set based on the U-Net, U-Net-R, U-Net-RS and MS-UNet models is shown in Table 2, wherein the "(P)" mark is used to mark the masking pre-training method of the present application, and the corresponding marked data size is 10, 20 and 50 images; it can be seen that the masking pre-training method proposed in the present application greatly improves the segmentation accuracy of ultrasound images, and the segmentation accuracy achieved under the condition that only 10 ultrasound images are used for training is close to the segmentation accuracy achieved by using five times the number of images; it can be seen that the masking pre-training method proposed in the present application can improve the accuracy of the segmentation model on a small number of annotated samples, thereby being used to segment the remaining unlabeled images to help the annotator determine the accurate ROI boundary.

[0068] Table 2

[0069]

[0070] In addition, the congenital diaphragmatic hernia ultrasound images are segmented, the congenital diaphragmatic hernia ultrasound image small sample set includes 94 ultrasound images belonging to 17 patients with congenital diaphragmatic hernia, and the training and test ratio is 2:8, 4:6 and 6:4, and all images belonging to the same case are used for training or testing, and the results are shown in Table 3; it can be seen that the masking pre-training method proposed in the present application improves the segmentation accuracy of all models on the congenital diaphragmatic hernia images, further verifying that the masking pre-training method proposed in the present application is very suitable for the segmentation of small sample ultrasound images of rare diseases.

[0071] Table 3

[0072]

[0073] The above description is merely that of a specific implementation of the application, any feature in the present description is capable of being, unless specifically stated, replaced by other equivalent or similar features, and each published feature, or step in a method or process, can be combined with any or all other features or steps, unless the features and / or steps are mutually exclusive.

Claims

1. A masking pre-training method for ultrasound image segmentation, characterized in that, Includes the following steps: Step 1: Perform data preprocessing on the ultrasound images to construct a training set, where the training samples are represented as ( x , y ), x Represents a sample image. y Indicates sample label; Step 2: Use the training set obtained in Step 1 to perform masking pre-training on the ultrasound image segmentation model and save the model pre-training parameters; the specific process of masking pre-training is as follows: Step 2.1: Generate a mask matrix M for the sample image; Step 2.2: Generate a (1-M) matrix based on the mask matrix M, and then use the (1-M) matrix with the sample image. x Perform a dot product operation to obtain the input image for the ultrasound image segmentation model. x 1; Use the mask matrix M and the sample image x Perform a dot product operation to obtain the masking pre-training labels. y 1; Step 2.3: Output image of the ultrasound image segmentation model Y Perform a dot product operation with the mask matrix M to obtain the output image. Y 1; Calculate the output image Y 1. Pre-training labels with occlusion y The mean squared error (MSE) of 1 was used as the loss, and the model was trained using the Adamax optimizer, with the parameters set as learning rate lr=0.001, β1=0.9, β2=0.999 to complete the occlusion pre-training. Step 3: Using the model pre-training parameters obtained in Step 2 as model initialization parameters, train the ultrasound image segmentation model on the segmentation task using the training set obtained in Step 1 to obtain the trained ultrasound image segmentation model. Step 4: After data preprocessing, the ultrasound image to be processed is input into the ultrasound image segmentation model trained in Step 3, and the ultrasound image segmentation model outputs the ultrasound image segmentation result.

2. The masking pre-training method for ultrasound image segmentation according to claim 1, characterized in that, In step 1, the specific process of data preprocessing is as follows: Step 1.1: Perform grayscale processing on the ultrasound image to convert it into a single-channel grayscale image; Step 1.2: Scaling the ultrasound image processed in Step 1.1 to unify the resolution of the ultrasound image to the preset size; Step 1.3: Convert the storage format of the ultrasound image processed in Step 1.2 to obtain a binary file and store it.

3. The masking pre-training method for ultrasound image segmentation according to claim 1, characterized in that, In step 2.1, the mask matrix M is generated as follows: Step 2.1.1: Transfer the sample image x Expand upwards and to the left with zero padding, the expansion size is... p 1 pixel, to obtain an expanded image; Step 2.1.2: Randomly select pixels in the expanded image as the starting point, so as to... p Generate a mask block for the side length; Step 2.1.3: Construct a sample image based on the mask blocks. x Mask matrices of the same size are used, and the mask matrices are 0-1 matrices, where the masked part is 1 and the unmasked part is 0.

4. The masking pre-training method for ultrasound image segmentation according to claim 3, characterized in that, In step 2.1.2, the probability of each pixel being masked. q for: ， in, w and h This indicates the length and width of the image.

5. The masking pre-training method for ultrasound image segmentation according to claim 3, characterized in that, In step 2.1.2, the side length of the generated mask block is... p The value ranges from 10 pixels to 100 pixels, and the occlusion ratio is 0.2 or 0.

6.

6. The masking pre-training method for ultrasound image segmentation according to claim 1, characterized in that, In step 3, the specific process of training the segmentation task is as follows: Step 3.1: Use the model pre-training parameters obtained in Step 2 as the model initialization parameters for the segmentation task training process; Step 3.2: Set the loss function to Tversky loss and use the Adamax optimizer to complete the segmentation task training.

7. The masking pre-training method for ultrasound image segmentation according to claim 6, characterized in that, In step 3.2, the Tversky loss is expressed as: ， in, y For sample labels, Y This is the output image of the ultrasound image segmentation model. This represents the matrix dot product operation, where β is the weighting coefficient.

Images