Small object segmentation system for lymph node nodules in ultrasound images based on attention self-distillation

Abstract

The present invention discloses a lymph node nodule ultrasound image small target segmentation system based on attention self-distillation, which relates to the field of image segmentation technology. A segmentation model including a U-net network, a single-layer convolutional neural network and a self-distillation supervision module is constructed, and attention-guided calculation is performed on nodule features encoded by the model and nodule areas in the original image through the self-distillation supervision module, and feature information of large nodules is used to compensate for information loss caused by reduced feature map resolution. Due to the similarity of nodule area imaging, nodule image mapping can guide the feature alignment relationship of nodule feature mapping, quantify the relationship between each input image and the corresponding feature map, and distill the feature knowledge learned from the large nodule area to the small nodule area, thereby improving the segmentation performance of small nodules in lymph node nodule segmentation, improving the robustness and generalization of the model for nodule detection of different sizes, and better adapting to data sets and actual clinical scenarios in different fields.

Description

Technical Field

[0001] The present invention relates to the field of segmentation technology, and in particular to a lymph node nodule ultrasound image small target segmentation system based on attention self-distillation. Background Art

[0002] With the high pressure of work and improper diet in modern times, the incidence of lymph node-related diseases has increased year by year. Among them, the formation of lymph node nodules is one of the most common clinical symptoms of lymph node diseases. Accurate detection of lymph node nodules is helpful for subsequent doctors to judge the benign or malignant nature of nodules and formulate treatment plans, which plays a key role in the physical and mental health of patients. With the maturity of ultrasound imaging technology and the development of artificial intelligence technology, AI-enabled lymph node nodule detection technology has become a hot research trend. Using deep neural networks to autonomously learn the potential pattern features of the connotation of lymph node ultrasound images and then complete the segmentation and detection of nodule areas has gradually become an important solution for lymph node nodule identification.

[0003] Although deep learning methods have been widely used in the segmentation of lymph node nodules, the model's detection rate of small lymph node nodules limits the overall segmentation performance of the model. The inability to accurately identify all small nodules is a key challenge facing the current lymph node nodule segmentation model. This is largely attributed to the fact that the model stacks a large number of convolutional neural network layers and pooling layers in the process of extracting features. Although these operations enhance the semantic information of the features, due to the reduction in resolution, the detailed information of the local area of ​​the image also becomes coarse, which ultimately makes it difficult to successfully identify the features of small nodules and complete accurate segmentation. However, for lymph node nodules, the detection of small nodules is more important. For professional doctors, it is more difficult to identify small lymph nodes. Therefore, how to accurately identify small lymph node nodules has important medical application significance.

[0004] Related deep learning-based technologies have begun to attempt to solve the problem of lymph node nodule segmentation: a mainstream idea is to enlarge the input image to improve the resolution of small nodules, and then generate a high-resolution feature map, but the increase in data size and feature dimension will lead to a large amount of training cost consumption and inference time consumption, which is difficult to meet the real-time requirements of medical reasoning; another common idea is to enhance the output feature map through feature aggregation methods such as skip connections, void convolutions, and feature pyramids, but many studies have shown that such strategies cannot meet the feature alignment of the same object and it is difficult to improve the performance of medical image segmentation. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a lymph node nodule ultrasound image small target segmentation system based on attention self-distillation, comprising

[0006] A data set construction module is used to collect, clean and annotate lymph node ultrasound images, construct an ultrasound image data set and obtain mask annotations;

[0007] A data set division module, used for dividing the ultrasound image data set into an in-domain data set and an out-of-domain data set;

[0008] A model building module is used to build a segmentation model model, which includes a U-net network, a single-layer convolutional neural network, and a self-distillation supervision module. The ultrasound image of the lymph node nodule and the mask annotation are used as inputs of the segmentation model model. The ultrasound image of the lymph node nodule is passed through a single-layer convolutional neural network to obtain a nodule image map, and the ultrasound image of the lymph node nodule is passed through a U-net network to obtain a nodule feature map. The nodule image map and the nodule feature map are simultaneously input into the self-distillation supervision module to calculate the overall loss function of the model.

[0009] The iterative training module is used to build the Adam optimizer, use the overall loss function to perform reverse gradient propagation on the segmentation model, obtain the optimal parameters of the segmentation model, and continue to iterate;

[0010] The model evaluation module is used to evaluate the performance of the segmentation model after each iteration and save the model with the highest evaluation index in the historical training;

[0011] The model generalization verification module is used to load the model with the highest evaluation index in historical training, use out-of-domain datasets to verify the generalization performance of the model, and evaluate the detection rate of the model on out-of-domain datasets and the accuracy of detecting lymph node nodules.

[0012] The technical solution further defined in the present invention is:

[0013] Furthermore, in the dataset construction module, data collection specifically involves collecting different lymph node ultrasound images and corresponding pathology reports from multiple hospitals; data cleaning specifically involves screening lymph node ultrasound images based on resolution, contrast, grayscale range, and dynamic range, and the lymph node ultrasound images must correspond one-to-one with the pathology reports; data annotation specifically involves doctors outlining the nodule range on the lymph node ultrasound image in combination with the pathology report.

[0014] As mentioned above, in the lymph node nodule ultrasound image small object segmentation system based on attention self-distillation, the in-domain dataset is also divided into a training set and a test set in the dataset partitioning module. The training set is used for iterative training of the segmentation model, and the test set is used for performance evaluation of the segmentation model.

[0015] As mentioned above, in the lymph node nodule ultrasound image small object segmentation system based on attention self-distillation, in the data set partitioning module, the in-domain data set is divided into a training set and a test set in a ratio of 8:2.

[0016] As mentioned above, in the lymph node nodule ultrasound image small object segmentation system based on attention self-distillation, in the segmentation model, the lymph node nodule ultrasound image is divided into different regions through a single-layer convolutional neural network, and the information of the nodule area is marked in combination with mask annotation to obtain the nodule image mapping.

[0017] As mentioned above, in the lymph node nodule ultrasound image small target segmentation system based on attention self-distillation, in the segmentation model model, the lymph node nodule ultrasound image is processed by each layer of the U-net network to obtain feature information of different depths, and the aggregation network is used to aggregate the feature information, and the aggregated feature map is divided into regions, and the aggregated features of the nodule are marked with mask annotation to obtain the nodule feature map.

[0018] As mentioned above, in the lymph node nodule ultrasound image small target segmentation system based on attention self-distillation, the metric distance between the nodule image map and the nodule feature map is calculated in the self-distillation supervision module to obtain the loss value l dis ; Then the segmentation loss of the model is calculated based on the predicted mask and the true mask, denoted as l seg ; Finally, the above loss values ​​are weighted and summed to get the total loss l all , the total loss is as follows:

[0019] l all =αl seg +βl dis

[0020] Among them, l all represents the overall loss, l dis represents the attention alignment relation loss, l seg represents the segmentation loss, α and β represent the weights of the segmentation loss and the attention alignment relationship loss, respectively.

[0021] As mentioned above, in the lymph node nodule ultrasound image small target segmentation system based on attention self-distillation, the Euclidean metric self-attention distillation loss is used as the attention alignment relationship loss in the self-distillation supervision module. The attention alignment relationship loss is shown as follows:

[0022]

[0023] Among them, Att represents the distilled attention relationship between the image and the feature map. represents the feature-aligned attention computed over the ground-truth segmentation mask.

[0024] As mentioned above, in the lymph node nodule ultrasound image small target segmentation system based on attention self-distillation, the weight of the segmentation loss is α=1, and the weight of the attention alignment relationship loss is β=0.2.

[0025] As mentioned above, in the lymph node nodule ultrasound image small object segmentation system based on attention self-distillation, in the model evaluation module, the evaluation index is set to the DICE coefficient, and the model with the highest DICE coefficient of historical results is saved during the training process.

[0026] The beneficial effects of the present invention are:

[0027] (1) In the present invention, the self-distillation supervision module is used to perform attention guidance calculation on the nodule features encoded by the model and the nodule area in the original image, and the feature information of the large nodule is used to compensate for the information loss caused by the reduction in the resolution of the feature map. Due to the similarity of the imaging of the nodule area, the nodule image mapping can guide the feature alignment relationship of the nodule feature mapping, quantify the relationship between each input image and the corresponding feature map, and distill the feature knowledge learned from the large nodule area to the small nodule area, thereby improving the segmentation performance of small nodules in lymph node segmentation, improving the robustness and generalization of the model for nodule detection of different sizes, and better adapting to data sets and actual clinical scenarios in different fields;

[0028] (2) In the present invention, the information loss of small nodules is compensated by the consistency of features of the same object; and the attention self-distillation technology is a plug-and-play module that can accept encoding network input of any type or specification and can be reused in any segmentation model, with higher operational freedom;

[0029] (3) In the present invention, in addition to lymph node nodules, the ultrasound image small target segmentation system can also be reused for small target segmentation tasks in various fields such as thyroid, skin diseases and breast, etc. according to actual usage, and has higher practicality and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a principle block diagram of the overall system of the present invention;

[0031] Figure 2 It is a schematic diagram of the structure of the segmentation model model in an embodiment of the present invention;

[0032] Figure 3 This is a diagram of the attention self-distillation technology architecture in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] This embodiment provides a lymph node ultrasound image small target segmentation system based on attention self-distillation, such as Figure 1 Shown, including

[0034] The data set construction module is used to collect, clean and annotate lymph node ultrasound images to form an ultrasound image data set, and senior professional doctors annotate them with reference to pathology reports to obtain mask annotations; data collection specifically involves collecting different lymph node ultrasound images and corresponding case conclusions or pathology reports from multiple hospitals; data cleaning specifically involves screening out high-quality lymph node ultrasound images; data annotation specifically involves outlining the nodule range on the lymph node ultrasound image by professional doctors in combination with their judgment of the pathology report.

[0035] High-quality lymph node ultrasound images must meet the following conditions: 1. The ultrasound image and pathology report can correspond one to one; 2. The ultrasound image has high resolution and contrast, without serious noise interference; 3. The ultrasound image should have a good grayscale range and dynamic range, and be able to clearly display echoes of different degrees.

[0036] The dataset partitioning module is used to divide the ultrasound image dataset into in-domain dataset and out-of-domain dataset. Factors such as region, age, and ultrasound equipment should be considered when dividing the dataset. The in-domain dataset is used for model training and tuning. The in-domain dataset is divided into training set and test set in a ratio of 8:2. The training set is used for iterative training of the model, and the test set is used for performance evaluation of the model. The out-of-domain dataset should include ultrasound diagnostic images of different regions, different ages, and different equipment as much as possible to verify the true generalization performance of the model.

[0037] The out-of-domain dataset is used for inference and prediction of the model to verify its actual application performance. At the same time, the in-domain dataset is divided into a training set and a test set in a ratio of 8:2. The training set is used for iterative training of the model, and the test set is used for performance evaluation of the model.

[0038] Model building module, used to build segmentation model model, such as Figure 2 As shown in the figure, the segmentation network used is the U-net network architecture. The segmentation model model includes the U-net network, a single-layer convolutional neural network and a self-distillation supervision module. The attention self-distillation technology is a plug-and-play module that can accept any type and specification of encoding network input, and after processing, it is input into the decoder to generate a semantic mask map.

[0039] The default input of the segmentation model model is the ultrasound image of lymph node nodules and mask annotations. First, the encoder will process the input image and aggregate the features at each level in turn. At the same time, the input image will be divided into different small areas by a single-layer convolutional neural network, which is used to mark the area where nodules exist, recorded as the nodule image map PatchImage; secondly, the aggregated feature map will also be divided into different small areas by a single-layer convolutional neural network, which is used to mark the area covered by nodule features, recorded as the nodule feature map PatchFeature. Due to the similarity of nodule area imaging, the nodule image map PatchImage can guide the feature alignment relationship of the nodule feature map PatchFeature, quantify the relationship between each input image and the corresponding feature map, and distill the feature knowledge learned in the large nodule area to the small nodule area, thereby improving the segmentation performance of small nodules in lymph node nodule segmentation, improving the robustness and generalization of the model for nodule detection of different sizes, and better adapting to data sets and actual clinical scenarios in different fields.

[0040] The attention self-distillation technology proposed in this embodiment performs attention-guided calculation on the nodule features encoded by the model and the nodule area in the original image, and uses the feature information of large nodules to compensate for the information loss caused by the reduction of feature map resolution. The specific network structure and forward propagation process are as follows: Figure 3 As shown in the figure, the attention self-distillation technology is based on the feature encoding of the image by the encoder and only contains two additional convolutional neural network layers. Its architecture is light in weight and small in parameters, which can meet the real-time requirements of medical image segmentation reasoning.

[0041] The ultrasound image of lymph node nodules is divided into different regions through a single-layer convolutional neural network, and the information of the nodule area is marked in combination with the real mask information (i.e., mask annotation) to obtain a nodule image mapping; the ultrasound image of lymph node nodules is processed by each layer of the U-net network to obtain feature information of varying depths, and the feature information is aggregated using an aggregation network, and the aggregated feature map is divided into regions, and the aggregated features of the nodules are marked in combination with the real mask information (i.e., mask annotation) to obtain a nodule feature mapping, where the aggregated features of the nodules are the features decoded at each layer of the model, and are fused.

[0042] The nodule image map and nodule feature map are simultaneously input into the self-distillation supervision module, and the metric distance between the nodule image map and the nodule feature map is calculated to obtain the loss value l dis ; Then the segmentation loss of the model is calculated based on the predicted mask and the true mask, denoted as l seg ; Finally, the above loss values ​​are weighted and summed to get the total loss l all , the total loss is as follows:

[0043] l all =αlseg +βl dis

[0044] For the basic segmentation task, this embodiment uses the classic cross entropy loss as the segmentation loss l seg , to obtain shape supervision based on the Dice coefficient. Therefore, the overall loss function consists of two parts: one is the segmentation task loss, and the other is the attention alignment relationship loss guided by self-distillation. In the above formula, l all represents the overall loss, l dis represents the attention alignment relation loss, l seg represents the segmentation loss, α and β represent the weights of the segmentation loss and the attention alignment loss, respectively. In this embodiment, the weight of the segmentation loss is α=1, and the weight of the attention alignment loss is β=0.2.

[0045] The goal of the self-attention distillation loss is to align the feature maps guided by the ultrasound image. In order to allow the model to better learn the alignment relationship, this embodiment calculates a reliable feature alignment relationship based on the real segmentation mask. By minimizing the distance between the two alignment relationships, the model can better learn the similarity relationship of the same target, thereby better utilizing the information of the large nodules; therefore, in order to preserve the integrity of the content information, this embodiment uses the Euclidean metric self-attention distillation loss, as shown in the following formula:

[0046]

[0047] Among them, Att represents the distilled attention relationship between the image and the feature map. represents the feature-aligned attention computed over the ground-truth segmentation mask.

[0048] The iterative training module is used to build the Adam optimizer, use the training set of the in-domain dataset to train the segmentation model model, use the overall loss function to perform reverse gradient propagation on the segmentation model model, obtain the optimal parameters of the segmentation model model, and continue to iterate.

[0049] The model evaluation module is used to evaluate the performance of the segmentation model on the test set of the in-domain dataset after each iteration. The evaluation indicator is set to the DICE coefficient, and the model with the highest DICE coefficient in the historical training is saved. The Dice coefficient is used to calculate the similarity between the predicted results and the true labels. It is defined as the ratio of twice the intersection area of ​​two sets (predicted results and true labels) to the total area of ​​the two sets. It is particularly suitable for binary image segmentation tasks. The higher the DICE coefficient, the higher the overlap between the predicted results and the true labels.

[0050] The model generalization verification module is used to load the model with the highest evaluation index in historical training, use out-of-domain datasets to verify the generalization performance of the model, and evaluate the detection rate of the model on out-of-domain datasets and the accuracy of detecting lymph node nodules.

[0051] This embodiment uses a self-distillation supervision module to perform attention-guided calculations on the nodule features encoded by the model and the nodule areas in the original image, and uses the feature information of large nodules to compensate for the information loss caused by the reduction in feature map resolution. Due to the similarity of nodule area imaging, nodule image mapping can guide the feature alignment relationship of nodule feature mapping, quantify the relationship between each input image and the corresponding feature map, and distill the feature knowledge learned from the large nodule area to the small nodule area, thereby improving the segmentation performance of small nodules in lymph node nodule segmentation, improving the robustness and generalization of the model for nodule detection of different sizes, and better adapting to data sets and actual clinical scenarios in different fields.

[0052] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope required by the present invention.

Claims

1. A small target segmentation system for lymph node nodules in ultrasound images based on attention self-distillation, characterized by: include A data set construction module is used to collect, clean and annotate lymph node ultrasound images, construct an ultrasound image data set and obtain mask annotations; A data set division module, used for dividing the ultrasound image data set into an in-domain data set and an out-of-domain data set; A model building module is used to build a segmentation model model, which includes a U-net network, a single-layer convolutional neural network, and a self-distillation supervision module. The ultrasound image of the lymph node nodule and the mask annotation are used as inputs of the segmentation model model. The ultrasound image of the lymph node nodule is passed through a single-layer convolutional neural network to obtain a nodule image map, and the ultrasound image of the lymph node nodule is passed through a U-net network to obtain a nodule feature map. The nodule image map and the nodule feature map are simultaneously input into the self-distillation supervision module to calculate the overall loss function of the model. The iterative training module is used to build the Adam optimizer, use the overall loss function to perform reverse gradient propagation on the segmentation model, obtain the optimal parameters of the segmentation model, and continue to iterate; The model evaluation module is used to evaluate the performance of the segmentation model after each iteration and save the model with the highest evaluation index in the historical training; The model generalization verification module is used to load the model with the highest evaluation index in historical training, verify the generalization performance of the model using an out-of-domain dataset, and evaluate the detection rate of the model on the out-of-domain dataset and the accuracy of detecting lymph node nodules; In the self-distillation supervision module, the metric distance between the nodule image map and the nodule feature map is calculated to obtain the loss value l dis ; Then the segmentation loss of the model is calculated based on the predicted mask and the true mask, denoted as l seg ; Finally, the above loss values ​​are weighted and summed to get the total loss l all , the total loss is as follows: l all ＝αl seg +βl dis Among them, l all represents the overall loss, l dis represents the attention alignment relation loss, l seg represents the segmentation loss, α and β represent the weights of the segmentation loss and the attention alignment relationship loss respectively; In the self-distillation supervision module, the Euclidean metric self-attention distillation loss is used as the attention alignment relationship loss. The attention alignment relationship loss is shown as follows: Among them, Att represents the distilled attention relationship between the image and the feature map. represents the feature-aligned attention computed over the ground-truth segmentation mask.

2. The lymph node nodule ultrasound image small target segmentation system based on attention self-distillation according to claim 1 is characterized by: In the dataset construction module, data collection specifically involves collecting different lymph node ultrasound images and corresponding pathology reports from multiple hospitals; data cleaning specifically involves screening lymph node ultrasound images based on resolution, contrast, grayscale range, and dynamic range, and the lymph node ultrasound images must correspond one-to-one with the pathology reports; data annotation specifically involves doctors outlining the nodule range on the lymph node ultrasound images in combination with the pathology reports.

3. The lymph node nodule ultrasound image small target segmentation system based on attention self-distillation according to claim 1 is characterized by: In the data set partitioning module, the in-domain data set is also divided into a training set and a test set. The training set is used for iterative training of the segmentation model model, and the test set is used for performance evaluation of the segmentation model model.

4. The lymph node nodule ultrasound image small target segmentation system based on attention self-distillation according to claim 3 is characterized by: In the data set division module, the data set in the domain is divided into a training set and a test set according to a ratio of 8:

2.

5. The lymph node nodule ultrasound image small target segmentation system based on attention self-distillation according to claim 1, characterized in that: In the segmentation model, the lymph node ultrasound image is divided into different regions through a single-layer convolutional neural network, and the information of the nodule area is marked in combination with mask annotation to obtain a nodule image map.

6. The lymph node nodule ultrasound image small target segmentation system based on attention self-distillation according to claim 1, characterized in that: In the segmentation model, the ultrasonic image of the lymph node nodule is processed by each layer of the U-net network to obtain feature information of different depths, the feature information is aggregated using the aggregation network, and the aggregated feature map is divided into regions, and the aggregated features of the nodule are marked in combination with mask annotation to obtain a nodule feature map.

7. The lymph node nodule ultrasound image small target segmentation system based on attention self-distillation according to claim 1, characterized in that: The weight of the segmentation loss is α=1, and the weight of the attention alignment relationship loss is β=0.

2.

8. The lymph node nodule ultrasound image small target segmentation system based on attention self-distillation according to claim 1, characterized in that: In the model evaluation module, the evaluation index is set to the DICE coefficient, and the model with the highest DICE coefficient of historical results is saved during the training process.

Images