Weakly supervised segmentation system for thyroid ultrasound nodules based on hybrid annotation

Abstract

A weakly supervised segmentation system for thyroid ultrasound image nodules based on hybrid annotation includes: a feature extraction module composed of a main model and an auxiliary model and a region segmentation module based on contrastive learning. The feature extraction module in the main model generates a feature map according to the original image and the hybrid annotation to the region segmentation module. The region segmentation module uses contrastive learning to divide the feature map into foreground and background representations of the thyroid nodule. After forming a foreground pair based on the diffusion area of ​​the point in the foreground area and a background pair based on the diffusion area of ​​the graffiti annotation in the background area, the two are compared based on the distance between them to achieve segmentation of the nodule area and obtain a rough segmentation of the thyroid nodule. The auxiliary model and the main model share weights. The present invention adopts a hybrid annotation method of points and graffiti to better enhance the repeatability and objectivity of the experiment.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, specifically a weakly supervised segmentation system for thyroid ultrasound image nodules based on mixed annotation. Background Art

[0002] In recent years, weakly supervised learning has been widely used in the automatic segmentation of thyroid ultrasound images. The goal of weakly supervised learning is to leverage large amounts of simple annotated data, such as image-level annotations, bounding box annotations, and point and scribble annotations, to reduce annotation costs and improve segmentation performance. However, due to the fuzzy edges and complex boundaries of thyroid nodules, it is difficult to distinguish them from surrounding tissue. Furthermore, the limited supervisory information provided by a single weak annotation can lead to insufficient model training and poor segmentation performance. Summary of the Invention

[0003] In response to the above-mentioned deficiencies in the prior art, the present invention proposes a weakly supervised segmentation system for thyroid ultrasound image nodules based on hybrid annotation. The system adopts a hybrid annotation method of dots and scribbles and utilizes contrastive learning technology to enhance the feature differences between nodules and background. The annotation is simpler and does not contain redundant background information. At the same time, the system adopts a fully automatic method, which can better enhance the repeatability and objectivity of the experiment.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a weakly supervised segmentation system for thyroid ultrasound image nodules based on hybrid annotation, comprising: a feature extraction module composed of a main model and an auxiliary model and a region segmentation module based on contrastive learning, wherein: the feature extraction module in the main model generates a feature map according to the original image and the hybrid annotation to the region segmentation module, the region segmentation module uses contrastive learning to divide the feature map into thyroid nodule foreground and background representations, forms a foreground pair according to the diffusion area of ​​the point in the foreground area, forms a background pair according to the diffusion area of ​​the graffiti annotation in the background area, and compares the two based on the distance between them to achieve segmentation of the nodule area and obtain a rough segmentation of the thyroid nodule, the auxiliary model and the main model share weights, and outputs a fine segmentation result based on the image to be tested after affine transformation through the trained auxiliary model in the online stage.

[0006] The hybrid annotation refers to: using a point and scribble method to perform hybrid annotation on thyroid nodule ultrasound images, that is, after using a point to mark the center point L1 of each thyroid nodule, using scribble annotation to describe the boundary of the lesion around the nodule and provide more detailed information about its shape and location L0; creating a point and scribble hybrid annotation for label input of the weakly supervised segmentation algorithm.

[0007] The affine transformations described above include random flipping and rotation operations. Flipping flips the image horizontally or vertically, while rotation rotates the image by a certain angle. During each training session, one of these types of transformations is randomly selected and applied to the network to improve its generalization capabilities.

[0008] The trained auxiliary model refers to: using the same affine transformation as the main model to process the image to be tested and inputting it into the auxiliary model to obtain a segmentation result, and using consistency constraints to process the obtained main model and auxiliary model segmentation results. By penalizing inconsistent segmentation boundaries, the network is encouraged to predict more coherent and consistent boundaries, thereby significantly improving the segmentation accuracy.

[0009] Technical Effects

[0010] The present invention adopts a hybrid labeling method to reduce the overall labeling workload while solving the problem of insufficient supervision information of single weak labeling and significantly improving the segmentation accuracy; the affine transformation consistency loss function is used to reduce the difference in the output results of the backbone network and the auxiliary network, encourage consistent prediction, and thus improve the segmentation results on the complex boundaries of thyroid nodules. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a system structure diagram of the present invention;

[0012] Figure 2 Schematic diagram of different labeling forms for three thyroid nodules;

[0013] In the figure: (a) is point annotation, (b) is mixed annotation, (c) is graffiti annotation, and (d) is the gold standard;

[0014] Figure 3 Visualize the output result graph for the main model branch;

[0015] In the figure: (a) is the output of the salient region activation branch, and (b) is the output of the similarity correction branch based on contrastive learning;

[0016] Figure 4 Schematic diagram of affine transformation consistency;

[0017] Figure 5 Schematic diagram of CAM vision with and without the affinity region segmentation module based on contrastive learning;

[0018] In the figure: (a) is the original image, (b) is the backbone network result, (c) is the result of adding the region segmentation module without contrastive learning, (d) is the result of adding the region segmentation module with contrastive learning, and (e) is the gold standard;

[0019] Figure 6Results of different affine transformation consistency strategies are presented;

[0020] In the figure: (a) is the original image, (b) is the basic framework result, (c) is the result of adding horizontal flip affine transformation, (d) is the result of adding rotation affine transformation, (e) is the gold standard of randomly selected affine transformation results, and (f) is the gold standard;

[0021] Figure 7 Schematic diagram of segmentation effects of different methods. DETAILED DESCRIPTION

[0022] like Figure 1 As shown, this embodiment relates to a weakly supervised segmentation system for thyroid ultrasound image nodules based on hybrid annotation, including: a feature extraction module composed of a main model and an auxiliary model and a region segmentation module based on contrastive learning, wherein: the main model generates a feature map according to the original image and the hybrid annotation to the region segmentation module, the region segmentation module uses contrastive learning to divide the feature map into thyroid nodule foreground and background representations, and then forms a foreground pair according to the diffusion area of ​​the point in the foreground area and a background pair according to the diffusion area of ​​the graffiti annotation in the background area, and compares the two based on the distance between them to achieve segmentation of the nodule area and obtain a rough segmentation of the thyroid nodule, and the auxiliary model and the main model share weights, and in the online stage, the trained auxiliary model outputs a fine segmentation result based on the image to be tested after affine transformation.

[0023] The main model and auxiliary model both use U-Net as the basic architecture. The main model performs back propagation to update parameters, and then uses the parameters of the main model to update the auxiliary model parameters through exponential moving average.

[0024] like Figure 1 As shown, the region segmentation module includes: a salient region activation branch and a similarity correction branch based on affinity contrast learning, wherein: the salient region activation branch identifies the region of interest in the feature map obtained by the main model and generates Figure 3 (a) shows the nodule probability heat map; the similarity correction branch captures the semantic similarity between vector pairs in the feature map obtained by the main model, generating Figure 3 The similarity matrix shown in (b) propagates the point annotations to other areas inside the unlabeled nodule, and then converts the similarity matrix into a propagation path matrix. The propagation path matrix is ​​applied to the nodule possibility heat map through the random step method to achieve the diffusion of the point annotations to other locations of the nodule and obtain a rough segmentation.

[0025] The semantic similarity refers to the distance between the center points L1 of each thyroid nodule in the feature map, which measures the difference in their representation.

[0026] The propagation path matrix Wij =exp{-||f sim (x i ,y i )-f sim (x j ,y j )||1}, where: (x i ,y i ) is the coordinate of the i-th feature on the similarity matrix, W ij Higher values ​​in correspond to closer L1 distances, indicating greater similarity between feature vectors. When the L1 distance between features is close, the similarity between the point annotation and pixels with similar features is stronger. When the distance is greater, the similarity approaches 0, thus preventing the influence of distant noise points on the segmentation results.

[0027] The propagation path matrix is ​​normalized by applying the Hadamard power value α to the similarity matrix to obtain T = L -1 W α ,in: The exponent α controls the degree of similarity between feature vectors. Higher values ​​of α make path propagation more conservative, making it more robust to small changes in pairwise distances in feature space.

[0028] like Figure 1 As shown in FIG, the present embodiment is a method for weakly supervised segmentation of thyroid ultrasound image nodules based on hybrid annotation based on the above system, comprising the following steps:

[0029] Step S1 : Desensitizing and data preprocessing the thyroid ultrasound image. Specifically, after deleting the patient's privacy information from the ultrasound image, the original image is cropped to obtain a region of interest with a size of 512×512 pixels as the input image.

[0030] Step S2: Hybrid annotation of thyroid nodule ultrasound images is performed using the dot and scribble method to construct a nodule segmentation training sample set. Specifically, the annotation L of the input image is used. n ∈{0,1,2} H×W , assigning each pixel one of three types of labels: label 0 is the background pixel marked by graffiti; label 1 is the point-marked pixel in the thyroid nodule, which is used to provide the location information of the nodule of interest; label 2 is the unlabeled pixel, which is used for the area other than graffiti and point-marked pixels. H and W are the height and width dimensions of the image respectively. Figure 2 As shown in the figure, point annotations are used to provide the location information of the nodules of interest, and scribbles are used to provide rough supervision information of the nodule boundaries. Compared with single weak annotations, it provides richer nodule location and boundary information, which is used to improve the accuracy of the segmentation results.

[0031] Step S3, using the thyroid nodule segmentation training sample set to train the weakly supervised segmentation system, that is, using the weighted combination of the main model loss function and the affine transformation consistency loss function for training, the loss function Where: N is the number of mixed annotation images in the training set, L s and L c They are the main model loss function and consistency loss function respectively.

[0032] The main model loss function is the standard cross entropy loss function Middle: C e is the set of categories present in the image, C + (C-) is the label set of the same (different) category as pixel i, is the predicted label and mixed annotation, L I is the loss function for the foreground region expansion, which is used to measure the prediction effect of the model on nodule expansion. For each category C∈C e , calculate the predicted label The predicted probability of class C in L is calculated and compared with the probability of class C in the actual label Y to implement the consistency constraint and expand the nodule region. P Contrastive learning is achieved by modeling the similarity between nodule regions. Specifically, for each pixel i, L P Calculate the probability of similar pixels and try to distinguish them from other dissimilar pixels. P Loss function, L P The network is encouraged to assign pixels of the same class into the same embedding space and to keep pixels of different classes separate.

[0033] The consistency loss function is used to measure the consistency of the prediction results between the main model and the auxiliary model, and is used to constrain the segmentation effect at the boundary. Specifically, L c =∑ j∈I |T(f θ (X i )) j -f θ (T(X i )) j |, where: T() is the affine transformation, f θ is the output result, and I is the input ultrasound image.

[0034] like Figure 4As shown, two types of affine transformations are used to augment the training data. The transformed images maintain consistent segmentation results between the main model and the untransformed images. These transformations include random flipping and rotation operations. Flipping flips the image horizontally or vertically, while rotation rotates the image by a certain angle. During each training step, one of these transformations is randomly selected and applied to the network to improve its generalization capabilities.

[0035] To maintain consistency, the prediction results of the main segmentation model are used for evaluation, that is, when calculating L c When the main model and the auxiliary model are trained, the differences between their prediction results are compared and their average mean square error is calculated to help optimize the boundary segmentation effect and improve the performance of the model.

[0036] In a specific practical experiment, using the Pytorch library and an Nvidia GTX 3090 graphics card, 350 ultrasound images of thyroid nodules were collected using ultrasound equipment of different brands. The contours of each nodule and the benign and malignant labels were manually annotated.

[0037] In the specific implementation, the Adam optimizer is used, and its weight decay value is 10 -3 The batch size for training is set to 4 and the initial learning rate is set to 10 -4 The training loop is executed for a total of 300 epochs, where one epoch is defined as one complete pass of the training dataset through the neural network. Furthermore, a weighted loss function is used during training. The weights of the loss function are set to λ1 = 1, corresponding to the weight of the main model loss function, and λ2 = 0.5, corresponding to the weight of the affine transformation consistency loss function.

[0038] This example evaluates the effectiveness of each module in the weakly supervised segmentation algorithm through the Dice coefficient, Hausdorff distance (HD), and mean surface distance error (μD), and verifies the performance of the weakly supervised and fully supervised segmentation algorithms in thyroid nodule segmentation.

[0039] In order to quantify and effectively evaluate the segmentation results, the Dice coefficient, HD and μD are calculated to evaluate the segmentation effect of the region and boundary. The Dice coefficient is a set similarity measurement indicator, which is usually used to calculate the similarity between the segmentation result and the gold standard. Its value is between 0 and 1. The closer the value is to 1, the higher the similarity between the segmentation result and the gold standard, that is, the better the segmentation result; the closer the value is to 0, the lower the similarity between the segmentation result and the gold standard, that is, the worse the segmentation result. HD and μD are the maximum and average values ​​of the surface distance error between the segmentation result and the gold standard, and are a measure of the degree of fit between the segmentation result and the gold standard at the boundary. The lower the HD and μD, the better the network of this embodiment works at the boundary.

[0040] Where: A and B are the segmentation results of the gold standard and this embodiment respectively, a and b are two corresponding points on A and B; d(a, B) and d(b, A) are the minimum distances from point a and b to B and A respectively.

[0041] First, an ablation experiment is conducted on the region segmentation module based on contrastive learning. Figure 5 The results show that the region segmentation module based on contrastive learning achieves significant performance improvement in the segmentation task. By introducing the affinity matrix without contrastive learning into the basic framework, the Dice coefficient of this embodiment is improved from 0.77±0.13 to 0.82±0.05, and the HD is reduced from 22.92±9.53 to 21.74±8.16. This shows that the affinity matrix strategy can effectively improve the segmentation accuracy. In addition, after introducing the region segmentation module based on contrastive learning into the basic framework, the performance of this embodiment is further improved, with the Dice coefficient reaching 0.84±0.07, an increase of 9.1 percentage points compared to the basic framework, the HD is reduced to 21.53±7.61, and the μD is reduced to 1.73±0.37. This improvement is highly competitive, indicating that the affinity region segmentation module based on contrastive learning is more effective in improving the accuracy of thyroid nodule segmentation.

[0042] Table 1. The improvement effect of the region segmentation module based on contrastive learning on network performance. The standard deviation is in brackets.

[0043]

[0044] like Figure 5As shown, the CAM graph visualization shows that this module enables this embodiment to focus more on the nodule salient area, thereby achieving better segmentation performance. After adding contrastive learning, this embodiment can spread the point annotation to a larger area, indicating that this embodiment can better separate the foreground and background and learn more comprehensive and accurate nodule features. In summary, the region segmentation module based on contrastive learning can guide this embodiment to focus on the nodule salient area, and is very effective in improving the nodule segmentation performance under weakly supervised learning.

[0045] Table 2 Ablation experiments for different types of bionic transformations. The standard deviations are in brackets.

[0046]

[0047]

[0048] Next, we explore the impact of two different types of affine transformations, horizontal flip and rotation, on the performance of this embodiment. The experimental results are shown in Table 2 and Figure 6 As shown. Experimental results show that the use of rotation transformation can significantly improve the segmentation results. Compared with the basic framework, the Dice coefficient increased from 0.80±0.05 to 0.83±0.06, an increase of 3.75%; the HD decreased from 30.05±12.69 to 22.84±8.43, a decrease of 24.0%; and the μD decreased from 4.11±0.96 to 2.65±0.62, a decrease of 35.5%. The horizontal flip transformation operation also had a positive impact on the results of this embodiment, with the Dice coefficient increased to 0.81±0.13; the HD decreased to 28.72±9.96; and the μD decreased to 3.96±0.94, achieving performance improvement. It should be noted that when both bionic transformation operations are used, the Dice coefficient is 0.84±0.07, the HD is 21.53±7.61, and the μD is 1.73±0.37, achieving the best results.

[0049] The rotation operation simulates different viewing angles and can provide the network with more diverse and robust features, better constraining the segmentation at irregular nodule boundaries. On the other hand, the horizontal flip operation can improve the generalization ability of the network by increasing the variability of the training data. Figure 6 As can be seen in the two nodule examples given, using only the horizontal flip affine transformation can only achieve good results on some smooth boundaries, but the effect is poor for more complex boundaries, and the segmentation results are less complete than the gold standard. The random rotation affine transformation was chosen because it can provide more diverse feature expression capabilities and can also achieve good results for complex boundaries, and the best results are achieved when a random affine transformation is used. The results show that incorporating the affine transformation operation into this embodiment can significantly improve the boundary segmentation ability and can be regarded as an effective strategy to improve the accuracy of the weakly supervised thyroid nodule segmentation task.

[0050] In addition, the experimental results of this embodiment are compared with other advanced weakly supervised and fully supervised segmentation methods, among which: the comparison of weakly supervised methods includes weakly supervised adaptive soft ablation and edge self-attention algorithm (WSASE), weakly supervised teacher-student semantic segmentation algorithm (WSMTL), weakly supervised PseudoEdgeNet algorithm and this embodiment. The comparison of fully supervised methods includes U-Net algorithm and MS-MTL algorithm. As shown in Table 3, the Dice coefficient, HD and μD values ​​of all compared methods in segmenting thyroid nodules are listed with the corresponding standard deviation. The results show that this method achieves performance comparable to that of the fully supervised method while using less annotation time (5 seconds), and significantly outperforms other weakly supervised methods.

[0051] Table 3 shows the comparison of segmentation results with the most advanced methods, with standard deviations in brackets.

[0052]

[0053]

[0054] like Figure 7 As shown, the segmentation results of three typical thyroid ultrasound images are compared. From top to bottom, thyroid nodules with complex boundaries, fuzzy edges and regular shapes are shown respectively. It can be seen from this that the superiority of this embodiment in these scenarios. For nodules with regular shapes, almost all methods can achieve good segmentation results. However, for nodules with fuzzy boundaries and complex boundaries, this embodiment performs best among the weakly supervised methods tested. These situations are very common in actual clinical applications, so these results are of great significance for applying this embodiment to actual scenarios. In addition, the performance of this embodiment on HD is even close to that of the fully supervised MS-MTL, which shows that when training data is limited, the accuracy close to that of the fully supervised method can be obtained using this embodiment.

[0055] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principles and purpose of the present invention. The scope of protection of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. All implementation schemes within its scope shall be subject to the constraints of the present invention.

Claims

1. A weakly supervised segmentation system for thyroid ultrasound nodules based on hybrid annotation, characterized by: include: The system consists of a feature extraction module and an auxiliary model, and a region segmentation module based on contrastive learning. The feature extraction module in the main model generates a feature map based on the original image and mixed annotations, which is then passed to the region segmentation module. The region segmentation module uses contrastive learning to separate the feature map into foreground and background representations of the thyroid nodule. Foreground pairs are formed based on the diffusion area of ​​the points in the foreground area, and background pairs are formed based on the diffusion area of ​​the graffiti annotations in the background area. The two pairs are then compared based on the distance between them to segment the nodule area and obtain a rough segmentation of the thyroid nodule. The auxiliary model and the main model share weights, and in the online stage, the trained auxiliary model outputs a refined segmentation result based on the image to be tested after affine transformation. The mixed annotation refers to the mixed annotation of the thyroid nodule ultrasound image using the dot and graffiti method, that is, using the dot to mark the center point of each thyroid nodule. Finally, use scribble annotations to describe the boundaries of the lesion around the nodule and provide more details about its shape and location. ; Create mixed point and scribble annotations for label input to weakly supervised segmentation algorithms; The trained auxiliary model refers to: using the same affine transformation as the main model to process the image to be tested and inputting it into the auxiliary model to obtain a segmentation result, and using consistency constraints to process the obtained main model and auxiliary model segmentation results, and improving the segmentation accuracy by penalizing inconsistent segmentation boundaries.

2. The thyroid ultrasound image nodule weakly supervised segmentation system based on hybrid annotation according to claim 1 is characterized in that: The affine transformation includes random flipping and rotation operations, and one type of transformation is randomly selected each time during the training phase.

3. The thyroid ultrasound image nodule weakly supervised segmentation system based on hybrid annotation according to claim 1 is characterized in that: The region segmentation module includes: a salient region activation branch and a similarity correction branch based on affinity contrast learning, wherein: the salient region activation branch identifies the region of interest in the feature map obtained by the main model and generates a nodule possibility heat map; the similarity correction branch captures the semantic similarity between vector pairs in the feature map obtained by the main model, generates a similarity matrix, propagates the point annotations to other regions inside the unlabeled nodule, converts the similarity matrix into a propagation path matrix, and applies the propagation path matrix to the nodule possibility heat map through a random step method to achieve the diffusion of the point annotations to other locations of the nodule, thereby obtaining a rough segmentation; The semantic similarity refers to: the center point of each thyroid nodule in the feature map The spacing between them is used to measure the difference in their representation.

4. The thyroid ultrasound image nodule weakly supervised segmentation system based on hybrid annotation according to claim 3 is characterized in that: The propagation path matrix ,in: No. The coordinates of the features on the similarity matrix, Higher values ​​in correspond to The closer the distance; The propagation path matrix is ​​obtained by applying the Hadamard power value on the similarity matrix After normalization, we get ,in: ,index Controls the degree of similarity between eigenvectors.

5. A method for weakly supervised segmentation of thyroid ultrasound nodules based on hybrid annotation based on the system of any one of claims 1 to 4, comprising the following steps: Step S1, desensitizing and data preprocessing the thyroid ultrasound image, specifically: after deleting the patient's privacy information from the ultrasound image, cropping the original image to obtain a region of interest with a size of 512×512 pixels as the input image; Step S2: Hybrid annotation of thyroid nodule ultrasound images using the dot and scribble method to construct a nodule segmentation training sample set. Specifically, according to the annotation of the input image , each pixel is assigned one of three types of labels: label 0 is the background pixel marked by graffiti; label 1 is the point-marked pixel in the thyroid nodule, which is used to provide the location information of the nodule of interest; label 2 is the unlabeled pixel, which is used for the area other than graffiti and point marking. H and W are the height and width dimensions of the image respectively; Step S3, using the thyroid nodule segmentation training sample set to train the weakly supervised segmentation system, that is, using the weighted combination of the main model loss function and the affine transformation consistency loss function for training, the loss function , where: N is the number of mixed annotation images in the training set, and They are the main model loss function and consistency loss function respectively.

6. The method for weakly supervised segmentation of thyroid ultrasound nodules based on hybrid annotation according to claim 5, characterized in that: The main model loss function is the standard cross entropy loss function ,in: is the set of categories present in the image, and and pixels respectively Sets of labels of the same or different categories, are the predicted labels and mixed annotations, is the loss function for the expansion of the foreground area, which is used to measure the prediction effect of the model on nodule expansion; for each category in the image , calculate the predicted label middle The predicted probability of the class and compare it with the actual label middle The probabilities of the classes are compared to enforce consistency constraints and expand the nodule region.

7. The method for weakly supervised segmentation of thyroid ultrasound nodules based on hybrid annotation according to claim 5, characterized in that: The consistency loss function is used to measure the consistency of the prediction results between the main model and the auxiliary model, and is used to constrain the segmentation effect at the boundary. Specifically, ,in: is an affine transformation, To output the result, is the input ultrasound image.

8. The method for weakly supervised segmentation of thyroid ultrasound nodules based on hybrid annotation according to claim 5, wherein: The consistency of the affine transformation is evaluated using the prediction results of the main segmentation model, that is, when calculating When , the differences between the prediction results of the main model and the auxiliary model are compared, and their average mean squared error is calculated.

Images