A thyroid ultrasound image unsupervised domain adaptive semantic segmentation method based on feature decoupling

Abstract

The application relates to a kind of unsupervised domain adaptive semantic segmentation methods of thyroid ultrasound image based on feature decoupling, including feature decoupling and image translation two parts, the feature decoupling and image translation utilize feature constraint module and domain label coding module to carry out semantic segmentation.The feature constraint module of the application can improve the decoupling mode of structure-texture, based on the maximum entropy principle from the angle of task correlation, will domain invariant feature be expanded into task-oriented feature with strong task correlation, so that task-oriented feature contains rich detail information, improves the segmentation accuracy of nodule.Domain label coding module can learn the hidden feature representation of domain bias using domain label containing domain-specific information, and combine hidden features with task-oriented features to improve the stability of detail features, and then improve the domain invariance of task-oriented features, so that the segmentation model can more accurately distinguish benign and malignant nodules.

Description

Technical Field

[0001] This invention belongs to the field of semantic segmentation technology, specifically relating to an unsupervised adaptive semantic segmentation method for thyroid ultrasound images based on feature decoupling. Background Technology

[0002] Ultrasound imaging is a common medical imaging diagnostic method, characterized by its convenience and real-time capability, and is one of the important means of diagnosing thyroid nodules. However, ultrasound images contain a large amount of speckle noise, leading to blurred nodule boundaries, uneven image intensity, and a low signal-to-noise ratio, making image analysis difficult. Furthermore, the physician's experience has a significant impact on the diagnostic results, and the diagnostic process is subjective; with a significantly increased number of cases, missed diagnoses or misdiagnoses are prone to occur. Utilizing computer-aided diagnostic models to provide decision support for physicians is an effective way to improve the accuracy and efficiency of medical image analysis. For serious diseases, timely detection of nodules and accurate differentiation between benign and malignant nodules help physicians develop reasonable treatment plans for patients, enabling them to take necessary treatment measures based on the condition, preventing the growth and spread of nodules and causing more serious consequences, which can significantly reduce mortality, and is of great significance.

[0003] Computer-aided diagnostic models are now being applied to ultrasound imaging diagnosis, enabling tasks such as ultrasound image preprocessing, classification, and lesion segmentation. Among these, lesion segmentation models for thyroid ultrasound images can automatically segment nodules in ultrasound images and differentiate between benign and malignant nodules, thus improving diagnostic efficiency.

[0004] Current segmentation models typically assume that training and testing data have the same distribution. However, ultrasound images acquired by different devices vary greatly in terms of resolution, contrast, speckle noise, etc. This leads to differences between images acquired by different models, and even between images acquired by the same model with different parameter settings. The data distribution is inconsistent, meaning that there is a domain shift between images acquired by different models. As a result, a model trained on images from one model will perform poorly on images from other models.

[0005] To address these issues, researchers divided the dataset into source and target domains based on data distribution and proposed a domain-adaptive semantic segmentation method based on transfer learning to obtain models with better generalization capabilities. Since medical images such as ultrasound images have high annotation costs, it is often difficult to obtain pixel-level annotations for different target domains. Therefore, unsupervised domain adaptation (UDA) semantic segmentation, which is not constrained by target domain annotations, has attracted widespread attention from researchers.

[0006] UDA semantic segmentation methods primarily improve model segmentation performance in the target domain through self-training, image translation, and feature alignment. The domain adaptation process achieved through the first two methods is typically multi-stage. Self-training-based methods optimize the model using pseudo-labels from the target domain, with segmentation performance dependent on the quality of the pseudo-labels. Image translation-based methods achieve pixel-level domain adaptation by mapping the source domain image to the target domain, with effectiveness dependent on the stability of the image translation algorithm. Feature alignment-based methods achieve domain adaptation by mapping the image to feature vectors, reducing the feature distance between the source and target domains. Feature alignment is usually end-to-end and not limited by the quality of the pseudo-labels or the stability of the image translation algorithm. However, when the differences between the source and target domains are too large, the assumption of perfectly aligning all features of both domains is unrealistic, and the alignment process may affect the distinguishability of different categories of features. To alleviate the problem of poor feature distinguishability caused by excessive feature alignment, a series of feature decoupling methods, represented by Domain Invariant Structure Extraction (DISE) proposed by Chang et al., improve the model's segmentation performance on natural images by extracting domain-invariant features from the image for UDA semantic segmentation.

[0007] However, ultrasound images possess rich textural features, typically manifested in details such as nodule edges, calcifications, and echoes. Feature-based decoupling methods, in their decoupling process based on structure and texture, neglect the impact of image detail on the segmentation task. When segmenting lesions from ultrasound images, they struggle to retain rich detail in domain-invariant features, exhibiting weak ability to identify nodule edges, calcifications, and echoes, resulting in inaccurate nodule segmentation edges and incomplete lesion segmentation. Furthermore, inter-domain distribution differences lead to instability of domain-invariant features, limiting the model's generalization ability in the target domain. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide an unsupervised adaptive semantic segmentation method for thyroid ultrasound images based on feature decoupling. By providing a task-oriented feature decoupling algorithm, the generalization ability of the segmentation model can be improved, thereby enabling nodule segmentation in thyroid ultrasound images from multiple models and improving the diagnostic efficiency of doctors.

[0009] The technical problem solved by this invention is achieved through the following technical solution:

[0010] An unsupervised adaptive semantic segmentation method for thyroid ultrasound images based on feature decoupling is characterized by the following: the method is a task-oriented feature decoupling method, comprising two parts: feature decoupling and image translation, representing the source domain image as... Its pixel-level label is y s∈{0,1} H×W×C The target domain image is represented as The image encoder is represented by E, the image decoder by D, and the classification head for acquiring the segmentation map by C. seg .

[0011] Furthermore, the feature decoupling part utilizes the domain label encoding module to extract task-oriented features from the source domain. and the task-oriented characteristics of the target domain Through domain-specific encoders and Extract domain-specific task-independent features separately and Using the feature constraint module to reduce task relevance constraints and Task relevance, through image reconstruction constraints and They are complementary. and They are complementary, making and It can retain richer task-related details, thus enabling the identification of nodule edges and calcification information during segmentation; through C seg get Segmentation results Combined with label y s Optimize the field label encoding module, utilizing C seg Obtain the segmentation result Achieve output space alignment, thereby driving and Alignment;

[0012] The image translation component combines task-oriented and task-independent features from different domains to obtain a new image. and and source domain tag y s As The label optimization model is improved, and a semantic classifier C is added. sem verify and The domain invariance is used to optimize the domain label encoding process in the domain label encoding module.

[0013] Furthermore, the feature constraint module utilizes task-related constraints based on the maximum entropy principle to prevent task-irrelevant features from retaining task-related details, and uses image reconstruction constraints to encourage task-oriented features to retain this information; specifically, the task-oriented feature is represented as z. o The task-independent feature is z u In the task-related constraints, the feature constraint module utilizes z oTraining C dis , for z u Classification is performed so that the predicted probabilities follow a uniform distribution. The task-independent feature encoder is optimized using the loss function shown in Equation (1), where E C This represents the class probability that follows a uniform distribution.

[0014]

[0015]

[0016] Furthermore, the feature constraint module utilizes image reconstruction loss to ensure that task-oriented features retain rich task-related details, thus improving the quality of task-oriented features in the source domain. Task-irrelevant features Use the corresponding encoder D s Reconstructed image As shown in formula (2), similarly, the task-oriented features of the target domain... Task-irrelevant features Encoder D of the target domain t Obtain the reconstructed image

[0017]

[0018] Image reconstruction loss L rec Perceptual loss is used, as shown in Equation (3). Perceptual loss is usually obtained by extracting features of the image using a pre-trained VGG network and calculating the difference at the feature level, as shown in Equation (4).

[0019] Where, ψ (l) N represents the l-th convolutional layer in the VGG network. (l) w represents the number of convolutional layers. (l) This represents the weight of the corresponding layer l;

[0020]

[0021]

[0022] Furthermore, given an input image x and a domain label d, the domain label encoding module obtains aligned task-oriented features as shown in formula (5), where f i (·), i = 1, 2 represent the process used for feature extraction, and g(·) represents the process of domain label encoding;

[0023] z o =DLEM(x,d)=f2(f1(x)+g(d)) (5)

[0024] To fully utilize the latent features encoded by domain labels and eliminate domain relevance of task-oriented features, a semantic consistency loss L relative to the domain is added to the image translation process. sem First, in the image translation part, task-oriented features and task-independent features from different domains can be used by the decoder to obtain the translated image. and As shown in formula (6):

[0025]

[0026] Utilizing structural loss Texture loss and combating losses The image translation loss consists of three parts that optimize image translation performance; structural loss. This is achieved by calculating the perceptual loss between the translated image and the original image; texture loss. After extracting image features using the VGG network, the differences between each channel are calculated separately.

[0027] The process of calculating texture feature differences is shown in formula (7), where C (l) w represents the number of channels in layer l of a VGG network. (l) μ represents the weight corresponding to the channel. c (·) represents the average activation value of channel c;

[0028]

[0029] The calculation process of adversarial loss is shown in formula (8): where TOFD uses the image classifier C trained on the original image. img As a discriminant, a and b represent the domain labels of different domains. This represents the overall image translation loss when translating an image from domain a to domain b. As shown in formula (9):

[0030]

[0031]

[0032] During training, the original task-oriented features are used to train the classifier C. sem Using C sem The domain label encoding module extracts task-oriented features from the translated image. and Then, the domain labels are classified to obtain the probability distribution shown in formula (10): where,

[0033]

[0034]

[0035] Based on this distribution, semantic consistency loss can be used. The quantification of field invariance is achieved as shown in formula (11), where d s and d t Tags representing the source and target domains respectively:

[0036]

[0037] The advantages and beneficial effects of this invention are as follows:

[0038] 1. The feature constraint module of the present invention can improve the decoupling method of structure-texture. Based on the maximum entropy principle, from the perspective of task relevance, the domain-invariant features are expanded into task-oriented features with strong task relevance, so that the task-oriented features contain rich detailed information and improve the segmentation accuracy of nodules.

[0039] 2. The domain label encoding module of the present invention can learn the latent feature representation of domain bias by using domain labels containing domain-specific information, and combine the latent features with task-oriented features to improve the stability of detailed features, thereby improving the domain invariance of task-oriented features, so that the segmentation model can more accurately distinguish between benign and malignant nodules. Attached Figure Description

[0040] Figure 1 This is a flowchart of the task-oriented feature decoupling method of the present invention;

[0041] Figure 2 This is a structural diagram of the feature constraint module of the present invention;

[0042] Figure 3 This is a structural diagram of the domain label encoding module of the present invention. Detailed Implementation

[0043] The present invention will be further described in detail below through specific embodiments. The following embodiments are merely descriptive and not limiting, and should not be used to limit the scope of protection of the present invention.

[0044] An unsupervised adaptive semantic segmentation method for thyroid ultrasound images based on feature decoupling is innovative in that it is a task-oriented feature decoupling (TOFD) method, comprising two parts: feature decoupling and image translation, representing the source domain image as... Its pixel-level label is y s ∈{0,1} H×W×cThe target domain image is represented as The image encoder is represented by E, the image decoder by D, and the classification head for acquiring the segmentation map by C. seg .

[0045] Furthermore, the feature decoupling part utilizes the Domain Label Encoding Module (DLEM) to extract task-oriented features from the source domain. and the task-oriented characteristics of the target domain Through domain-specific encoders and Extract domain-specific task-independent features separately and The Feature Constraint Module (FCM) is used to reduce the impact of task-related constraints. and Task relevance, through image reconstruction constraints and They are complementary. and They are complementary, making and It can retain richer task-related details, thus enabling the identification of nodule edges and calcification information during segmentation; through C seg get Segmentation results Combined with label y s Optimize the field label encoding module, utilizing C seg Obtain the segmentation result Achieve output space alignment, thereby driving and Alignment;

[0046] The image translation component combines task-oriented and task-independent features from different domains to obtain a new image. and and source domain tag y s As The label optimization model is improved, and a semantic classifier C is added. sem verify and The domain invariance is used to optimize the domain label encoding process in the domain label encoding module.

[0047] Furthermore, the feature constraint module utilizes task-related constraints based on the maximum entropy principle to prevent task-irrelevant features from retaining task-related details, and uses image reconstruction constraints to encourage task-oriented features to retain this information; specifically, the task-oriented feature is represented as z. oThe task-independent feature is z u In the task-related constraints, the feature constraint module utilizes z o Training C dis , for z u Classification is performed so that the predicted probabilities follow a uniform distribution. The task-independent feature encoder is optimized using the loss function shown in Equation (1), where E C This represents the class probability that follows a uniform distribution.

[0048]

[0049]

[0050] Furthermore, the feature constraint module utilizes image reconstruction loss to ensure that task-oriented features retain rich task-related details, thus improving the quality of task-oriented features in the source domain. Task-irrelevant features Use the corresponding encoder D s Reconstructed image As shown in formula (2), similarly, the task-oriented features of the target domain... Task-irrelevant features Encoder D of the target domain t Obtain the reconstructed image

[0051]

[0052] Image reconstruction loss L rec Perceptual loss is used, as shown in Equation (3). Perceptual loss is usually obtained by extracting features of the image using a pre-trained VGG network and calculating the difference at the feature level, as shown in Equation (4).

[0053] Where, ψ (l) N represents the l-th convolutional layer in the VGG network. (l) w represents the number of convolutional layers. (l) This represents the weight of the corresponding layer l;

[0054]

[0055]

[0056] Furthermore, given an input image x and a domain label d, the domain label encoding module obtains aligned task-oriented features as shown in formula (5), where f i (·), i = 1, 2 represent the process used for feature extraction, and g(·) represents the process of domain label encoding;

[0057] z o =DLEM(x,d)=f2(f1(x)+g(d)) (5)

[0058] To fully utilize the latent features encoded by domain labels and eliminate domain relevance of task-oriented features, a semantic consistency loss L relative to the domain is added to the image translation process. sem First, in the image translation part, task-oriented features and task-independent features from different domains can be used by the decoder to obtain the translated image. and As shown in formula (6):

[0059]

[0060] Utilizing structural loss Texture loss and combating losses The image translation loss consists of three parts that optimize image translation performance; structural loss. This is achieved by calculating the perceptual loss between the translated image and the original image; texture loss. After extracting image features using the VGG network, the differences between each channel are calculated separately.

[0061] The process of calculating texture feature differences is shown in formula (7), where C (l) w represents the number of channels in layer l of a VGG network. (l) μ represents the weight corresponding to the channel. c (·) represents the average activation value of channel c;

[0062]

[0063] The calculation process of adversarial loss is shown in formula (8): where TOFD uses the image classifier C trained on the original image. img As a discriminant, a and b represent the domain labels of different domains. This represents the overall image translation loss when translating an image from domain a to domain b. As shown in formula (9):

[0064]

[0065]

[0066] During training, the original task-oriented features are used to train the classifier C. sem Using C sem The domain label encoding module extracts task-oriented features from the translated image. and Then, the domain labels are classified to obtain the probability distribution shown in formula (10): where,

[0067]

[0068]

[0069] Based on this distribution, semantic consistency loss can be used. The quantification of field invariance is achieved as shown in formula (11), where d s and d t Tags representing the source and target domains respectively:

[0070]

[0071] TOFD can be trained using labeled source domain data and unlabeled target domain data, improving the model's generalization ability. This reduces the impact of domain offset on the model's segmentation performance when segmenting ultrasound images in the target domain. When predicting images, the model's runtime is only 30ms, enabling accurate and real-time nodule segmentation of unlabeled target domain images.

[0072] The TOFD method, after training the model through the above process, does not add any additional parameters or runtime to the segmentation model. The resulting segmentation model exhibits good segmentation performance in both the target and source domain images. The evaluation metrics for the segmentation model are the Intersections over Union (IOU) and the average IoU for each class.

[0073] This invention was tested on a multi-model thyroid ultrasound image dataset. Images acquired by the P2 model were used as the source domain, and images acquired by the P1, P3, T1, and T2 models were used as the target domains. The experimental results are shown in Tables 1 to 4. The comparison shows that the segmentation effect of this invention on nodules is higher than that of other methods.

[0074] Table 1 shows images acquired by the P1 camera, with the target area defined as the target region.

[0075] method Background IoU IoU of malignant nodules benign nodule IoU mIoU direct migration 97.4 55.0 68.6 73.7 DISE 97.8 59.3 74.0 77.0 TOFD (This invention) 97.8 63.5 74.0 78.5

[0076] Table 2 shows images acquired by the P3 camera, with the target area defined as the target region.

[0077] method Background IoU IoU of malignant nodules benign nodule IoU mIoU direct migration 97.6 23.3 72.3 64.4 DISE 98.5 54.9 79.1 77.5 TOFD (This invention) 98.6 59.2 78.1 78.7

[0078] Table 3 shows images acquired by the T1 camera, with the target area defined as the target region.

[0079] method Background IoU IoU of malignant nodules benign nodule IoU mIoU direct migration 96.1 41.2 56.9 64.7 DISE 97.9 64.7 61.3 74.5 TOFD (This invention) 98.1 64.2 73.6 78.6

[0080] Table 4 shows images acquired by the T2 camera, with the target area defined as the target region.

[0081] method Background IoU IoU of malignant nodules benign nodule IoU mIoU direct migration 96.2 26.3 55.8 59.4 DISE 97.8 58.6 72.6 76.3 TOFD (This invention) 98.6 62.0 73.9 77.8

[0082] Although embodiments and drawings of the present invention have been disclosed for illustrative purposes, those skilled in the art will understand that various substitutions, variations and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. An unsupervised adaptive semantic segmentation method for thyroid ultrasound images based on feature decoupling, characterized in that: The method described is a task-oriented feature decoupling method, comprising two parts: feature decoupling and image translation, representing the source domain image as... Its pixel-level tags are The target domain image is represented as The image encoder is represented as The image decoder is represented as The classification header representation of the segmentation map is obtained as follows: ; The feature decoupling part uses a domain label encoding module to extract task-oriented features from the source domain. and the task-oriented characteristics of the target domain Through domain-specific encoders and Extract domain-specific task-independent features separately and The feature constraint module reduces the time required by task relevance constraints. and Task relevance, through image reconstruction constraints and They are complementary. and They are complementary, making and It can retain richer task-related details, thus enabling the identification of nodule edges and calcification information during segmentation; through get Segmentation results Combine tags Optimize the domain label encoding module and utilize Obtain the segmentation result Achieve output space alignment, thereby driving and Alignment; The image translation component combines task-oriented and task-independent features from different domains to obtain a new image. and and source domain tag As The label optimization model was improved, and a semantic classifier was added. verify and The domain invariance is used to optimize the domain label encoding process in the domain label encoding module.

2. The unsupervised adaptive semantic segmentation method for thyroid ultrasound images based on feature decoupling according to claim 1, characterized in that: The feature constraint module utilizes task-related constraints based on the maximum entropy principle to prevent task-irrelevant features from retaining task-related details, and uses image reconstruction constraints to encourage task-oriented features to retain this information; specifically, task-oriented features are represented as follows: Task-independent features are In task-related constraints, the feature constraint module utilizes... Training ,right Classification is performed so that the predicted probabilities follow a uniform distribution. The task-independent feature encoder is optimized using the loss function shown in formula (1), where, This represents the class probability that follows a uniform distribution. ； （1） Furthermore, the feature constraint module utilizes image reconstruction loss to ensure that task-oriented features retain rich task-related details, thus improving the quality of task-oriented features in the source domain. Task-irrelevant features Use the corresponding encoder Reconstructed image As shown in formula (2), similarly, the task-oriented features of the target domain... Task-irrelevant features Encoder using the target domain Obtain the reconstructed image : ； （2） Image reconstruction loss Perceptual loss is used, as shown in formula (3). Perceptual loss is usually obtained by extracting features of the image using a pre-trained VGG network and calculating the difference at the feature level, as shown in formula (4). in, Indicates the first in the VGG network Layered convolution, Indicates the number of convolutional layers. Represents corresponding Layer weights; ； （3） ； （4）。 3. The unsupervised adaptive semantic segmentation method for thyroid ultrasound images based on feature decoupling according to claim 1, characterized in that: The domain label encoding module provides a representation of the input image. Domain tags The process of obtaining the aligned task-oriented features is shown in formula (5), where, These represent the processes used for feature extraction. This indicates the process of encoding field labels; ； （5） To fully utilize the latent features encoded by domain labels and eliminate domain relevance of task-oriented features, a semantic consistency loss relative to the domain is added to the image translation process. First, in the image translation part, task-oriented features and task-independent features from different domains can be used by the decoder to obtain the translated image. and As shown in formula (6): ； （6） Utilizing structural loss Texture loss and combating losses The image translation loss consists of three parts that optimize image translation performance; structural loss. This is achieved by calculating the perceptual loss between the translated image and the original image; texture loss. After extracting image features using the VGG network, the differences between each channel are calculated separately. The process of calculating texture feature differences is shown in formula (7), where, Representing the VGG network Number of channels in the layer This represents the weight corresponding to the channel. Indicates channel The average activation value; ； （7） The calculation process of adversarial loss is shown in formula (8): where TOFD utilizes the image classifier trained on the original image. As a discriminator and Domain tags that represent different domains Indicates to transferring the image from Domain translation Domain, overall image translation loss As shown in formula (9): ； （8） ； （9） During training, the original task-oriented features are used to train the classifier. ;use The domain label encoding module extracts task-oriented features from the translated image. and Then, the domain labels are classified to obtain the probability distribution shown in formula (10): where, ； ； （10） Based on this distribution, semantic consistency loss can be used. The quantification of field invariance is achieved as shown in formula (11), where and Tags representing the source and target domains respectively: ；（11）。

Images