Semi-supervised dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency

Abstract

The present invention discloses a semi-supervised dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency, comprising the following steps: Step 1, obtaining a spatial domain weakly enhanced image, a spatial domain strongly enhanced image and a frequency domain strongly enhanced data based on an unlabeled image; Step 2, constructing a dual-domain branched semi-supervised learning network; Step 3, obtaining a final dermoscopic image segmentation network; Step 4, inputting the dermoscopic image to be segmented into the final dermoscopic image segmentation network to obtain a segmented lesion mask. The frequency domain strong enhancement method proposed in the present invention can effectively enhance the edge of skin lesions, which helps the segmentation model to perceive the edge of the lesion. Furthermore, the present invention proposes a learning method based on the spatial domain and frequency domain dual-domain consistency to further enhance the segmentation model's perception accuracy and robustness of skin lesions.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a semi-supervised dermatoscope image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency. Background Art

[0002] Current fully supervised medical image segmentation methods based on deep learning rely heavily on manual annotation by physicians. Unlike natural images, medical image segmentation requires specialized medical domain knowledge. Furthermore, unlike relatively consistent anatomical structures across cases, dermoscopic lesions exhibit more complex morphology, blurred edges, and irregular shapes. Consequently, the cost of labeling dermoscopic lesions is extremely high. To reduce this expensive labeling cost, semi-supervised methods have emerged. These methods aim to closely approach the performance of fully supervised methods by leveraging a small amount of labeled data with a large amount of unlabeled data. Currently, the most mainstream semi-supervised methods are based on consistency regularization techniques. The latest and most advanced consistency regularization methods leverage strong and weak consistency in the spatial domain to fully exploit the semantic information hidden in unlabeled data. This so-called strong and weak consistency in the spatial domain involves first applying weak augmentation (such as flipping or rotating) to an input image, slightly altering the image's content. Then, strong augmentation (such as Gaussian blurring or color dithering) is applied to the weakly enhanced image, which more aggressively alters the image's context. While strong and weak consistency methods demonstrate strong performance, they fail to fully capture the edge information between normal skin tissue and lesions in dermoscopic images. For example, Gaussian blurring, while lesion edge information represents high-frequency information in an image, is essentially a low-pass filter. Therefore, using Gaussian blurring to enhance an image results in a loss of lesion edge information, resulting in the segmentation model failing to discern lesion edges, negatively impacting the segmentation of skin lesions. Summary of the Invention

[0003] To solve the above problems, the present invention discloses a semi-supervised dermatoscope image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency.

[0004] To achieve the above object, the technical solution of the present invention is:

[0005] A semi-supervised dermatoscope image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency includes the following steps:

[0006] Step 1: For the unlabeled image x u Perform weak enhancement in the spatial domain to obtain Weakly enhanced images in the spatial domain Perform strong enhancement to obtain a strongly enhanced image in the spatial domain At the same time, for the unlabeled image x uPerform frequency domain strong enhancement to obtain frequency domain strong enhanced image

[0007] Step 2: Construct a dual-domain branch semi-supervised learning network. The two branches of the dual-domain branch semi-supervised learning network are both neural networks with the same structure, and one of the branches inputs a weakly enhanced image in the spatial domain. And the spatial domain is strongly enhanced to obtain It is called the spatial domain branch SDB; the other branch input data is the spatial domain weakly enhanced image and frequency domain strongly enhanced image It is called frequency domain branch FDB;

[0008] Step 3: As the input data of the spatial domain branch SDB, The frequency domain branch FDB is used as the input data for training to obtain a trained dual-domain branch semi-supervised learning network. The branch network with the best validation set effect in the trained dual-domain branch semi-supervised learning network is used as the final dermoscopic image segmentation network;

[0009] Step 4: Input the dermoscopic image to be segmented directly into the final dermoscopic image segmentation network to obtain the segmented lesion image.

[0010] As a further improvement, in step 1, the weak enhancement method is to rotate or flip the image; and the strong enhancement method is to perform Gaussian blur or color dithering on the image.

[0011] As a further improvement, in step 2, the neural network is a UNet network.

[0012] Further improvement, the frequency domain strong enhancement data The method of obtaining is as follows:

[0013] For an unlabeled image x u Perform discrete wavelet transform to obtain low-frequency image and high-frequency images in the horizontal direction High-frequency image in the vertical direction And the high-frequency image in the diagonal direction

[0014]

[0015] DWT( ) represents discrete wavelet transform; then the low-frequency image High-frequency image in the horizontal direction High-frequency image in the vertical direction And the high-frequency image in the diagonal direction Randomly adjust the brightness and saturation:

[0016]

[0017] Where RM( ) represents a random regulator; They are and The image obtained after random adjustment; and Then perform inverse wavelet transform to obtain the inverse transformed image

[0018]

[0019] iDWT( ) represents inverse wavelet transform; take the inverse transform image Strongly enhanced image in spatial domain The absolute value of the residual is used as the extracted edge information

[0020]

[0021] Using the extracted edge information Further enhance the edge information of the lesion to obtain strong frequency domain enhancement data

[0022]

[0023] Further improvement, the total loss function L of the dual-domain branch semi-supervised learning network is:

[0024] L=L sup +L unsup +L con (6)

[0025] Among them, L sup represents the supervised loss function of the dual-domain branch semi-supervised learning network, L unsup represents the unsupervised loss function of the dual-domain branch semi-supervised learning network, L con Represents the spatial-frequency domain consistency loss function of a dual-domain branch semi-supervised learning network.

[0026] Further improvement, the unsupervised loss function L of the dual-domain branch semi-supervised learning network unsup The method of obtaining is as follows:

[0027] The output of the spatial domain branch SDB is The output of the frequency domain branch FDB is in is the output corresponding to the spatial domain weak enhancement data of the spatial domain branch, is the output corresponding to the strongly enhanced data of the spatial branch space, is the output corresponding to the weakly enhanced data in the frequency domain branch and the spatial domain, is the output corresponding to the frequency domain branch frequency domain strong enhancement data; then the unique hot encoding method is used to respectively and Convert to pseudo labels and

[0028] The unsupervised loss functions of the spatial domain branch SDB and the frequency domain branch FDB are as follows:

[0029]

[0030] Among them, CE() is the cross entropy loss function, and Dice() is the Dice loss function. The pseudo labels generated for the output of the weakly enhanced input data of the frequency domain branch space are: The pseudo labels generated for the spatial domain branch spatial weak enhancement input data corresponding to the output, is the unsupervised loss function for the spatial domain branch, is the unsupervised loss function of the frequency domain branch;

[0031] Using the pseudo label output by the frequency domain branch FDB To supervise the spatial domain branch SDB. Vice versa, the pseudo label output by the spatial domain branch SDB is used To supervise the frequency domain branch FDB, the unsupervised loss function L for dual domain consistency unsup for:

[0032]

[0033] Further improvements, Among them, L con ( ) is the consistency constraint loss function in the spatial and frequency domains; MSE() represents the minimum mean square error.

[0034] Further improvements, L sup The calculation of is as follows:

[0035] The supervised loss functions of the spatial domain branch SDB and the frequency domain branch FDB are as follows:

[0036]

[0037] y l is x l The label, the superscript l indicates that there is labeled data, and The output of the labeled data corresponding to the spatial domain branch SDB and the frequency domain branch FDB; Indicates the output corresponding to the spatial domain branch with labeled data input, Indicates the output corresponding to the frequency domain branch with labeled data input; In addition, the labeled data x l Randomly sampled from the training set, the rest of the data in the training set is regarded as unlabeled data xu.

[0038] The supervised loss function of the dual-domain branch semi-supervised learning network is:

[0039]

[0040] The beneficial effects brought about by the technical solution of the present invention are:

[0041] The frequency domain strong enhancement method proposed in this paper can effectively enhance the edges of skin lesions, helping the segmentation model to perceive the lesion edges. Furthermore, this paper proposes a learning method based on the consistency of the spatial and frequency domains to further enhance the segmentation model's perception and robustness of skin lesions. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of the present invention, where (a) is the frequency domain random adjustment schematic part, (b) is the lesion edge extraction and edge information enhancement part. (c) is the dual-domain consistency learning part. Note: FDB refers to the frequency domain branch (Frequency Domain Branch), SDB (Spatial Domain Branch), Refers to the spatial domain weak enhancement image (Spatial WeakPerturbation, SWP), Refers to Spatial Strong Perturbation (SSP), Refers to frequency domain strongly enhanced data (FSP), and the superscript u refers to unlabeled data. For the sake of simplicity and intuitiveness, only the unlabeled data learning part is shown.

[0043] Figure 2 This is the visualization result of the ISIC 2017 dataset with 1% annotation amount;

[0044] Figure 3 This is the visualization result of the ISIC 2017 dataset with 5% annotation.

[0045] Figure 4 This is the visualization result of the ISIC 2018 dataset with 1% annotation amount;

[0046] Figure 5Visualization results for the ISIC 2018 dataset with 5% annotation;

[0047] Figure 6 Visualization results for the HAM10000 dataset with 1% annotation. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0049] like Figure 1 As shown, first, we input the unlabeled image x u Perform Discrete Wavelet Transform (DWT) to obtain a low-frequency image and level vertical and diagonal lines High frequency image of direction:

[0050]

[0051] Then the brightness, saturation, etc. of the above four images are randomly adjusted:

[0052]

[0053] In the above formula, They are the randomly adjusted low-frequency and high-frequency images, and then the inverse transformation is obtained:

[0054]

[0055] At this time, there is a huge difference between the inverse transformed image and its spatially enhanced image. Figure 1 (a), further, take the absolute value of the residual between the inverse transformed image and the spatial domain strongly enhanced image:

[0056]

[0057] The absolute value of the residual is used as the edge extraction. Figure 1 As shown in (b), using the extracted edge information The subscript "mut" means mutation (i.e., the mutation point between the lesion and normal tissue), which further enhances the edge information of the lesion:

[0058]

[0059] At this point, there are three versions of enhanced images: spatial domain weak enhancement Strong spatial enhancement Frequency domain strong enhancement To this end, this method designs a dual-domain branch semi-supervised learning method. Figure 1 As shown in (c), one branch is the spatial domain branch SDB, and the other branch is the frequency domain branch FDB. As the input image of SDB, As the input image of FDB, the corresponding output of SDB is The output of FDB is represented as Using one-hot encoding technology and Convert to pseudo label, that is and Then the SDB and FDB unsupervised loss functions are as follows:

[0060]

[0061] In the above formula, CE is the cross entropy loss function, and Dice is the Dice loss function. The pseudo labels output by SDB are used to supervise the strong enhancement output of FDB. Similarly, the pseudo labels output by FDB are used to supervise the learning of SDB. Then, the unsupervised loss function for dual-domain consistency is:

[0062]

[0063] In the present invention, the dual-domain consistency constraints proposed are:

[0064]

[0065] In the above formula, L con For consistency constraints, the minimum mean square error MSE is used for measurement. l The supervised loss functions of SDB and FDB are as follows:

[0066]

[0067] In the above formula, y l is x l The label, the superscript l indicates labeled data (labeled), and is the output of the labeled data corresponding to SDB and FDB respectively. Similarly, the supervised loss function of the dual-domain consistency framework is:

[0068]

[0069] Finally, the loss function of the entire dual-domain consistency learning framework is:

[0070] L=L sup +L unsup +L con (12).

[0071] To verify the effectiveness of this invention, the open-source dermatoscope datasets ISIC 2017, ISIC 2018, and HAM10000 were used for full validation. Regarding the comparison method, the proposed method was compared with eight methods published in top journals and conferences in the past three years: DTC (AAAI'21), SSNet (MICCAI'22), URPC (MedIA'22), BSNet (TMI'23), BCPNet (CVPR'23), AC-MT (MedIA'23), CauSSL (ICCV'23), and UniMatch (CVPR'23). All comparative experiments used the same UNet network for comparison. To ensure fair comparison, the proposed method used single-branch SDB (same UNet architecture) for comparison.

[0072] Verification indicators: Referring to the above comparison method, the method of the present invention also uses Dice (%) coefficient, Jaccard (%) coefficient, Hausdorff Distance (HD95) and Average Surface Distance (ASD) as quantitative indicators to evaluate the performance of the method.

[0073] Comparative experiment on ISIC 2017: The ISIC 2017 dataset consists of 2000 training images, 150 validation images, and 600 test images. We randomly sampled 1% (20 images), 5% (100 images), and 10% (200 images) of the images as labeled data, and the remaining training images as unlabeled data. We repeated the random sampling three times and took the average and standard deviation of the test results:

[0074] Table 1. ISIC 2017 dataset, 1% annotation amount, three repeated sampling experiment results

[0075]

[0076] Table 2. ISIC 2017 dataset, 5% annotation amount, three repeated sampling experimental results

[0077]

[0078] Table 3. ISIC 2017 dataset, 10% annotation amount, three repeated sampling experimental results

[0079]

[0080] In the above three tables, UNet (Fully Supervised, FS) is a fully supervised method, that is, it uses all the data and labels in the training set for training. Similarly, the visualization results of the ISIC 2017 experiment are as follows Figure 2 and Figure 3 shown.

[0081] Experiments on the ISIC 2018 dataset: The ISIC 2018 dataset consists of 2594 training images, 100 validation images, and 1000 test images. We randomly sampled 1% (25 images), 5% (129 images), and 10% (259 images) of the images as labeled data, and the remaining training images as unlabeled data. We repeated the random sampling three times and took the average and standard deviation of the test results:

[0082] Table 4. ISIC 2018 dataset, 1% annotation amount, three repeated sampling experimental results

[0083]

[0084] Table 5. ISIC 2018 dataset, 5% annotation amount, three repeated sampling experimental results

[0085]

[0086] Table 6. ISIC 2018 dataset, 10% annotation amount, three repeated sampling experimental results

[0087]

[0088] Regarding the HAM10000 dataset experiments: HAM10000 contains 10,015 dermoscopic images. 6,000 images were randomly sampled as the training set, 1,015 as the validation set, and 3,000 as the test set. Then, 1% (60 images) of the training set were randomly sampled as labeled data, and the remaining training set images were used as unlabeled data. This sampling process was repeated three times, and the average and standard deviation of the results were taken. Given that the UniMatch method exhibited highly competitive performance, only UniMatch was compared in the HAM10000 dataset comparison experiments.

[0089]

[0090] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and the embodiments. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A semi-supervised dermatoscope image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency, characterized by: The steps include: Step 1: For the unlabeled image x u Perform weak enhancement in the spatial domain to obtain Weakly enhanced images in the spatial domain Perform spatial domain strong enhancement to obtain spatial domain strong enhanced image At the same time, for the unlabeled image x u Perform frequency domain strong enhancement to obtain frequency domain strong enhanced image Step 2: Construct a dual-domain branch semi-supervised learning network. The two branches of the dual-domain branch semi-supervised learning network are both neural networks with the same structure, and one of the branches inputs a weakly enhanced image in the spatial domain. And the spatial domain is strongly enhanced to obtain It is called the spatial domain branch SDB; the other branch input data is the spatial domain weakly enhanced image and frequency domain strongly enhanced image It is called frequency domain branch FDB; Step 3: As the input data of the spatial domain branch SDB, The frequency domain branch FDB is used as the input data for training to obtain a trained dual-domain branch semi-supervised learning network. The branch network with the best validation set effect in the trained dual-domain branch semi-supervised learning network is used as the final dermoscopic image segmentation network; Step 4: Input the dermoscopic image to be segmented directly into the final dermoscopic image segmentation network to obtain the segmented lesion image.

2. The dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency according to claim 1, characterized in that: In the step 1, the weak enhancement method is to rotate or flip the image; and the spatial domain strong enhancement method is to perform Gaussian blur or color dithering on the image.

3. The dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency according to claim 1, characterized in that: In the step 2, the neural network is a UNet network.

4. The dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency according to claim 1, characterized in that: The frequency domain strongly enhanced image The method of obtaining is as follows: For an unlabeled image x u Perform discrete wavelet transform to obtain low-frequency image and high-frequency images in the horizontal direction High-frequency image in the vertical direction And the high-frequency image in the diagonal direction DWT() represents discrete wavelet transform; then the low-frequency image High-frequency image in the horizontal direction High-frequency image in the vertical direction And the high-frequency image in the diagonal direction Randomly adjust the brightness and saturation: Where RM() represents a random regulator; They are and The image obtained after random adjustment; and Then perform inverse wavelet transform to obtain the inverse transformed image iDWT() means inverse wavelet transform; get the inverse transform image Strongly enhanced image in spatial domain The absolute value of the residual is used as the extracted edge information Using the extracted edge information Obtain frequency domain strong enhancement data 5. The dermatoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency according to claim 1, characterized in that: The total loss function L of the dual-domain branch semi-supervised learning network is: L=L sup +L unsup +L con Among them, L sup represents the supervised loss function of the dual-domain branch semi-supervised learning network, L unsup represents the unsupervised loss function of the dual-domain branch semi-supervised learning network, L con Represents the spatial-frequency domain consistency loss function of a dual-domain branch semi-supervised learning network.

6. The dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency according to claim 5, characterized in that: Unsupervised loss function L for dual-domain branch semi-supervised learning network unsup The method of obtaining is as follows: the output of the spatial domain branch SDB is The output of the frequency domain branch FDB is in is the output corresponding to the spatial domain weak enhancement data of the spatial domain branch, is the output corresponding to the strongly enhanced data of the spatial branch space, is the output corresponding to the weakly enhanced data in the frequency domain branch and the spatial domain, is the output corresponding to the frequency domain branch frequency domain strong enhancement data; then the unique hot encoding method is used to respectively and Convert to pseudo labels and The unsupervised loss functions of the spatial domain branch SDB and the frequency domain branch FDB are as follows: Among them, CE() is the cross entropy loss function, Dice() is the Dice loss function, The pseudo labels generated for the output of the weakly enhanced input data of the frequency domain branch space are: The pseudo labels generated for the spatial domain branch spatial weak enhancement input data corresponding to the output, is the unsupervised loss function for the spatial domain branch, is the unsupervised loss function of the frequency domain branch; Using the pseudo label output by the frequency domain branch FDB To supervise the spatial domain branch SDB, use the pseudo label output by the spatial domain branch SDB To supervise the frequency domain branch FDB, the unsupervised loss function L for dual domain consistency unsup for:

7. The dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency according to claim 6, characterized in that: Among them, L con () is the consistency constraint loss function in the spatial and frequency domains; MSE() represents the minimum mean square error.

8. The dermatoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency according to claim 6, characterized in that: L sup The calculation is as follows: The supervised loss functions of the spatial domain branch SDB and the frequency domain branch FDB are as follows: y l For label data x l The label, the superscript l indicates that there is labeled data, and The output of the labeled data corresponding to the spatial domain branch SDB and the frequency domain branch FDB; Indicates that there is labeled data input to the output corresponding to the spatial domain branch, Indicates the output corresponding to the frequency domain branch with labeled data input; the labeled data x l Randomly sampled from the training set, the rest of the training set data is considered as unlabeled data x u ; The supervised loss function of the dual-domain branch semi-supervised learning network is:

Images