Semi-supervised dermatoscope image segmentation method based on frequency domain disturbance and space-frequency dual-domain consistency

Abstract

The invention discloses a semi-supervised dermatoscope image segmentation method based on frequency domain disturbance and space-frequency dual-domain consistency. The method comprises the following steps: step 1, obtaining a spatial domain weakly enhanced image, a spatial domain strongly enhanced image and frequency domain strongly enhanced data according to an unlabeled image; step 2, constructing a double-domain branch semi-supervised learning network; step 3, obtaining a final dermatoscope image segmentation network; and step 4, inputting the dermatoscope image to be segmented into the final dermatoscope image segmentation network to obtain a segmented focus mask. The frequency domain strong enhancement method provided by the invention can effectively enhance the skin lesion edge, and facilitates the perception of the segmentation model on the lesion edge. Furthermore, the invention provides a learning method based on the consistency of a space domain and a frequency domain, and the perception accuracy and robustness of the segmentation model to the skin focus are further enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a semi-supervised dermatoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency. Background Art

[0002] The current fully supervised medical image segmentation methods based on deep learning rely heavily on manual annotation by doctors. Unlike natural images, medical image segmentation requires professional knowledge in the medical field. In addition, unlike the relatively consistent anatomical structures across cases, dermoscopic lesions show more complex morphology, blurred edges, and irregular shapes. Therefore, the annotation cost of dermoscopic lesions is extremely high. In order to reduce the above expensive annotation costs, semi-supervised methods have emerged. Semi-supervised methods aim to use a small amount of labeled data and a large amount of unlabeled data to approach the performance of fully supervised methods as much as possible. The most mainstream semi-supervised method is based on consistency regularization technology, and the latest and most advanced consistency regularization method is based on the strong and weak consistency of the spatial domain to fully explore the semantic information hidden in the unlabeled data. The so-called strong and weak consistency in the spatial domain means that an input image is first weakly enhanced (such as flipping, rotating, etc.) to slightly change the content presented by the image. Then, the weakly enhanced image is strongly enhanced (such as Gaussian blurring, color jittering), which is a relatively more radical change in the contextual information in the image. Although the strong and weak consistency methods have shown strong performance, they have not been able to fully mine the edge information of normal skin tissue and lesions when facing dermoscopic images. Taking Gaussian blur as an example, the edge information of lesions belongs to the high-frequency information part of the image, and Gaussian blur is essentially a low-pass filter. Therefore, using the Gaussian blur method to enhance the image will cause the lesion edge information to be lost, resulting in the segmentation model failing to keenly perceive the lesion edge, which has a negative effect on the segmentation of skin lesions. Summary of the invention

[0003] To solve the above problems, the present invention discloses a semi-supervised dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency.

[0004] To achieve the above object, the technical solution of the present invention is:

[0005] A semi-supervised dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency comprises the following steps:

[0006] Step 1: For the unlabeled image x u Perform weak enhancement in the spatial domain to obtain Weakly enhanced images in the spatial domain Perform strong enhancement to obtain a strongly enhanced image in the spatial domain At the same time, for the unlabeled image x uPerform frequency domain strong enhancement to obtain frequency domain strong enhanced image

[0007] Step 2: Construct a dual-domain branch semi-supervised learning network. The two branch networks of the dual-domain branch semi-supervised learning network are both neural networks with the same structure, and one of the branches inputs a weakly enhanced image in the spatial domain. And the spatial domain is strongly enhanced to obtain It is called the spatial domain branch SDB; the other branch input data is the spatial domain weakly enhanced image And frequency domain strong enhanced image It is called frequency domain branch FDB;

[0008] Step 3: As the input data of the spatial domain branch SDB, The frequency domain branch FDB is used as the input data for training to obtain a trained dual-domain branch semi-supervised learning network, and the branch network with the best verification set effect in the trained dual-domain branch semi-supervised learning network is used as the final dermoscopic image segmentation network;

[0009] Step 4: directly input the dermoscopic image to be segmented into the final dermoscopic image segmentation network to obtain the segmented lesion image.

[0010] As a further improvement, in step 1, the weak enhancement method is to rotate or flip the image; and the strong enhancement method is to perform Gaussian blur or color dithering on the image.

[0011] As a further improvement, in step 2, the neural network is a UNet network.

[0012] Further improvement, the frequency domain strongly enhanced data The method of obtaining is as follows:

[0013] For an unlabeled image x u Perform discrete wavelet transform to obtain low-frequency image And the high-frequency image in the horizontal direction High-frequency image in vertical direction And the high-frequency image in the diagonal direction

[0014]

[0015] DWT ( ) represents discrete wavelet transform; then the low-frequency image High-frequency image in horizontal direction High-frequency image in vertical direction And the high-frequency image in the diagonal direction The brightness and saturation are randomly adjusted:

[0016]

[0017] Where RM( ) represents a random regulator; They are and The image obtained after random adjustment; and Then perform inverse wavelet transform to obtain the inverse transformed image

[0018]

[0019] iDWT( ) stands for inverse wavelet transform; take the inverse transform image Strongly enhanced image in spatial domain The absolute value of the residual is used as the extracted edge information

[0020]

[0021] Using the extracted edge information Further enhance the edge information of the lesion to obtain strong frequency domain enhancement data

[0022]

[0023] For further improvement, the total loss function L of the dual-domain branch semi-supervised learning network is:

[0024] L=L sup +L unsup +L con (6)

[0025] Among them, L sup represents the supervised loss function of the dual-domain branch semi-supervised learning network, L unsup represents the unsupervised loss function of the dual-domain branch semi-supervised learning network, L con Represents the spatial-frequency domain consistency loss function of a dual-domain branch semi-supervised learning network.

[0026] Further improvement, the unsupervised loss function L of the dual-domain branch semi-supervised learning network unsup The method of obtaining is as follows:

[0027] The output of the spatial domain branch SDB is The output of the frequency domain branch FDB is in is the output corresponding to the spatial domain weak enhancement data of the spatial domain branch, is the output corresponding to the spatial domain branch space strong enhancement data, is the output corresponding to the weakly enhanced data in the frequency domain branch space domain, is the output corresponding to the frequency domain strong enhancement data of the frequency domain branch; then the one-hot encoding method is used to respectively and Convert to pseudo labels and

[0028] The unsupervised loss functions of the spatial domain branch SDB and the frequency domain branch FDB are as follows:

[0029]

[0030] Among them, CE() is the cross entropy loss function and Dice() is the Dice loss function. The pseudo labels generated for the output of the weakly enhanced input data of the frequency domain branch space are: The pseudo labels generated for the output of the spatial branch spatial weak enhancement input data are: is the unsupervised loss function of the spatial domain branch, is the unsupervised loss function of the frequency domain branch;

[0031] Using the pseudo label output by the frequency domain branch FDB To supervise the spatial domain branch SDB. Vice versa, the pseudo label output by the spatial domain branch SDB is used To supervise the frequency domain branch FDB, the unsupervised loss function L for dual domain consistency unsup for:

[0032]

[0033] Further improvements, Among them, L con ( ) is the consistency constraint loss function in the spatial and frequency domains; MSE() represents the minimum mean square error.

[0034] Further improvements, L sup The calculation of is as follows:

[0035] The supervised loss functions of the spatial domain branch SDB and the frequency domain branch FDB are as follows:

[0036]

[0037] y l For x l The label of the superscript l indicates that there is labeled data. and The output of the labeled data corresponding to the spatial domain branch SDB and the frequency domain branch FDB; Indicates the output corresponding to the spatial domain branch with labeled data input, Indicates the output corresponding to the frequency domain branch with labeled data input; In addition, the labeled data x l Randomly sampled from the training set, the rest of the data in the training set is regarded as unlabeled data xu.

[0038] The supervised loss function of the dual-domain branch semi-supervised learning network is:

[0039]

[0040] Beneficial effects brought by the technical solution of the present invention:

[0041] The frequency domain strong enhancement method proposed in the present invention can effectively enhance the edge of skin lesions, which helps the segmentation model to perceive the edge of lesions. Furthermore, the present invention proposes a learning method based on the consistency of the spatial domain and the frequency domain to further enhance the perception and robustness of the segmentation model for skin lesions. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The flowchart of the present invention is shown in Figure 1, where (a) is the frequency domain random adjustment schematic part, (b) is the lesion edge extraction and edge information enhancement part, and (c) is the dual domain consistency learning part. Note: FDB refers to the frequency domain branch (Frequency Domain Branch), SDB (Spatial Domain Branch), Refers to Spatial Weak Perturbation (SWP), Refers to Spatial Strong Perturbation (SSP), It refers to frequency domain strongly enhanced data (Frequency Strong Perturbation, FSP), and the superscript u refers to unlabeled data (unlabeled). For the sake of simplicity and intuitiveness, only the unlabeled data learning part is shown.

[0043] Figure 2 This is the visualization result of the ISIC 2017 dataset with 1% annotation amount;

[0044] Figure 3 This is the visualization result of the ISIC 2017 dataset with 5% annotation.

[0045] Figure 4 This is the visualization result of the ISIC 2018 dataset with 1% annotation amount;

[0046] Figure 5Visualization results for the ISIC 2018 dataset with 5% annotation;

[0047] Figure 6 Visualization results for the HAM10000 dataset with 1% annotation. DETAILED DESCRIPTION

[0048] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0049] like Figure 1 As shown, first, we input the unlabeled image x u Perform a wavelet transform (Discrete WaveletTransform, DWT) to obtain a low-frequency image and level Vertical and diagonal lines High frequency image of direction:

[0050]

[0051] Then the brightness, saturation, etc. of the above four images are randomly adjusted:

[0052]

[0053] In the above formula, They are the randomly adjusted low-frequency and high-frequency images, and then the inverse transformation is obtained:

[0054]

[0055] At this time, there is a huge difference between the inverse transformed image and its spatially enhanced image. Figure 1 (a), further, take the absolute value of the residual between the inverse transformed image and the spatial domain strongly enhanced image:

[0056]

[0057] The absolute value of the residual is used as the edge extraction. Figure 1 As shown in (b), using the extracted edge information The subscript mut means mutation (i.e. the mutation point between the lesion and normal tissue), which further enhances the edge information of the lesion:

[0058]

[0059] At this point, three versions of enhanced images are known: spatial domain weak enhancement Spatial domain strong enhancement Frequency domain strong enhancement To this end, this method designs a dual-domain branch semi-supervised learning method. Figure 1 As shown in (c), one branch is the spatial domain branch SDB, and the other branch is the frequency domain branch FDB. As the input image of SDB, As the input image of FDB, the corresponding output of SDB is The output of FDB is represented as Using one-hot encoding technology and Convert to a pseudo label, that is and Then the unsupervised loss functions of SDB and FDB are as follows:

[0060]

[0061] In the above formula, CE is the cross entropy loss function, and Dice is the Dice loss function. The output pseudo-label of SDB is used to supervise the strong enhancement output of FDB. Similarly, the pseudo-label output of FDB is used to supervise the learning of SDB. Then further, the unsupervised loss function of dual-domain consistency is:

[0062]

[0063] In the present invention, the dual-domain consistency constraint proposed is:

[0064]

[0065] In the above formula, L con For consistency constraints, the minimum mean square error MSE is used for measurement. l The supervised loss functions of SDB and FDB are as follows:

[0066]

[0067] In the above formula, y l For x l The label, the superscript l indicates labeled data (labeled), and is the output of the labeled data corresponding to SDB and FDB respectively. Similarly, the supervised loss function of the dual-domain consistency framework is:

[0068]

[0069] Finally, the loss function of the entire dual-domain consistency learning framework is:

[0070] L=L sup +L unsup +L con (12).

[0071] To verify the effectiveness of the present invention, the open source dermoscopy datasets ISIC 2017, ISIC 2018 and HAM10000 are used for full verification. Regarding the comparison method, the method proposed in the present invention is compared with eight methods published in top journals and conferences in the past three years, namely DTC (AAAI'21), SSNet (MICCAI'22), URPC (MedIA'22), BSNet (TMI'23), BCPNet (CVPR'23), AC-MT (MedIA'23), CauSSL (ICCV'23), and UniMatch (CVPR'23). All comparative experiments use the same UNet network for comparison. In order to ensure fair comparison, the method of the present invention uses single-branch SDB (same UNet architecture) for comparison.

[0072] Verification index: Referring to the above comparison method, the method of the present invention also uses Dice (%) coefficient, Jaccard (%) coefficient, Hausdorff Distance (HD95) and Average Surface Distance (ASD) as quantitative indicators to evaluate the performance of the method.

[0073] Comparative experiment on ISIC 2017: The ISIC 2017 dataset includes 2000 training images, 150 validation images, and 600 test images. Randomly sample 1% (20 images), 5% (100 images), and 10% (200 images) of images as labeled data, and the remaining training images as unlabeled data. Repeat the random sampling three times and take the average and standard deviation of the test results:

[0074] Table 1. ISIC 2017 dataset, 1% annotation amount, three repeated sampling experimental results

[0075]

[0076] Table 2. ISIC 2017 dataset, 5% annotation amount, three repeated sampling experimental results

[0077]

[0078] Table 3. ISIC 2017 dataset, 10% annotation amount, three repeated sampling experimental results

[0079]

[0080] In the above three tables, UNet (Fully Supervised, FS) is a fully supervised method, that is, all data and their labels in the training set are used for training. Similarly, the visualization results of the ISIC 2017 experiment are as follows Figure 2 and Figure 3 shown.

[0081] About the ISIC 2018 dataset experiment: The ISIC 2018 dataset includes 2594 training images, 100 validation images, and 1000 test images. Randomly sample 1% (25 images), 5% (129 images), and 10% (259 images) of images as labeled data, and the remaining training images as unlabeled data. Repeat the random sampling three times and take the average and standard deviation of the test results:

[0082] Table 4. ISIC 2018 dataset, 1% annotation amount, three repeated sampling experimental results

[0083]

[0084] Table 5. ISIC 2018 dataset, 5% annotation amount, three repeated sampling experimental results

[0085]

[0086] Table 6. ISIC 2018 dataset, 10% annotation amount, three repeated sampling experimental results

[0087]

[0088] About the HAM10000 dataset experiment: HAM10000 contains 10015 dermoscopic images, 6000 of which are randomly sampled as training set, 1015 as validation set, and 3000 as test set. Then, in the training set, 1% (60 images) are randomly sampled as labeled data, and the remaining training set images are used as unlabeled data. Repeat the above sampling process three times, and take the average and standard deviation of the results. In view of the highly competitive performance of the UniMatch method. In the HAM10000 dataset comparison experiment, only UniMatch is compared.

[0089]

[0090] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the embodiments. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A semi-supervised dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency, characterized in that: The steps include: Step 1: For the unlabeled image x u Perform weak enhancement in the spatial domain to obtain Weakly enhanced images in the spatial domain Perform strong enhancement to obtain a strongly enhanced image in the spatial domain At the same time, for the unlabeled image x u Perform frequency domain strong enhancement to obtain frequency domain strong enhanced image Step 2: Construct a dual-domain branch semi-supervised learning network. The two branch networks of the dual-domain branch semi-supervised learning network are both neural networks with the same structure, and one of the branches inputs a weakly enhanced image in the spatial domain. And the spatial domain is strongly enhanced to obtain It is called the spatial domain branch SDB; the other branch input data is the spatial domain weakly enhanced image And frequency domain strong enhanced image It is called frequency domain branch FDB; Step 3: As the input data of the spatial domain branch SDB, The frequency domain branch FDB is used as the input data for training to obtain a trained dual-domain branch semi-supervised learning network, and the branch network with the best verification set effect in the trained dual-domain branch semi-supervised learning network is used as the final dermoscopic image segmentation network; Step 4: directly input the dermoscopic image to be segmented into the final dermoscopic image segmentation network to obtain the segmented lesion image.

2. The dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency according to claim 1 is characterized in that: In the step 1, the weak enhancement method is to rotate or flip the image; the strong enhancement method is to perform Gaussian blur or color dithering on the image.

3. The dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency according to claim 1, characterized in that: In the step 2, the neural network is a UNet network.

4. The dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency according to claim 1, characterized in that: The frequency domain strong enhancement data The method of obtaining is as follows: For an unlabeled image x u Perform discrete wavelet transform to obtain low-frequency image And the high-frequency image in the horizontal direction High-frequency image in vertical direction And the high-frequency image in the diagonal direction DWT() stands for discrete wavelet transform; then the low-frequency image High-frequency image in horizontal direction High-frequency image in vertical direction And the high-frequency image in the diagonal direction The brightness and saturation are randomly adjusted: Where RM() represents a random regulator; They are and The image obtained after random adjustment; and Then perform inverse wavelet transform to obtain the inverse transformed image iDWT() stands for inverse wavelet transform; get the inverse transform image Strongly enhanced image in spatial domain The absolute value of the residual is used as the extracted edge information Using the extracted edge information Further enhance the edge information of the lesion to obtain strong frequency domain enhancement data 5. The dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual domain consistency according to claim 1, characterized in that: The total loss function L of the dual-domain branch semi-supervised learning network is: L=L sup +L unsup +L com (6) Among them, L sup represents the supervised loss function of the dual-domain branch semi-supervised learning network, L unsup represents the unsupervised loss function of the dual-domain branch semi-supervised learning network, L con Represents the spatial-frequency domain consistency loss function of a dual-domain branch semi-supervised learning network.

6. The dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual-domain consistency according to claim 5, characterized in that: Unsupervised loss function L for dual-domain branch semi-supervised learning network unsup The method of obtaining is as follows: The output of the spatial domain branch SDB is The output of the frequency domain branch FDB is in is the output corresponding to the spatial domain weak enhancement data of the spatial domain branch, is the output corresponding to the spatial domain branch space strong enhancement data, is the output corresponding to the weakly enhanced data in the frequency domain branch space domain, is the output corresponding to the frequency domain strong enhancement data of the frequency domain branch; then the one-hot encoding method is used to respectively and Convert to pseudo labels and The unsupervised loss functions of the spatial domain branch SDB and the frequency domain branch FDB are as follows: Among them, CE() is the cross entropy loss function, Dice() is the Dice loss function, The pseudo labels generated for the output of the weakly enhanced input data of the frequency domain branch space are: The pseudo labels generated for the output of the spatial branch spatial weak enhancement input data are: is the unsupervised loss function of the spatial domain branch, is the unsupervised loss function of the frequency domain branch; Using the pseudo label output by the frequency domain branch FDB To supervise the spatial domain branch SDB, use the pseudo label output by the spatial domain branch SDB To supervise the frequency domain branch FDB, the unsupervised loss function L for dual domain consistency unsup for:

7. The dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual domain consistency according to claim 6, characterized in that: Among them, L con () is the consistency constraint loss function in the spatial and frequency domains; MSE() represents the minimum mean square error.

8. The dermoscopic image segmentation method based on frequency domain perturbation and space-frequency dual domain consistency according to claim 6, characterized in that: L sup The calculation of is as follows: The supervised loss functions of the spatial domain branch SDB and the frequency domain branch FDB are as follows: y l For label data x l The label of the superscript l indicates that there is labeled data. and The output of the labeled data corresponding to the spatial domain branch SDB and the frequency domain branch FDB; Indicates the output corresponding to the spatial domain branch with labeled data input, Indicates the output corresponding to the frequency domain branch with labeled data input; the labeled data x l Randomly sampled from the training set, the rest of the training set data is considered as unlabeled data x u ; The supervised loss function of the dual-domain branch semi-supervised learning network is:

Images