Domain adaptation medical semantic segmentation method based on style adaptation and boundary enhancement

Abstract

The application provides a domain adaptive medical semantic segmentation method based on style adaptation and boundary enhancement, comprising: obtaining a to-be-tested medical image; introducing the to-be-tested medical image into a trained generator to obtain an image segmentation result of the to-be-tested medical image; obtaining first amplitude information and first phase information of the to-be-tested medical image through fast Fourier transform of the image segmentation result of the to-be-tested medical image; further combining the first amplitude information with the first phase information after signal filtering of the first amplitude information through a low-pass filter, and obtaining a style feature map of the to-be-tested medical image through inverse Fourier transform; introducing the style feature map of the to-be-tested medical image into a trained first discriminator to obtain a first domain classification result of the to-be-tested medical image, and determining whether the to-be-tested medical image is a source domain image or a target domain image. The application can effectively improve the segmentation capability of a semantic segmentation model without additional parameters and labels.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and particularly relates to a domain adaptive medical semantic segmentation method based on style self-adaptation and boundary enhancement. BACKGROUND

[0002] Medical image segmentation aims to separate the region of interest from the medical image, but the complexity and scarcity of medical images make it difficult for ordinary image segmentation methods to achieve ideal performance. Therefore, deep learning medical image segmentation methods based on unsupervised learning have attracted extensive research attention in recent years.

[0003] To solve the medical image segmentation problem, many solutions have been proposed by the academic and industrial communities. For example, Y. Zhang et al. (Zhang Y, Miao S, Mansi T, et al. Task driven generative modeling for unsupervised domain adaptation: Application to x-ray image segmentation [C] / / Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference) used the CycleGAN structure to realize the automatic segmentation and image synthesis of DRRs and X-ray images. For another example, J. Jiang et al. (Jiang J, Hu Y C, Tyagi N, et al. Tumor-aware, adversarial domain adaptation from CT to MRI for lung cancer segmentation [C] / / Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference) proposed a new tumor-aware loss function to preserve the semantic information of the generated MRI images from CT images. However, the overall process of the method of generating target domain images to expand the training samples and improve the semantic segmentation model is relatively complicated, and the stability of the generator is poor, which may cause model shock during training, and the generalization ability of the overall model is limited. For another example, S. Wang et al. (Wang S, Yu L, Li K, et al. Boundary and entropy-driven adversarial learning for fundus image segmentation [C] / / Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference) designed a boundary prediction branch to provide more accurate boundaries and suppress uncertain predictions for fundus image segmentation.Q. Liu et al. (Liu Q, Chen C, Qin J, et al. Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021) introduced a boundary-oriented episodic learning paradigm to enhance the model's generalization ability on fuzzy boundaries, but at the same time, it needs to train additional network parameters to improve the model's boundary segmentation performance, which increases the training burden of the model and weakens the model's generalization ability in the case of small sample training.

[0004] Therefore, the current medical semantic segmentation processing scheme has the following problems: (1) high labeling cost: medical image semantic segmentation requires a lot of professional knowledge and time, and manual labeling costs are high, and there may be subjective errors. (2) Poor generalization ability: due to the complexity and scarcity of medical images, deep learning methods based on large samples and traditional image segmentation methods are difficult to achieve ideal performance, and the generalization ability of the model needs to be further improved.

[0005] Therefore, there is an urgent need for a new domain adaptive medical image segmentation method that can overcome the problems of high labeling cost and poor generalization ability of existing medical semantic segmentation processing methods. SUMMARY

[0006] The technical problem to be solved by the embodiments of the present application is to provide a domain adaptive medical semantic segmentation method based on style adaptation and boundary enhancement, which does not require additional parameters and labels, and can effectively improve the segmentation ability of the semantic segmentation model, thereby overcoming the problems of high labeling cost and poor generalization ability of existing medical semantic segmentation processing methods.

[0007] To solve the above technical problems, the embodiments of the present application provide a domain adaptive medical semantic segmentation method based on style adaptation and boundary enhancement, which comprises the following steps:

[0008] Obtaining a to-be-tested medical image;

[0009] Importing the to-be-tested medical image into a trained generator to obtain an image segmentation result of the to-be-tested medical image;

[0010] The image segmentation result of the to-be-tested medical image is subjected to fast Fourier transform to obtain first amplitude information and first phase information, the first amplitude information is subjected to signal filtering by using a low-pass filter, and then combined with the first phase information, and inverse Fourier transform is performed to obtain a style feature map of the to-be-tested medical image;

[0011] The style feature map of the to-be-tested medical image is introduced into the trained first discriminator to obtain a first domain classification result of the to-be-tested medical image, and according to the first domain classification result of the to-be-tested medical image, it is determined that the to-be-tested medical image is a source domain image or a target domain image.

[0012] The method further comprises:

[0013] The image segmentation result of the to-be-tested medical image is subjected to fast Fourier transform to obtain second amplitude information and second phase information, the second phase information is subjected to inverse Fourier transform, and then added to the image segmentation result of the to-be-tested medical image by feature addition to obtain a boundary-enhanced semantic segmentation result of the to-be-tested medical image;

[0014] The boundary-enhanced semantic segmentation result of the to-be-tested medical image is introduced into the trained second discriminator to obtain a second domain classification result of the to-be-tested medical image, and according to the second domain classification result of the to-be-tested medical image, it is determined that the to-be-tested medical image is a source domain image or a target domain image.

[0015] The loss function of the first discriminator comprises and wherein L B is a binary cross-entropy loss function, and respectively represent the source domain and the target domain Shannon cross-entropy obtained according to the first discriminator D s ;

[0016] The loss function of the second discriminator comprises L Gb =L B (E t ,0) and L Db =L B (E s ,0)+L B (E t ,1); wherein E s and E t respectively represent the source domain and the target domain Shannon cross-entropy obtained according to the second discriminator D b ;

[0017] The loss function of the generator is L=L seg +αL Gb +λLGs ; wherein, both a and l are pre-defined hyperparameters, and a = 0.03, l = 0.001; L Seg = L CE (Y s , P s ) + L Dice (Y s , P s ), P s e C×H×W represents pixel-wise prediction, C, H and W represent the number of label categories, the height and the width of the input image respectively, L CE is a cross-entropy loss function, L Dice represents a Dice loss function commonly used in segmentation tasks, Y s represents a label.

[0018] wherein, the low-pass filter is expressed by the formula M(h, w) = 1 (h,w)∈[-βH:βH,-βW:βW] ; wherein, in M, the values of all regions except the data in the middle region are 0: the center point coordinate of M is (0, 0); (h, w) is a point in M, h and w are the vertical coordinate and the horizontal coordinate of the pixel point respectively; H and W are the height and the width of M respectively; b e (0, 1) is a pre-defined hyperparameter for controlling the filtering range and b = 0.01.

[0019] wherein, the first domain classification result and the second domain classification result are both expressed as wherein, represents that the image comes from the source domain, i.e. the source domain image; represents that the image comes from the target domain, i.e. the target domain image.

[0020] The embodiment of the present application has the following beneficial effects:

[0021] 1. After the image segmentation result of the to-be-tested medical image is obtained by the generator, the image segmentation result is processed into a feature map containing style features through fast Fourier transform and the style-adaptive structure of the low-pass filter, and then the domain classification result of the to-be-tested medical image is quickly identified in the first discriminator, without the need for additional parameters and additional labels, and the segmentation capability of the semantic segmentation model can be effectively improved, thereby overcoming the problems of high labeling cost and insufficient generalization capability of the existing medical semantic segmentation processing method;

[0022] 2. After the image segmentation result of the to-be-tested medical image is obtained by the generator, the image segmentation result is processed into a feature map with enhanced boundary information through fast Fourier transform and the boundary enhancement structure of information superposition, and then the domain classification result of the to-be-tested medical image is quickly identified in the second discriminator, also without the need for additional parameters and additional labels, and the segmentation capability of the semantic segmentation model can be further effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings obtained according to these drawings without creative labor are still within the scope of the present application.

[0024] Figure 1 A flow chart of a domain adaptive medical semantic segmentation method based on style adaptation and boundary enhancement provided by an embodiment of the present application;

[0025] Figure 2 A working principle diagram of a generator, a first discriminator and a second discriminator identifying medical images in an application scenario of a domain adaptive medical semantic segmentation method based on style adaptation and boundary enhancement provided by an embodiment of the present application;

[0026] Figure 3 For Figure 2 An enlarged view of a source domain image and a target domain image; wherein a is a source domain image, and b is a target domain image;

[0027] Figure 4 A semantic segmentation result graph calculated for a target domain image in an application scenario of a domain adaptive medical semantic segmentation method based on style adaptation and boundary enhancement provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present application more clear, the following will further describe the present application in combination with the drawings.

[0029] As shown in Figure 1 As shown in Fig. 1, a domain adaptive medical semantic segmentation method based on style adaptation and boundary enhancement provided by an embodiment of the present application, the method comprises the following steps:

[0030] Step S1, obtaining a to-be-tested medical image;

[0031] Step S2, importing the to-be-tested medical image into a trained generator to obtain an image segmentation result of the to-be-tested medical image;

[0032] Step S3, obtaining first amplitude information and first phase information of the image segmentation result of the to-be-tested medical image through fast Fourier transform, filtering the first amplitude information through a low-pass filter, further combining with the first phase information, and obtaining a style feature graph of the to-be-tested medical image through inverse Fourier transform;

[0033] Step S4, introducing the style feature map of the to-be-tested medical image into the trained first discriminator to obtain a first domain classification result of the to-be-tested medical image, and determining whether the to-be-tested medical image is a source domain image or a target domain image according to the first domain classification result of the to-be-tested medical image.

[0034] Specifically, before step S1, a generator G, a first discriminator D s and a second discriminator D b are constructed in advance and trained. At this time, the training process of the generator G, the first discriminator D s and the second discriminator D b is as follows:

[0035] Let D S ={X S , Y S} represent a labeled source data set, and D t ={X t} represent an unlabeled target data set, where X s , X t represent input medical images, and Y s represents a label.

[0036] Step 1: randomly sample n groups from the data sets D S ={X S , Y S} and D t ={X t} to form training sample data X, X ∈ {X s , X t};

[0037] Step 2: input the training sample data X into the generator G to obtain image segmentation result P;

[0038] Step 3: input the image segmentation result P into the style self-adaptive structure to obtain the style feature map P s ;

[0039] Step 4: input P s into the first discriminator D s to obtain the domain classification result of D s , and calculate the loss functions L Gs and L Ds :

[0040]

[0041]

[0042] wherein L B is a binary cross-entropy loss function, and respectively represent the Shannon cross-entropy of the source domain and the target domain obtained according to the first discriminator D s

[0043] Step 5: input the image segmentation result P into the boundary enhancement structure to obtain a boundary-enhanced semantic segmentation result P b

[0044] Step 6: input P b into the second discriminator D b to obtain the domain classification result of D b , and calculate the loss functions L Gb and L Db :

[0045] L Gb = L B (E t ,0)

[0046] L Db = L B (E s ,0)+L B (E t ,1)

[0047] wherein E s and E t respectively represent the Shannon cross-entropy of the source domain and the target domain obtained according to the second discriminator D b

[0048] Step 7: calculate the segmentation loss function L b of P Seg on the source domain:

[0049] L Seg = L CE (Y s ,P s )+L Dice (Y s ,P s )

[0050] wherein P b ∈ C×H×W represents pixel-wise prediction, C, H and W represent the number of label categories, the height and the width of the input image respectively, L CE is a cross-entropy loss function, and L Dice represents a Dice loss function commonly used in segmentation tasks;

[0051] Step 8: calculate the mixed loss function of the generator G:

[0052] L = L seg + αL Gb + λL​​​Gs

[0053] wherein, a and l are pre-defined hyperparameters, such as a = 0.03, l = 0.001;

[0054] Step 9: Generator G, first discriminator D f and second discriminator D b respectively according to loss functions L, L Gs and L Gb , and then according to the errors to calculate the forward propagation error values, and then according to the errors to perform back propagation;

[0055] Step 10: repeat steps 1 to 9 until e iterations are completed, to obtain the trained generator G, first discriminator D s and second discriminator D b ; wherein, e is a pre-defined training round number.

[0056] In step S1, a single to-be-tested medical image I is read.

[0057] In step S2, the to-be-tested medical image I is input into the trained generator G, to obtain an image segmentation result P of the to-be-tested medical image I.

[0058] In step S3, the image segmentation result P of the to-be-tested medical image I is subjected to fast Fourier transform to obtain first amplitude information A1 and first phase information P h1 , the first amplitude information A1 is subjected to signal filtering using a low-pass filter, and then combined with the first phase information P h1 , and then subjected to inverse Fourier transform to obtain a style feature map P s of the to-be-tested medical image I; wherein, the low-pass filter is expressed by a formula M(h, w) = 1 (h,w)∈[-βH:βH,-βW:βW] ; at this time, in M, the values of all regions except the data in the middle region are 0: the center point coordinates of M are (0, 0); (h, w) is a point in M, h and w are respectively the vertical coordinate and horizontal coordinate of a pixel point; H and W are respectively the height and width of M; b e (0, 1) is a pre-defined hyperparameter for controlling the filtering range and b = 0.01.

[0059] In step S4, the style feature map P s of the to-be-tested medical image I is input into the trained first discriminator D s , to obtain a first domain classification result of the to-be-tested medical image I, and then determine whether the to-be-tested medical image I is a source domain image or a target domain image according to the first domain classification result of the to-be-tested medical image I; wherein, represents that the image comes from the source domain, i.e., the source domain image; If the image is from the target domain, i.e., a target domain image.

[0060] In the embodiment of the present application, the feature map with enhanced boundary information can also be processed by the boundary enhancement structure, and then introduced into the second discriminator to quickly identify the domain classification result of the to-be-tested medical image. Therefore, the method further comprises:

[0061] The image segmentation result P of the to-be-tested medical image I is subjected to fast Fourier transform to obtain second amplitude information A2 and second phase information P h2 , and the second phase information P h2 is added to the image segmentation result P of the to-be-tested medical image I by inverse Fourier transform and feature addition to obtain the boundary-enhanced semantic segmentation result P of the to-be-tested medical image I b .

[0062] The boundary-enhanced semantic segmentation result P of the to-be-tested medical image I is introduced into the trained second discriminator D b to obtain the second domain classification result of the to-be-tested medical image I b . According to the second domain classification result of the to-be-tested medical image I , it is determined whether the to-be-tested medical image I is a source domain image or a target domain image; wherein, If the image is from the source domain, i.e., a source domain image; If the image is from the target domain, i.e., a target domain image.

[0063] As shown in Figures 2 to 4 , it is a related scene diagram of a domain adaptive medical semantic segmentation method based on style adaptation and boundary enhancement provided in an embodiment of the present application. Among them, Figure 2 is a working principle diagram of the generator, the first discriminator and the second discriminator for identifying medical images; Figure 4 is a semantic segmentation result diagram calculated for a target domain image.

[0064] The embodiment of the present application has the following beneficial effects:

[0065] 1、The present application obtains the image segmentation result of the to-be-tested medical image through the generator, and then processes the image segmentation result into a feature map containing style features through the style adaptation structure of fast Fourier transform and low-pass filter, and then introduces the feature map into the first discriminator to quickly identify the domain classification result of the to-be-tested medical image, without the need for additional parameters and additional labels, and can effectively improve the segmentation ability of the semantic segmentation model, thereby overcoming the problems of high labeling cost and insufficient generalization ability of the existing medical semantic segmentation processing method;

[0066] 2、The application obtains the image segmentation result of the to-be-tested medical image through the generator, and then processes the image segmentation result into a feature map with enhanced boundary information through fast Fourier transform and boundary enhancement structure processing of information superposition, and then imports the feature map into the second discriminator to quickly identify the domain classification result of the to-be-tested medical image, without additional parameters and additional labels, and the segmentation capability of the semantic segmentation model can be further improved effectively.

[0067] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment methods can be completed by programs instructing relevant hardware, and the programs can be stored in a computer readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc.

[0068] The above only describes the preferred embodiments of the present application, and of course cannot limit the scope of the present application, so equivalent changes made according to the claims of the present application are still within the scope of the present application.

Claims

1. A domain adaptation medical semantic segmentation method based on style adaptation and boundary enhancement, characterized in that, The method comprises the following steps: obtaining a to-be-tested medical image; introducing the to-be-tested medical image into a trained generator to obtain an image segmentation result of the to-be-tested medical image; obtaining first amplitude information and first phase information of the image segmentation result of the to-be-tested medical image through fast Fourier transform, further combining the first amplitude information with the first phase information after signal filtering of the first amplitude information by using a low-pass filter, and obtaining a style feature map of the to-be-tested medical image through inverse Fourier transform; introducing the style feature map of the to-be-tested medical image into a trained first discriminator to obtain a first domain classification result of the to-be-tested medical image, and determining whether the to-be-tested medical image is a source domain image or a target domain image according to the first domain classification result of the to-be-tested medical image; The method further comprises: obtaining second amplitude information and second phase information of the image segmentation result of the to-be-tested medical image through fast Fourier transform, adding the second phase information to the image segmentation result of the to-be-tested medical image through inverse Fourier transform by using a feature addition method to obtain a boundary-enhanced semantic segmentation result of the to-be-tested medical image; introducing the boundary-enhanced semantic segmentation result of the to-be-tested medical image into a trained second discriminator to obtain a second domain classification result of the to-be-tested medical image, and determining whether the to-be-tested medical image is a source domain image or a target domain image according to the second domain classification result of the to-be-tested medical image.

2. The style-adaptive and boundary-enhanced domain adaptation medical semantic segmentation method based on style according to claim 1, wherein, The loss function of the first discriminator comprises and ; wherein, is a binary cross-entropy loss function, and respectively represent the Shannon cross-entropy of the source domain and the target domain obtained according to the first discriminator D s . The loss function of the second discriminator comprises and ; wherein, and respectively represent the Shannon cross-entropy of the source domain and the target domain obtained according to the second discriminator D b ​ The loss function of the generator is ; wherein α and λ are all pre-given hyperparameters, and α = 0.03, λ = 0.001; , P s ∈ ℝ C×H×W represents a pixel-by-pixel prediction, C , H and W respectively represent the number of label categories, the height and the width of the input image, L CE is a cross-entropy loss function, L Dice represents a Dice loss function commonly used in segmentation tasks, represents a label.

3. The style-adaptive and boundary-enhanced domain adaptation medical semantic segmentation method based on style according to claim 1, wherein, The low-pass filter is expressed by the formula wherein, in M , the values of the rest of the regions are 0 except for the data of the middle region being 1: M the center point coordinate of h , w is a point in M , h and w are the vertical and horizontal coordinates of the pixel point, respectively; H and W are the height and width of M , respectively; β ∈(0, 1) is a hyperparameter of the control filter range given in advance and β = 0.

01.

4. The style-adaptive and boundary-enhanced domain adaptation medical semantic segmentation method of claim 1, wherein, The first domain classification result and the second domain classification result are both represented as l^∈{0, 1} ; wherein, l^= 1 represents that the image comes from the source domain, i.e., a source domain image; l^=0 then represents that the image comes from the target domain, i.e., a target domain image.

Images